All posts by Jeff Barr

Amazon WorkSpaces Pools: Cost-effective, non-persistent virtual desktops

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-workspaces-pools-cost-effective-non-persistent-virtual-desktops/

You can now create a pool of non-persistent virtual desktops using Amazon WorkSpaces and share them across a group of users. As the desktop administrator you can manage your entire portfolio of persistent and non-persistent virtual desktops using one GUI, command line, or set of API-powered tools. Your users can log in to these desktops using a browser, a client application (Windows, Mac, or Linux), or a thin client device.

Amazon WorkSpaces Pools (non-persistent desktops)
WorkSpaces Pools ensures that each user gets the same applications and the same experience. When the user logs in, they always get access to a fresh WorkSpace that’s based on the latest configuration for the pool, centrally managed by their administrator. If the administrator enables application settings persistence for the pool, users can configure certain application settings, such as browser favorites, plugins, and UI customizations. Users can also access persistent file or object storage external to the desktop.

These desktops are a great fit for many types of users and use cases including remote workers, task workers (shared service centers, finance, procurement, HR, and so forth), contact center workers, and students.

As the administrator for the pool, you have full control over the compute resources (bundle type) and the initial configuration of the desktops in the pool, including the set of applications that are available to the users. You can use an existing custom WorkSpaces image, create a new one, or use one of the standard ones. You can also include Microsoft 365 Apps for Enterprise on the image. You can configure the pool to accommodate the size and working hours of your user base, and you can optionally join the pool to your organization’s domain and active directory.

Getting started
Let’s walk through the process of setting up a pool and inviting some users. I open the WorkSpaces console and choose Pools to get started:

I have no pools, so I choose Create WorkSpace on the Pools tab to begin the process of creating a pool:

The console can recommend workspace options for me, or I can choose what I want. I leave Recommend workspace options… selected, and choose No – non-persistent to create a pool of non-persistent desktops. Then I select my use cases from the menu and pick the operating system and choose Next to proceed:

The use case menu has lots of options:

On the next page I start by reviewing the WorkSpace options and assigning a name to my pool:

Next, I scroll down and choose a bundle. I can pick a public bundle or a custom one of my own. Bundles must use the WSP 2.0 protocol. I can create a custom bundle to provide my users with access to applications or to alter any desired system settings.

Moving right along, I can customize the settings for each user session. I can also enable application settings persistence to save application customizations and Windows settings on a per-user basis between sessions:

Next, I set the capacity of my pool, and optionally establish one or more schedules based on date or time. The schedules give me the power to match the size of my pool (and hence my costs) to the rhythms and needs of my users:

If the amount of concurrent usage is more dynamic and not aligned to a schedule, then I can use manual scale out and scale in policies to control the size of my pool:


I tag my pool, and then choose Next to proceed:

The final step is to select a WorkSpaces pool directory or create a new one following these steps. Then, I choose Create WorkSpace pool.

WorkSpaces Pools Directory

After the pool has been created and started, I can send registration codes to users, and they can log in to a WorkSpace:

WorkSpaces Pools Login with Registration Code

I can monitor the status of the pool from the console:

WorkSpaces Pool Status On Console

Things to know
Here are a couple of things that you should know about WorkSpaces Pools:

Programmatic access – You can automate the setup process that I showed above by using functions like CreateWorkSpacePool, DescribeWorkSpacePool, UpdateWorkSpacePool, or the equivalent AWS command line interface (CLI) commands.

Regions – WorkSpaces Pools is available in all commercial AWS Regions where WorkSpaces Personal is available, except Israel (Tel Aviv), Africa (Cape Town), and China (Ningxia). Check the full Region list for future updates.

Pricing – Refer to the Amazon WorkSpaces Pricing page for complete pricing information.

Visit Amazon WorkSpaces Pools to learn more.

Jeff;

Optimizing Amazon Simple Queue Service (SQS) for speed and scale

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/optimizing-amazon-simple-queue-service-sqs-for-speed-and-scale/

After several public betas, we launched Amazon Simple Queue Service (Amazon SQS) in 2006. Nearly two decades later, this fully managed service is still a fundamental building block for microservices, distributed systems, and serverless applications, processing over 100 million messages per second at peak times.

Because there’s always a better way, we continue to look for ways to improve performance, security, internal efficiency, and so forth. When we do find a potential way to do something better, we are careful to preserve existing behavior, and often run new and old systems in parallel to allow us to compare results.

Today I would like to tell you how we recently made improvements to Amazon SQS to reduce latency, increase fleet capacity, mitigate an approaching scalability cliff, and reduce power consumption.

Improving SQS
Like many AWS services, Amazon SQS is implemented using a collection of internal microservices. Let’s focus on two of them today:

Customer Front-End – The customer-facing front-end accepts, authenticates, and authorizes API calls such as CreateQueue and SendMessage. It then routes each request to the storage back-end.

Storage Back-End -This internal microservice is responsible for persisting messages sent to standard (non-FIFO) queues. Using a cell-based model, each cluster in the cell contains multiple hosts, each customer queue is assigned to one or more clusters, and each cluster is responsible for a multitude of queues:

Connections – Old and New
The original implementation used a connection per request between these two services. Each front-end had to connect to many hosts, which mandated the use of a connection pool, and also risked reaching an ultimate, hard-wired limit on the number of open connections. While it is often possible to simply throw hardware at problems like this and scale out, that’s not always the best way. It simply moves the moment of truth (the “scalability cliff”) into the future and does not make efficient use of resources.

After carefully considering several long-term solutions, the Amazon SQS team invented a new, proprietary binary framing protocol between the customer front-end and storage back-end. The protocol multiplexes multiple requests and responses across a single connection, using 128-bit IDs and checksumming to prevent crosstalk. Server-side encryption provides an additional layer of protection against unauthorized access to queue data.

It Works!
The new protocol was put into production earlier this year and has processed 744.9 trillion requests as I write this. The scalability cliff has been eliminated and we are already looking for ways to put this new protocol to work in other ways.

Performance-wise, the new protocol has reduced dataplane latency by 11% on average, and by 17.4% at the P90 mark. In addition to making SQS itself more performant, this change benefits services that build on SQS as well. For example, messages sent through Amazon Simple Notification Service (Amazon SNS) now spend 10% less time “inside” before being delivered. Finally, due to the protocol change, the existing fleet of SQS hosts (a mix of X86 and Graviton-powered instances) can now handle 17.8% more requests than before.

More to Come
I hope that you have enjoyed this little peek inside the implementation of Amazon SQS. Let me know in the comments, and I will see if I can find some more stories to share.

Jeff;

IAM Access Analyzer Update: Extending custom policy checks & guided revocation

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/iam-access-analyzer-update-extending-custom-policy-checks-guided-revocation/

We are making IAM Access Analyzer even more powerful, extending custom policy checks and adding easy access to guidance that will help you to fine-tune your IAM policies. Both of these new features build on the Custom Policy Checks and the Unused Access analysis that were launched at re:Invent 2023. Here’s what we are launching:

New Custom Policy Checks – Using the power of automated reasoning, the new checks help you to detect policies that grant access to specific, critical AWS resources, or that grant any type of public access. Both of the checks are designed to be used ahead of deployment, possibly as part of your CI/CD pipeline, and will help you proactively detect updates that do not conform to your organization’s security practices and policies.

Guided Revocation – IAM Access Analyzer now gives you guidance that you can share with your developers so that they can revoke permissions that grant access that is not actually needed. This includes unused roles, roles with unused permissions, unused access keys for IAM users, and unused passwords for IAM users. The guidance includes the steps needed to either remove the extra items or to replace them with more restrictive ones.

New Custom Policy Checks
The new policy checks can be invoked from the command line or by calling an API function. The checks examine a policy document that is supplied as part of the request and return a PASS or FAIL value. In both cases, PASS indicates that the policy document properly disallows the given access, and FAIL indicates that the policy might allow some or all of the permissions. Here are the new checks:

Check No Public Access – This check operates on a resource policy, and checks to see if the policy grants public access to a specified resource type. For example, you can check a policy to see if it allows public access to an S3 bucket by specifying the AWS::S3::Bucket resource type. Valid resource types include DynamoDB tables and streams, EFS file systems, OpenSearch domains, Kinesis streams and stream consumers, KMS keys, Lambda functions, S3 buckets and access points, S3 Express directory buckets, S3 Outposts buckets and access points, Glacier, Secrets Manager secrets, SNS topics and queues, and IAM policy documents that assume roles. The list of valid resource types will expand over time and can be found in the CheckNoPublicAccess documentation,

Let’s say that I have a policy which accidentally grants public access to an Amazon Simple Queue Service (Amazon SQS) queue. Here’s how I check it:

$ aws accessanalyzer check-no-public-access --policy-document file://resource.json \
  --resource-type AWS::SQS::Queue --output json

And here is the result:

{
    "result": "FAIL",
    "message": "The resource policy grants public access for the given resource type.",
    "reasons": [
        {
            "description": "Public access granted in the following statement with sid: SqsResourcePolicy.",
            "statementIndex": 0,
            "statementId": "SqsResourcePolicy"
        }
    ]
}

I edit the policy to remove the access grant and try again, and this time the check passes:

{
    "result": "PASS",
    "message": "The resource policy does not grant public access for the given resource type."
}

Check Access Not Granted – This check operates on a single resource policy or identity policy at a time. It also accepts an list of actions and resources, both in the form that are acceptable as part of an IAM policy. The check sees if the policy grants unintended access to any of the resources in the list by way of the listed actions. For example, this check could be used to make sure that a policy does not allow a critical CloudTrail trail to be deleted:

$ aws accessanalyzer check-access-not-granted --policy-document file://ct.json \
  --access resources="arn:aws:cloudtrail:us-east-1:123456789012:trail/MySensitiveTrail" \
  --policy-type IDENTITY_POLICY --output json

IAM Access Analyzer indicates that the check fails:

{
    "result": "FAIL",
    "message": "The policy document grants access to perform one or more of the listed actions or resources.",
    "reasons": [
        {
            "description": "One or more of the listed actions or resources in the statement with index: 0.",
            "statementIndex": 0
        }
    ]
}

I fix the policy and try again, and this time the check passes, indicating that the policy does not grant access to the listed resources:

{
    "result": "PASS",
    "message": "The policy document does not grant access to perform the listed actions or resources."
}

Guided Revocation
In my earlier post I showed you how IAM Access Analyzer discovers and lists IAM items that grant access which is not actually needed. With today’s launch, you now get guidance to help you (or your developer team) to resolve these findings. Here are the latest findings from my AWS account:

Some of these are leftovers from times when I was given early access to a service so that I could use and then blog about it; others are due to my general ineptness as a cloud admin! Either way, I need to clean these up. Let’s start with the second one, Unused access key. I click on the item and can see the new Recommendations section at the bottom:

I can follow the steps and delete the access key or I can click Archive to remove the finding from the list of active findings and add it to the list of archived ones. I can also create an archive rule that will do the same for similar findings in the future. Similar recommendations are provided for unused IAM users, IAM roles, and passwords.

Now let’s take a look at a finding of Unused permissions:

The recommendation is to replace the existing policy with a new one. I can preview the new policy side-by-side with the existing one:

As in the first example I can follow the steps or I can archive the finding.

The findings and the recommendations are also available from the command line. I generate the recommendation by specifying an analyzer and a finding from it:

$ aws accessanalyzer generate-finding-recommendation \
  --analyzer-arn arn:aws:access-analyzer-beta:us-west-2:123456789012:analyzer/MyAnalyzer \
  --id 67110f3e-05a1-4562-b6c2-4b009e67c38e

Then I retrieve the recommendation. In this example, I am filtering the output to only show the steps since the entire JSON output is fairly rich:

$ aws accessanalyzer get-finding-recommendation \
  --analyzer-arn arn:aws:access-analyzer-beta:us-west-2:123456789012:analyzer/MyAnalyzer \
  --id 67110f3e-05a1-4562-b6c2-4b009e67c38e --output json | \
  jq .recommendedSteps[].unusedPermissionsRecommendedStep.recommendedAction
"CREATE_POLICY"
"DETACH_POLICY"

You can use these commands (or the equivalent API calls) to integrate the recommendations into your own tools and systems.

Available Now
The new checks and the resolution steps are available now and you can start using them today in all public AWS regions!

Jeff;

Amazon EC2 high memory U7i Instances for large in-memory databases

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-ec2-high-memory-u7i-instances-for-large-in-memory-databases/

Announced in preview form at re:Invent 2023, Amazon Elastic Compute Cloud (Amazon EC2) U7i instances with up to 32 TiB of DDR5 memory and 896 vCPUs are now available. Powered by custom fourth generation Intel Xeon Scalable Processors (Sapphire Rapids), these high memory instances are designed to support large, in-memory databases including SAP HANA, Oracle, and SQL Server. Here are the specs:

Instance Name vCPUs
Memory (DDR5)
EBS Bandwidth
Network Bandwidth
u7i-12tb.224xlarge 896 12,288 GiB 60 Gbps 100 Gbps
u7in-16tb.224xlarge 896 16,384 GiB 100 Gbps 200 Gbps
u7in-24tb.224xlarge 896 24,576 GiB 100 Gbps 200 Gbps
u7in-32tb.224xlarge 896 32,768 GiB 100 Gbps 200 Gbps

The new instances deliver the best compute price performance for large in-memory workloads, and offer the highest memory and compute power of any SAP-certified virtual instance from a leading cloud provider.

Thanks to AWS Nitro System, all of the memory on the instance is available for use. For example, here’s the 32 TiB instance:

In comparison to the previous generation of EC2 High Memory instances, the U7i instances offer more than 135% of the compute performance, up to 115% more memory performance, and 2.5x the EBS bandwidth. This increased bandwidth allows you to transfer 30 TiB of data from EBS into memory in an hour or less, making data loads and cache refreshes faster than ever before. The instances also support ENA Express with 25 Gbps of bandwidth per flow, and provide an 85% improvement in P99.9 latency between instances.

Each U7i instance supports attachment of up to 128 General Purpose (gp2 and gp3) or Provisioned IOPS (io1 and io2 Block Express) EBS volumes. Each io2 Block Express volume can be as big as 64 TiB and can deliver up to 256K IOPS at up to 32 Gbps, making them a great match for U7i instances.

The instances are SAP certified to run Business Suite on HANA, Business Suite S/4HANA, Business Warehouse on HANA (BW), and SAP BW/4HANA in production environments. To learn more, consult the Certified and Supported SAP HANA Hardware and the SAP HANA to AWS Migration Guide. Also, be sure to take a look at the AWS Launch Wizard for SAP.

Things to Know
Here are a couple of things that you should know about these new instances:

Regions – U7i instances are available in the US East (N. Virginia), US West (Oregon), and Asia Pacific (Seoul, Sydney) AWS Regions.

Operating Systems – Supported operating systems include Amazon Linux, Red Hat Enterprise Linux, SUSE Linux Enterprise Server, Ubuntu, and Windows Server.

Larger Instances – We are also working on offering even larger instance later this year with increased compute to meet our customer needs.

Jeff;

Amazon Bedrock model evaluation is now generally available

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-bedrock-model-evaluation-is-now-generally-available/

The Amazon Bedrock model evaluation capability that we previewed at AWS re:Invent 2023 is now generally available. This new capability helps you to incorporate Generative AI into your application by giving you the power to select the foundation model that gives you the best results for your particular use case. As my colleague Antje explained in her post (Evaluate, compare, and select the best foundation models for your use case in Amazon Bedrock):

Model evaluations are critical at all stages of development. As a developer, you now have evaluation tools available for building generative artificial intelligence (AI) applications. You can start by experimenting with different models in the playground environment. To iterate faster, add automatic evaluations of the models. Then, when you prepare for an initial launch or limited release, you can incorporate human reviews to help ensure quality.

We received a lot of wonderful and helpful feedback during the preview and used it to round-out the features of this new capability in preparation for today’s launch — I’ll get to those in a moment. As a quick recap, here are the basic steps (refer to Antje’s post for a complete walk-through):

Create a Model Evaluation Job – Select the evaluation method (automatic or human), select one of the available foundation models, choose a task type, and choose the evaluation metrics. You can choose accuracy, robustness, and toxicity for an automatic evaluation, or any desired metrics (friendliness, style, and adherence to brand voice, for example) for a human evaluation. If you choose a human evaluation, you can use your own work team or you can opt for an AWS-managed team. There are four built-in task types, as well as a custom type (not shown):

After you select the task type you choose the metrics and the datasets that you want to use to evaluate the performance of the model. For example, if you select Text classification, you can evaluate accuracy and/or robustness with respect to your own dataset or a built-in one:

As you can see above, you can use a built-in dataset, or prepare a new one in JSON Lines (JSONL) format. Each entry must include a prompt and can include a category. The reference response is optional for all human evaluation configurations and for some combinations of task types and metrics for automatic evaluation:

{
  "prompt" : "Bobigny is the capitol of",
  "referenceResponse" : "Seine-Saint-Denis",
  "category" : "Capitols"
}

You (or your local subject matter experts) can create a dataset that uses customer support questions, product descriptions, or sales collateral that is specific to your organization and your use case. The built-in datasets include Real Toxicity, BOLD, TREX, WikiText-2, Gigaword, BoolQ, Natural Questions, Trivia QA, and Women’s Ecommerce Clothing Reviews. These datasets are designed to test specific types of tasks and metrics, and can be chosen as appropriate.

Run Model Evaluation Job – Start the job and wait for it to complete. You can review the status of each of your model evaluation jobs from the console, and can also access the status using the new GetEvaluationJob API function:

Retrieve and Review Evaluation Report – Get the report and review the model’s performance against the metrics that you selected earlier. Again, refer to Antje’s post for a detailed look at a sample report.

New Features for GA
With all of that out of the way, let’s take a look at the features that were added in preparation for today’s launch:

Improved Job Management – You can now stop a running job using the console or the new model evaluation API.

Model Evaluation API – You can now create and manage model evaluation jobs programmatically. The following functions are available:

  • CreateEvaluationJob – Create and run a model evaluation job using parameters specified in the API request including an evaluationConfig and an inferenceConfig.
  • ListEvaluationJobs – List model evaluation jobs, with optional filtering and sorting by creation time, evaluation job name, and status.
  • GetEvaluationJob – Retrieve the properties of a model evaluation job, including the status (InProgress, Completed, Failed, Stopping, or Stopped). After the job has completed, the results of the evaluation will be stored at the S3 URI that was specified in the outputDataConfig property supplied to CreateEvaluationJob.
  • StopEvaluationJob – Stop an in-progress job. Once stopped, a job cannot be resumed, and must be created anew if you want to rerun it.

This model evaluation API was one of the most-requested features during the preview. You can use it to perform evaluations at scale, perhaps as part of a development or testing regimen for your applications.

Enhanced Security – You can now use customer-managed KMS keys to encrypt your evaluation job data (if you don’t use this option, your data is encrypted using a key owned by AWS):

Access to More Models – In addition to the existing text-based models from AI21 Labs, Amazon, Anthropic, Cohere, and Meta, you now have access to Claude 2.1:

After you select a model you can set the inference configuration that will be used for the model evaluation job:

Things to Know
Here are a couple of things to know about this cool new Amazon Bedrock capability:

Pricing – You pay for the inferences that are performed during the course of the model evaluation, with no additional charge for algorithmically generated scores. If you use human-based evaluation with your own team, you pay for the inferences and $0.21 for each completed task — a human worker submitting an evaluation of a single prompt and its associated inference responses in the human evaluation user interface. Pricing for evaluations performed by an AWS managed work team is based on the dataset, task types, and metrics that are important to your evaluation. For more information, consult the Amazon Bedrock Pricing page.

Regions – Model evaluation is available in the US East (N. Virginia) and US West (Oregon) AWS Regions.

More GenAI – Visit our new GenAI space to learn more about this and the other announcements that we are making today!

Jeff;

Amazon CloudWatch Internet Weather Map – View and analyze internet health

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/amazon-cloudwatch-internet-weather-map-view-and-analyze-internet-health/

The Internet has a plethora of moving parts: routers, switches, hubs, terrestrial and submarine cables, and connectors on the hardware side, and complex protocol stacks and configurations on the software side. When something goes wrong that slows or disrupts the Internet in a way that affects your customers, you want to be able to localize and understand the issue as quickly as possible.

New Map
The new Amazon CloudWatch Internet Weather Map is here to help! Built atop of collection of global monitors operated by AWS, you get a broad, global view of Internet weather, with the ability to zoom in and understand performance and availability issues that affect a particular city. To access the map, open the CloudWatch Console, expand Network monitoring on the left, and click Internet Monitor. The map appears and displays weather for the entire world:

The red and yellow circles indicate current, active issues that affect availability or performance, respectively. The grey circles represent issues that have been resolved within the last 24 hours, and the blue diamonds represent AWS regions. The map will automatically refresh every 15 minutes if you leave it on the screen.

Each issue affects a specific city-network, representing a combination of a location where clients access AWS resources, and the Autonomous System Number (ASN) that was used to access the resources. ASNs typically represent individual Internet Service Providers (ISPs).

The list to the right of the map shows active events at the top, followed by events that have been resolved in the recent past, looking back up to 24 hours:

I can hover my mouse over any of the indicators to see the list of city-networks in the geographic area:

If I zoom in a step or two, I can see that those city-networks are spread out over the United States:

I can zoom in even further and see a single city-network:

This information is also available programmatically. The new ListInternetEvents function returns up to 100 performance or availability events per call, with optional filtering by time range, status (ACTIVE or RESOLVED), or type (PERFORMANCE or AVAILABILITY). Each event includes full details including latitude and longitude.

The new map is accessible from all AWS regions and there is no charge to use it. Going forward, we have a lot of powerful additions on the roadmap, subject to prioritization based on your feedback. Right now we are thinking about:

  1. Displaying causes of certain types of outages such as DDoS attacks, BGP route leaks, and issues with route interconnects.
  2. Adding a view that is specific to a chosen ISP.
  3. Displaying the impact to public SaaS applications.

Please feel free to send feedback on this feature to [email protected] .

CloudWatch Internet Monitor
The information in the map applies to everyone who makes use of applications built on AWS. If you want to understand how internet weather affects your particular AWS applications and to take advantage of other features such as health event notification and traffic insights, you can make use of CloudWatch Internet Monitor. As my colleague Sébastien wrote when he launched this feature in late 2022:

You told us one of your challenges when monitoring internet-facing applications is to gather data outside of AWS to build a realistic picture of how your application behaves for your customers connected to multiple and geographically distant internet providers. Capturing and monitoring data about internet traffic before it reaches your infrastructure is either difficult or very expensive.

After you review the map, you can click Create monitor to get started with CloudWatch Internet Monitor:

After that you enter a name for your monitor, choose the AWS resources (VPCs, CloudFront distributions, Network Load Balancers, and Amazon WorkSpace Directories) to monitor, then select the desired percentage of internet-facing traffic to monitor. The monitor will begin to operate within minutes, using entries from your VPC Flow Logs, CloudFront Access Logs, and other telemetry to identify the most relevant city-networks.

Here are some resources to help you learn more about this feature:

Jeff;

Congratulations to the PartyRock generative AI hackathon winners

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/congratulations-to-the-partyrock-generative-ai-hackathon-winners/

The PartyRock Generative AI Hackathon wrapped up earlier this month. Entrants were asked to use PartyRock to build a functional app based on one of four challenge categories, with the option to remix an existing app as well. The hackathon attracted 7,650 registrants who submitted over 1,200 projects, and published over 250 project blog posts on community.aws .

As a member of the judging panel, I was blown away by the creativity and sophistication of the entries that I was asked to review. The participants had the opportunity to go hands-on with prompt engineering and to learn about Foundation Models, and pushed the bounds of what was possible.

Let’s take a quick look at the winners of the top 3 prizes…

First Place
First up, taking home the top overall prize of $20,000 in AWS credits is Parable Rhythm – The Interactive Crime Thriller by Param Birje. This project immerses you in a captivating interactive story using PartyRock’s generative capabilities. Just incredible stuff.

To learn more, read the hackathon submission and the blog post.

Second Place
In second place, earning $10,000 in credits, is Faith – Manga Creation Tools by Michael Oswell. This creative assistant app lets you generate original manga panels and panels with the click of a button. So much potential there.

To learn more, read the hackathon submission.

Third Place
And rounding out the top 3 overall is Arghhhh! Zombie by Michael Eziamaka. This is a wildly entertaining generative AI-powered zombie game that had the judges on the edge of their seats. Great work, Michael!

To learn more, read the hackathon submission.

Round of Applause
I want to give a huge round of applause to all our category winners as well:

Category / Place Submission Prize (USD) AWS Credits
Overall 1st Place Parable Rhythm $20,000
Overall 2nd Place Faith – Manga Creation Tools $10,000
Overall 3rd Place Arghhhh! Zombie $5,000
Creative Assistants 1st Place Faith – Manga Creation Tools $4,000 $1,000
Creative Assistants 2nd Place MovieCreator $1,500 $1,000
Creative Assistants 3rd Place WingPal $500 $1,000
Experimental Entertainment 1st Place Parable Rhythm $4,000 $1,000
Experimental Entertainment 2nd Place Arghhhh! Zombie $1,500 $1,000
Experimental Entertainment 3rd Place Find your inner potato $500 $1,000
Interactive Learning 1st Place DeBeat Coach $4,000 $1,000
Interactive Learning 2nd Place Asteroid Mining Assistant $1,500 $1,000
Interactive Learning 3rd Place Unlock your pet’s language $500 $1,000
Freestyle 1st Place MindMap Party $1,000 $1,000
Freestyle 2nd Place Angler Advisor $750 $1,000
Freestyle 3rd Place SafeScares $250 $1,000
BONUS: Remix ChatRPG Inferno $2,500
BONUS: Remix ChatRPG Chat RPG Generator $2,500

From interactive learning experiences to experimental entertainment, the creativity and technical execution on display was off the charts.

And of course, a big thank you to all 7,650 participants who dove in and pushed the boundaries of what’s possible with generative AI. You all should be extremely proud.

Join the Party
You can click on any of the images above and try out the apps for yourself. You can remix and customize them, and you can build your own apps as well (read my post, Build AI apps with PartyRock and Amazon Bedrock to see how to get started).

Alright, that’s a wrap. Congrats again to our winners, and a huge thanks to the PartyRock team and all our amazing sponsors. I can’t wait to see what you all build next. Until then, keep building, keep learning, and keep having fun!

Jeff;

AWS Supply Chain update: Three new modules supporting upstream activities

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-supply-chain-update-three-new-modules-supporting-upstream-activities/

We are launching three new modules for AWS Supply Chain today. These modules are designed to help you collaborate with your suppliers across all tiers of your supply chain, with the goal of helping you to maintain optimum inventory levels at each site in the chain. Here’s an overview:

Supply Planning – This module helps you to accurately forecast and plan purchases of raw materials, components, and finished goods. It uses multiple algorithms to create supply plans that include purchase orders and inventory transfer requirements.

N-Tier Visibility – This module extends visibility and collaboration beyond your enterprise’s internal systems to multiple external tiers of trading partners.

Sustainability – this module creates a more secure and efficient way for you to request, collect, and review data for carbon emissions, as well as reports on hazardous material used in the acquisition, manufacturing, transportation, and disposal of goods. You can now send data requests to multiple tiers of trading partners, track responses, send reminders to absentees, and provide a central repository to store and view responses.

Let’s take a look at each one…

Supply Planning
AWS Supply Chain already includes a Demand Planning module which uses proprietary machine learning algorithms to forecast demand and generate a demand plan that is based on two or more years of historical order line data. The forecasts are granular and specific, including distribution centers and retail outlets.

The new Supply Planning module uses the demand plan as an input. It looks at existing inventory, accounts for uncertainties, and supports additional business input including stocking strategies, ultimately generating purchase orders for components and raw materials, ready for review and approval. Here is the main page of the Supply Planning module:

The module also supports auto replenishment and manufacturing plans. The manufacturing plans work backward from a Bill of Materials (BoM) which is broken down (exploded) into individual components that are sourced from multiple direct and indirect upstream sources.

Supply Planning is done with respect to a planning horizon and on a plan schedule, both of which are defined in the organization profile:

The settings for this module also allow for customization of purchase order review and approval:

N-Tier Visibility
This module helps you to work in a collaborative fashion with your vendors, the vendors that supply your vendors, and so forth. It automatically detects vendors and sets them up for on-boarding into AWS Supply Chain. The module supports manual and EDI-powered collaboration on purchase orders, while also helping to identify discrepancies and risks, and to find substitute vendors if necessary.

The main page of the module displays an overview of my trading partners:

The Portal status column indicates that some of these partners have already onboarded, others have been invited (and one let the invite expire), and others have yet to be invited. I can click Invite partners to extend invitations. I select the partners (these have generally been auto-discovered using data in the Supply Chain Data Lake), and click Continue:

Then I enter the contact information for each partner that I selected, and click Send invites:

The contact receives an invitation via email and can then accept the invite. After they have accepted, they can receive and respond to supply plans and purchase orders electronically (via email or EDI).

Sustainability
The Sustainability module helps you to request, receive, and review compliance and sustainability data from your partners. It builds on the partner network that I already described, and tracks requests for data:

To request data, I select the type of data that I need and the partners that I need it from, then click Continue:

Then I enter the details that define my request, including a due date. I can ask the chosen partners for a text response and/or a file response:

The responses and files provided by each partner are written to the Supply Chain Data Lake and can also be exported to an Amazon Simple Storage Service (Amazon S3) bucket.

AWS Supply Chain Resources
If you are new to AWS Supply Chain and would like to learn more, here are some resources to get you started:

Jeff;

Use AWS Fault Injection Service to demonstrate multi-region and multi-AZ application resilience

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/use-aws-fault-injection-service-to-demonstrate-multi-region-and-multi-az-application-resilience/

AWS Fault Injection Service (FIS) helps you to put chaos engineering into practice at scale. Today we are launching new scenarios that will let you demonstrate that your applications perform as intended if an AWS Availability Zone experiences a full power interruption or connectivity from one AWS region to another is lost.

You can use the scenarios to conduct experiments that will build confidence that your application (whether single-region or multi-region) works as expected when something goes wrong, help you to gain a better understanding of direct and indirect dependencies, and test recovery time. After you have put your application through its paces and know that it works as expected, you can use the results of the experiment for compliance purposes. When used in conjunction with other parts of AWS Resilience Hub, FIS can help you to fully understand the overall resilience posture of your applications.

Intro to Scenarios
We launched FIS in 2021 to help you perform controlled experiments on your AWS applications. In the post that I wrote to announce that launch, I showed you how to create experiment templates and to use them to conduct experiments. The experiments are built using powerful, low-level actions that affect specified groups of AWS resources of a particular type. For example, the following actions operate on EC2 instances and Auto Scaling Groups:

With these actions as building blocks, we recently launched the AWS FIS Scenario Library. Each scenario in the library defines events or conditions that you can use to test the resilience of your applications:

Each scenario is used to create an experiment template. You can use the scenarios as-is, or you can take any template as a starting point and customize or enhance it as desired.

The scenarios can target resources in the same AWS account or in other AWS accounts:

New Scenarios
With all of that as background, let’s take a look at the new scenarios.

AZ Availability: Power Interruption – This scenario temporarily “pulls the plug” on a targeted set of your resources in a single Availability Zone including EC2 instances (including those in EKS and ECS clusters), EBS volumes, Auto Scaling Groups, VPC subnets, Amazon ElastiCache for Redis clusters, and Amazon Relational Database Service (RDS) clusters. In most cases you will run it on an application that has resources in more than one Availability Zone, but you can run it on a single-AZ app with an outage as the expected outcome. It targets a single AZ, and also allows you to disallow a specified set of IAM roles or Auto Scaling Groups from being able to launch fresh instances or start stopped instances during the experiment.

The New actions and targets experience makes it easy to see everything at a glance — the actions in the scenario and the types of AWS resources that they affect:

The scenarios include parameters that are used to customize the experiment template:

The Advanced parameters – targeting tags lets you control the tag keys and values that will be used to locate the resources targeted by experiments:

Cross-Region: Connectivity – This scenario prevents your application in a test region from being able to access resources in a target region. This includes traffic from EC2 instances, ECS tasks, EKS pods, and Lambda functions attached to a VPC. It also includes traffic flowing across Transit Gateways and VPC peering connections, as well as cross-region S3 and DynamoDB replication. The scenario looks like this out of the box:

This scenario runs for 3 hours (unless you change the disruptionDuration parameter), and isolates the test region from the target region in the specified ways, with advanced parameters to control the tags that are used to select the affected AWS resources in the isolated region:

You might also find that the Disrupt and Pause actions used in this scenario useful on their own:

For example, the aws:s3:bucket-pause-replication action can be used to pause replication within a region.

Things to Know
Here are a couple of things to know about the new scenarios:

Regions – The new scenarios are available in all commercial AWS Regions where FIS is available, at no additional cost.

Pricing – You pay for the action-minutes consumed by the experiments that you run; see the AWS Fault Injection Service Pricing Page for more info.

Naming – This service was formerly called AWS Fault Injection Simulator.

Jeff;

Introducing highly durable Amazon OpenSearch Service clusters with 30% price/performance improvement

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/introducing-highly-durable-amazon-opensearch-service-clusters-with-30-price-performance-improvement/

You can use the new OR1 instances to create Amazon OpenSearch Service clusters that use Amazon Simple Storage Service (Amazon S3) for primary storage. You can ingest, store, index, and access just about any imaginable amount of data, while also enjoying a 30% price/performance improvement over existing instance types, eleven nines of data durability, and a zero-time Recovery Point Objective (RPO). You can use this to perform interactive log analytics, monitor application in real time, and more.

New OR1 Instances
These benefits are all made possible by the new OR1 instances, which are available in eight sizes and used for the data nodes of the cluster:

Instance Name vCPUs
Memory
EBS Storage Max (gp3)
or1.medium.search 1 8 GiB 400 GiB
or1.large.search 2 16 GiB 800 GiB
or1.xlarge.search 4 32 GiB 1.5 TiB
or1.2xlarge.search 8 64 GiB 3 TiB
or1.4xlarge.search 16 128 GiB 6 TiB
or1.8xlarge.search 32 256 GiB 12 TiB
or1.12xlarge.search 48 384 GiB 18 TiB
or1.16xlarge.search 64 512 GiB 24 TiB

To choose a suitable instance size, read Sizing Amazon OpenSearch Service domains.

The Amazon Elastic Block Store (Amazon EBS) volumes are used for primary storage, with data copied synchronously to S3 as it arrives. The data in S3 is used to create replicas and to rehydrate EBS after shards are moved between instances as a result of a node failure or a routine rebalancing operation. This is made possible by the remote-backed storage and segment replication features that were recently released for OpenSearch.

Creating a Domain
To create a domain I open the Amazon OpenSearch Service Console, select Managed clusters, and click Create domain:

I enter a name for my domain (my-domain), select Standard create, and use the Production template:

Then I choose the Domain with standby deployment option. This option will create active data nodes in two Availability Zones and a standby one in a third. I also choose the latest engine version:

Then I select the OR1 instance family and (for my use case) configure 500 GiB of EBS storage per data node:

I set the other settings as needed, and click Create to proceed:

I take a quick lunch break and when i come back my domain is ready:

Things to Know
Here are a couple of things to know about this new storage option:

Engine Versions – Amazon OpenSearch Service engines version 2.11 and above support OR1 instances.

Regions – The OR1 instance family is available for use with OpenSearch in the US East (Ohio, N. Virginia), US West (N. California, Oregon), Asia Pacific (Mumbai, Singapore, Sydney, Tokyo), and Europe (Frankfurt, Ireland, Spain, Stockholm) AWS Regions.

Pricing – You pay On-Demand or Reserved prices for data nodes, and you also pay for EBS storage. See the Amazon OpenSearch Service Pricing page for more information.

Jeff;

Join the preview for new memory-optimized, AWS Graviton4-powered Amazon EC2 instances (R8g)

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/join-the-preview-for-new-memory-optimized-aws-graviton4-powered-amazon-ec2-instances-r8g/

We are opening up a preview of the next generation of Amazon Elastic Compute Cloud (Amazon EC2) instances. Equipped with brand-new Graviton4 processors, the new R8g instances will deliver better price performance than any existing memory-optimized instance. The R8g instances are suitable for your most demanding memory-intensive workloads: big data analytics, high-performance databases, in-memory caches and so forth.

Graviton history
Let’s take a quick look back in time and recap the evolution of the Graviton processors:

November 2018 – The Graviton processor made its debut in the A1 instances, optimized for both performance and cost, and delivering cost reductions of up to 45% for scale-out workloads.

December 2019 – The Graviton2 processor debuted with the announcement of M6g, M6gd, C6g, C6gd, R6g, and R6gd instances with up to 40% better price performance than equivalent non-Graviton instances. The second-generation processor delivered up to 7x performance of the first one, including twice the floating point performance.

November 2021 – The Graviton3 processor made its debut with the announcement of the compute-optimized C7g instances. In addition to up to 25% better compute performance, this generation of processors once again doubled floating point and cryptographic performance when compared to the previous generation.

November 2022 – The Graviton 3E processor was announced, for use in the Hpc7g and C7gn instances, with up to 35% higher vector instruction processing performance than the Graviton3.

Today, every one of the top 100 Amazon Elastic Compute Cloud (EC2) customers makes use of Graviton, choosing between more than 150 Graviton-powered instances.

New Graviton4
I’m happy to be able to tell you about the latest in our series of innovative custom chip designs, the energy-efficient AWS Graviton4 processor.

96 Neoverse V2 cores, 2 MB of L2 cache per core, and 12 DDR5-5600 channels work together to make the Graviton4 up to 40% faster for databases, 30% faster for web applications, and 45% faster for large Java applications than the Graviton3.

Graviton4 processors also support all of the security features from the previous generations, and includes some important new ones including encrypted high-speed hardware interfaces and Branch Target Identification (BTI).

R8g instance sizes
The 8th generation R8g instances will be available in multiple sizes with up to triple the number of vCPUs and triple the amount of memory of the 7th generation (R7g) of memory-optimized, Graviton3-powered instances.

Join the preview
R8g instances with Graviton4 processors

Jeff;

Announcing the new Amazon S3 Express One Zone high performance storage class

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-s3-express-one-zone-high-performance-storage-class/

The new Amazon S3 Express One Zone storage class is designed to deliver up to 10x better performance than the S3 Standard storage class while handling hundreds of thousands of requests per second with consistent single-digit millisecond latency, making it a great fit for your most frequently accessed data and your most demanding applications. Objects are stored and replicated on purpose built hardware within a single AWS Availability Zone, allowing you to co-locate storage and compute (Amazon EC2, Amazon ECS, and Amazon EKS) resources to further reduce latency.

Amazon S3 Express One Zone
With very low latency between compute and storage, the Amazon S3 Express One Zone storage class can help to deliver a significant reduction in runtime for data-intensive applications, especially those that use hundreds or thousands of parallel compute nodes to process large amounts of data for AI/ML training, financial modeling, media processing, real-time ad placement, high performance computing, and so forth. These applications typically keep the data around for a relatively short period of time, but access it very frequently during that time.

This new storage class can handle objects of any size, but is especially awesome for smaller objects. This is because for smaller objects the time to first byte is very close to the time for last byte. In all storage systems, larger objects take longer to stream because there is more data to download during the transfer, and therefore the storage latency has less impact on the total time to read the object. As a result, smaller objects receive an outsized benefit from lower storage latency compared to large objects. Because of S3 Express One Zone’s consistent very low latency, small objects can be read up to 10x faster compared to S3 Standard.

The extremely low latency provided by Amazon S3 Express One Zone, combined with request costs that are 50% lower than for the S3 Standard storage class, means that your Spot and On-Demand compute resources are used more efficiently and can be shut down earlier, leading to an overall reduction in processing costs.

Each Amazon S3 Express One Zone directory bucket exists in a single Availability Zone that you choose, and can be accessed using the usual set of S3 API functions: CreateBucket, PutObject, GetObject, ListObjectsV2, and so forth. The buckets also support a carefully chosen set of S3 features including byte-range fetches, multi-part upload, multi-part copy, presigned URLs, and Access Analyzer for S3. You can upload objects directly, write code that uses CopyObject, or use S3 Batch Operations,

In order to reduce latency and to make this storage class as efficient & scalable as possible, we are introducing a new bucket type, a new authentication model, and a bucket naming convention:

New bucket type – The new directory buckets are specific to this storage class, and support hundreds of thousands of requests per second. They have a hierarchical namespace and store object key names in a directory-like manner. The path delimiter must be “/“, and any prefixes that you supply to ListObjectsV2 must end with a delimiter. Also, list operations return results without first sorting them, so you cannot do a “start after” retrieval.

New authentication model – The new CreateSession function returns a session token that grants access to a specific bucket for five minutes. You must include this token in the requests that you make to other S3 API functions that operate on the bucket or the objects in it, with the exception of CopyObject, which requires IAM credentials. The newest versions of the AWS SDKs handle session creation automatically.

Bucket naming – Directory bucket names must be unique within their AWS Region, and must specify an Availability Zone ID in a specially formed suffix. If my base bucket name is jbarr and it exists in Availability Zone use1-az5 (Availability Zone 5 in the US East (N. Virginia) Region) the name that I supply to CreateBucket would be jbarr--use1-az5--x-s3. Although the bucket exists within a specific Availability Zone, it is accessible from the other zones in the region, and there are no data transfer charges for requests from compute resources in one Availability Zone to directory buckets in another one in the same region.

Amazon S3 Express One Zone in action
Let’s put this new storage class to use. I will focus on the command line, but AWS Management Console and API access are also available.

My EC2 instance is running in my us-east-1f Availability Zone. I use jq to map this value to an Availability Zone Id:

$ aws ec2 describe-availability-zones --output json | \
  jq -r  '.AvailabilityZones[] | select(.ZoneName == "us-east-1f") | .ZoneId'
use1-az5

I create a bucket configuration (s3express-bucket-config.json) and include the Id:

{
        "Location" :
        {
                "Type" : "AvailabilityZone",
                "Name" : "use1-az5"
        },
        "Bucket":
        {
                "DataRedundancy" : "SingleAvailabilityZone",
                "Type"           : "Directory"
        }
}

After installing the newest version of the AWS Command Line Interface (AWS CLI), I create my directory bucket:

$ aws s3api create-bucket --bucket jbarr--use1-az5--x-s3 \
  --create-bucket-configuration file://s3express-bucket-config.json \
  --region us-east-1
-------------------------------------------------------------------------------------------
|                                       CreateBucket                                      |
+----------+------------------------------------------------------------------------------+
|  Location|  https://jbarr--use1-az5--x-s3.s3express-use1-az5.us-east-1.amazonaws.com/   |
+----------+------------------------------------------------------------------------------+

Then I can use the directory bucket as the destination for other CLI commands as usual (the second aws is the directory where I unzipped the AWS CLI):

$ aws s3 sync aws s3://jbarr--use1-az5--x-s3

When I list the directory bucket’s contents, I see that the StorageClass is EXPRESS_ONEZONE:

$ aws s3api list-objects-v2 --bucket jbarr--use1-az5--x-s3 --output json | \
  jq -r '.Contents[] | {Key: .Key, StorageClass: .StorageClass}'
...
{
  "Key": "install",
  "StorageClass": "EXPRESS_ONEZONE"
}
...

The Management Console for S3 shows General purpose buckets and Directory buckets on separate tabs:

I can import the contents of an existing bucket (or a prefixed subset of the contents) into a directory bucket using the Import button, as seen above. I select a source bucket, click Import, and enter the parameters that will be used to generate an inventory of the source bucket and to create and an S3 Batch Operations job.

The job is created and begins to execute:

Things to know
Here are some important things to know about this new S3 storage class:

Regions – Amazon S3 Express One Zone is available in the US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Stockholm) Regions, with plans to expand to others over time.

Other AWS Services – You can use Amazon S3 Express One Zone with other AWS services including Amazon SageMaker Model Training, Amazon Athena, Amazon EMR, and AWS Glue Data Catalog to accelerate your machine learning and analytics workloads. You can also use Mountpoint for Amazon S3 to process your S3 objects in file-oriented fashion.

Pricing – Pricing, like the other S3 storage classes, is on a pay-as-you-go basis. You pay $0.16/GB/month in the US East (N. Virginia) Region, with a one-hour minimum billing time for each object, and additional charges for certain request types. You pay an additional per-GB fee for the portion of any request that exceeds 512 KB. For more information, see the Amazon S3 Pricing page.

Durability – In the unlikely case of the loss or damage to all or part of an AWS Availability Zone, data in a One Zone storage class may be lost. For example, events like fire and water damage could result in data loss. Apart from these types of events, our One Zone storage classes use similar engineering designs as our Regional storage classes to protect objects from independent disk, host, and rack-level failures, and each are designed to deliver 99.999999999% data durability.

SLA – Amazon S3 Express One Zone is designed to deliver 99.95% availability with an availability SLA of 99.9%; for information see the Amazon S3 Service Level Agreement page.

This new storage class is available now and you can start using it today!

Learn more
Amazon S3 Express One Zone

Jeff;

Announcing new diagnostic tools for AWS Partner-Led Support (PLS) participants

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/announcing-new-diagnostic-tools-for-aws-partner-led-support-pls-participants/

We have added a set of diagnostic tools that will give participants in the AWS Partner-Led Support program access to diagnostic tools that will empower them to do an even better job of supporting their customers.

Intro to AWS Partner-Led Support
This AWS Partner Network (APN) program enables AWS Partners to act as the customer’s sole point of contact for technical support. Customers contact their support partner for technical assistance instead of directly contacting AWS. In many cases the partner can resolve the issue directly. If the partner cannot do this, they get guidance from AWS via their AWS Support plan.

Diagnostic tools
These are the same tools that AWS Support Engineers use to assist AWS customers.

When a customer contacts their partner for support, the partner will federate into the customer’s AWS account. Then they will use the new diagnostic tools to access the customer metadata that will help them to identify and diagnose the issue.

The tools are enabled by a set of IAM roles set up by the customer. The tools can access and organize metadata and CloudWatch metrics, but they cannot access customer data and they cannot make any changes to any of the customer’s AWS resources. Here is a small sample of the types of information that partners will be able to access:

  • EC2 Capacity Reservations
  • Lambda Functions List
  • GuardDuty Findings
  • Load Balancer Responses
  • RDS and Redshift Clusters

Each tool operates on a list of regions selected when the tool is run, all invocations of each tool are logged and are easily accessible for review, and the output from each invocation can be directed to one of several different regions.

The tools can be invoked from the AWS Management Console, with API access available in order to support in-house tools, automation, and integration.

Learn more

The service is available today for partners that have joined the Partner-Led Support program. For more information, see the AWS Partner Led Support page.

If you are a current AWS Partner and would like to learn more about this program with an eye toward qualifying and participating, please visit AWS Partner Central.

Learn more about AWS Diagnostic Tools here.

Jeff;

FlexGroup Volume Management for Amazon FSx for NetApp ONTAP is now available

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/flexgroup-volume-management-for-amazon-fsx-for-netapp-ontap-is-now-available/

You can now create, manage, and back up your Amazon FSx for NetApp ONTAP FlexGroup volumes using the AWS Management Console, the Amazon FSx CLI, and the AWS SDK. FlexGroups can be as large as 20 petabytes and offer greater performance for demanding workloads. Before this launch you could only create them using the ONTAP CLI and the ONTAP REST API (these options remain available). Also new to this launch is the ability to create Amazon FSx backups of your FlexGroup volumes.

FlexVol and FlexGroup
FSx for ONTAP supports two volume styles:

FlexVol – Support for up to 300 TiB of storage, making these volumes a good fit for general-purpose workloads.

FlexGroup – Support for up to 20 PiB of storage and billions of files per volume, making these volumes a good fit for more demanding electronic design automation (EDA), seismic analysis, and software build/test workloads.

Using FlexGroups
I will use the AWS Management Console to create a new file system. I select Amazon FSx for NetApp ONTAP, and click Next:

I select Standard create, enter a name for my file system (FS-Jeff-1), and select Single-AZ as the deployment type:

I can use the recommended throughput capacity, or I can specify it explicitly:

As you can surmise from the values above, the throughput is determined by the number of high availability (HA) pairs that will be used to host your file system. A single-AZ file system can be hosted on up to 6 such pairs; a multi-AZ file system must reside on a single pair. To learn more about these options visit New – Scale-out file systems for Amazon FSx for NetApp ONTAP.

After making my selections for Network & security, Encryption, and Default storage virtual machine configuration, I select the FlexGroup volume style, assign a name to the initial volume, and either accept the recommended number of constituents or specify it myself:

On the next page I review my choices and click Create file system:

The creation process is a good time for a lunch break. When I return the initial volume (Vol1) of my file system is ready to use. I can create additional FlexVol or FlexGroup volumes as needed:

Things to Know
Here are a couple of things to keep in mind about FlexGroup volumes:

Constituents – Although each FlexGroup volume can have as many as 200 constituents, we recommend 8 per HA pair. Given the 300 TiB per-constituent size limit, this allows you to create volumes with up to 2.4 PiB of storage per HA pair. ONTAP will balance your files across constituents automatically.

File Counts – If you are using NFSv3 and expect to store many billions of files on a FlexGroup volume, be sure to enable 64-bit identifiers on the storage virtual machine associated with the file system.

Backups – Starting today you can also create backups of FlexGroup volumes, giving you the same fully-managed built-in options that you already have for FlexVol volumes.

NetApp System Manager – You can use the ONTAP CLI and the browser-based NetApp System Manager to perform advanced operations on your ONTAP file systems, storage virtual machines, and volumes. The management endpoint and administrator credentials are available on the File system details page:

Regions – Both volume styles are available in all AWS Regions where Amazon FSx for NetApp ONTAP is supported.

Pricing – You pay for the SSD storage, SSD IOPS, and throughput capacity that you provision, with separate charges for capacity pool usage, backups, and SnapLock licensing; see the Amazon FSx for NetApp ONTAP Pricing page to learn more.

Jeff;

New – Scale-out file systems for Amazon FSx for NetApp ONTAP

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-scale-out-file-systems-for-amazon-fsx-for-netapp-ontap/

You can now create Amazon FSx for NetApp ONTAP file systems that are up to 9x faster than even before. As is already the case with this service, the file systems are fully managed, with latency to primary storage at the sub-millisecond level and latency to the capacity pool in the tens of milliseconds. This new level of performance will allow you to bring even more of your most demanding electronic design automation (EDA), visual effects (VFX), and statistical computing workloads (to name just a few) to the cloud.

Existing FSx for ONTAP scale-up file systems are powered by a single pair of servers in an active/passive high availability (HA) configuration, and can support a maximum of 4 GBps of throughput and 192 TiB of SSD storage. The server pair can be deployed across distinct fault domains of a single Availability Zone, or in two separate Availability Zones to provide continuous availability even when an Availability Zone is unavailable.

With today’s launch you can now create scale-out FSx for ONTAP file systems that are powered by two to six HA pairs. Here are the specs for the scale-up and scale-out file systems (these are all listed as “up to” since you can specify your desired values for each one when you create your file system):

Deployment Type
Read
Throughput
Write
Throughput
SSD IOPS
SSD Storage
Availability Zones
Scale-up Up to 4 GBps Up to 1.8 GBps Up to 160K Up to 192 TiB Single or Multiple
Scale-out Up to 36 GBps Up to 6.6 GBps Up to 1.2M Up to 1 PiB Single

The amount of throughput that you specify for your file system will determine the actual server configuration as follows:

Specified Throughput
Deployment
Type
HA Pairs
Throughput
(Per Server)
SSD Storage
(Per Server)
SSD IOPS
(Per Server)
 4 GBps or less Scale-up Single Up to 4 GBps Read
Up to 1.1 GBps Write (Single-AZ)
Up to 1.8 GBps Write (Multi-AZ)
1-192 TiB Up to 160K
More than 4 GBps Scale-out Up to 6 Up to 6 GBps Read
Up to 1.1 GBps Write
1-512 TiB Up to 200K

To learn more about your choices, visit Amazon FSx for NetApp ONTAP performance.

Creating a Scale-Out File System
I can create a scale-out file system using the AWS Management Console, AWS Command Line Interface (AWS CLI), or by writing code that calls the Amazon FSx CreateFileSystem function. I will use the console, and start by choosing Amazon FSx for NetApp ONTAP:

I choose Standard create, enter a name, select a Single-AZ deployment, and enter my desired SSD storage capacity. I can accepted the recommended throughput capacity, choose a value from the dropdown, or enter a value and see my options (more on that in a sec):

The dropdown has a helpful magical feature. If I type my desired throughput capacity it will show me one or more options:

The console will sometimes show me several options for the same amount of desired throughput capacity. Here are some guidelines to help you make a choice that will be a good fit for your workload:

Low Throughput – If you choose an option that provides 4 GBps or less, you will be running on a single HA pair. This is the simplest option to choose if you don’t need a high degree of throughput.

High Throughput and/or High Storage – Maximum throughput scales with the number of HA pairs that you provision. Also, choosing an option with more pairs will maximize your headroom for future growth in provisioned storage.

I make selections and enter values for the remaining options as usual, and click Next to review my settings. I check to make sure that I have made good choices for any of the attributes that can’t be edited after creation, and click Create file system.

I take a break to see what’s happening upstairs, feed my dog, and when I return my new 2 TB file system is ready to go:

I can increase the storage capacity and I can change provisioned IOPS as frequently as every 6 hours:

At the moment, the provisioned throughput capacity cannot be changed after creation if the file system uses more than one HA pair.

Available Now
Scale-out file systems are available in the US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Sydney), and Europe (Ireland) Regions and you can start to create and use them today.

Learn more

Jeff;

Introducing shared VPC support for Amazon FSx for NetApp ONTAP

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/introducing-shared-vpc-support-for-amazon-fsx-for-netapp-ontap/

You can now create Multi-AZ FSx for ONTAP file systems in VPCs that have been shared with you by other accounts in the same AWS Organization. This highly requested feature enables a clean separation of duties between network administrators and storage administrators, and makes it possible to create storage that’s durable, highly available, and accessible from multiple VPCs.

Shared VPC support
Before today’s launch, you had the ability to create Single-AZ FSx for ONTAP file systems in subnets that were shared with you by another AWS account, as well as both Single – and Multi-AZ file systems in subnets that you own.

With today’s launch you can now do the same for file systems in multiple Availability Zones. Multi-AZ FSx for ONTAP file systems offer even higher availability than Single-AZ file systems, and are a great way to address and support large-scale enterprise storage needs. This new support for shared VPCs gives enterprises, many of which make use of multiple VPCs for technical and organizational reasons, to use FSx for ONTAP in Multi-AZ deployments, while allowing network administrators and storage administrators to work independently.

This is easy to set up, but you do need to make sure that there are no IP address conflicts between subnets that are not shared between VPCs. I don’t have an AWS Organization set up, so I will hand-wave through part of this process. As a network administrator (the owner account), I use the AWS Resource Access Manager (RAM) to share the appropriate subnets of my VPC with the desired participant accounts in my Organization:

Then I (or the administrators for those accounts) accept the resource shares.

Next, I use the new FSx for ONTAP Settings to enable route table updates from participant accounts, and click Submit (this gives the FSx ONTAP service permission to modify route table entries in shared subnets on behalf of participant accounts):

At this point, the storage administrators for the participant accounts can create Multi-AZ FSx for ONTAP file systems in the subnets that have been shared with them by the owner accounts.

There is no additional charge for this feature and it is available in all AWS Regions where FSx for ONTAP is supported.

Jeff;

IAM Access Analyzer updates: Find unused access, check policies before deployment

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/iam-access-analyzer-updates-find-unused-access-check-policies-before-deployment/

We are launching two new features for AWS Identity and Access Management (IAM) Access Analyzer today:

Unused Access Analyzer – A new analyzer that continuously monitors roles and users looking for permissions that are granted but not actually used. Central security teams can take advantage of a dashboard view that will help them to find the accounts that can most benefit from a review of unused permissions, roles, and IAM users.

Custom Policy Checks – Validation that newly authored policies do not grant additional (and perhaps unintended) permissions. You can exercise tighter control over your IAM policies and accelerate the process of moving AWS applications from development to production by adding automated policy reviews to your CI/CD pipelines and custom policy tools.

Let’s take a look at today’s launches!

Unused Access Analyzer
You can already create an analyzer that monitors for external access. With today’s launch you can create one that looks for access permissions that are either overly generous or that have fallen into disuse. This includes unused IAM roles, unused access keys for IAM users, unused passwords for IAM users, and unused services and actions for active IAM roles and users.

After reviewing the findings generated by an organization-wide or account-specific analyzer, you can take action by removing permissions that you don’t need. You can create analyzers and analyze findings from the AWS Management Console, CLI, or API. Let’s start with the IAM Console. I click Analyzers and settings in the left-side navigation:

I can see my current analyzers (none, in this case). I click Create analyzer to proceed:

I specify Unused access analysis, leave the default tracking period of 90 days as-is, and opt to check my account rather than my Organization, then I click Create analyzer:

My analyzer is created, and I check back a little while later to see what it finds. My findings were available within a minute, but this will vary. Here are some of the findings:

As you can see, I have lots of unused IAM roles and permissions (clearly I am a bad Role model). I can click on a Finding to learn more:

If this is a role that I need, I can click Archive to remove it from the list of active findings. I can also create archive rules that will do the same for similar findings:

The external access analyzer works in a similar way, and is a perfect place to start when you are new to Access Analyzer and are ready to find and remove extra permissions:

The dashboard gives me an overview of all active findings:

If I create an analyzer and specify my Organization as the Zone of trust, I can also view a list that shows the accounts that have the largest number of active findings:

This feature is also available from the command line. I can create a new analyzer like this:

$ aws access-analyzer create-analyzer --type ACCOUNT_UNUSED_ACCESS \
  --analyzer-name OneWeek \
  --configuration '{"unusedAccess" : {"unusedAccessAge" : 90}}'
----------------------------------------------------------------------------
|                              CreateAnalyzer                              |
+-----+--------------------------------------------------------------------+
|  arn|  arn:aws:access-analyzer:us-east-1:348414629041:analyzer/OneWeek   |
+-----+--------------------------------------------------------------------+

I can list the findings, perhaps all I want is the actual resource Ids to start:

$  aws access-analyzer list-findings-v2 \
  --analyzer-arn  arn:aws:access-analyzer:us-east-1:123456789012:analyzer/OneWeek 
  --output json |
 jq -r '.findings[] | .resource'

arn:aws:iam::123456789012:role/MobileHub_Service_Role
arn:aws:iam::123456789012:role/EKSClusterRole
arn:aws:iam::123456789012:role/service-role/AWSDataSyncS3BucketAccess-jbarr-data
arn:aws:iam::123456789012:role/rds-monitoring-role
arn:aws:iam::123456789012:role/IsengardRoleForDependencyAssuranceIamAnalyzer
arn:aws:iam::123456789012:role/service-role/s3crr_role_for_rep-src_to_rep-dest
arn:aws:iam::123456789012:role/service-role/AWSDeepRacerServiceRole
...

I can archive findings by Id:

$ aws access-analyzer update-findings  \
  --analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/OneWeek 
  --status ARCHIVED --ids "f0492061-8638-48ac-b91a-f0583cc839bf"

And I can perform the same operations using the IAM Access Analyzer API.

This feature is priced based on the number of IAM roles analyzed each month and is available in all AWS Regions where IAM is available.

Custom Policy Checks
You can now validate that IAM policies adhere to your security standards ahead of deployments and proactively detect non-conformant updates to policies. This will help you to innovate more quickly, move apps from development to production more efficiently, and to have confidence that any changes you make represent your intent.

Let’s start with my allow-all-ssm policy:

For illustrative purposes, I edit it to add S3 access:

Then I click Check for new access, confirm that I understand that a charge will be made, and click Check policy:

The automated reasoning verifies the policy and tells me that I did enable new access. If that was my intent I click Next to proceed, otherwise I rethink my changes to the policy:

This is a very simple and contrived example, but I am confident that you can see how useful and valuable this can be to your security efforts. You can also access this from the CLI (check-no-new-access) and API (CheckNoNewAccess).

There’s also another command and function that is designed to be used in your CI/CD pipelines, AWS CloudFormation hooks, and custom policy tools. check-access-not-granted and CheckAccessNotGranted accept a policy document and a permission such as s3:Get*, and check to make sure that the policy does not grant the permission. You could use this, for example, to make sure that a policy which specifies that Security Hub should be disabled cannot be deployed. This will help you to move from development to production with the confidence that your policies adhere to your organization’s security standards.

This feature is priced based on the number of checks that are performed each month and is available in all AWS commercial and AWS GovCloud Regions.

Learn more
AWS Identity and Access Management (IAM) Access Analyzer

Jeff;

New Amazon WorkSpaces Thin Client provides cost-effective, secure access to virtual desktops

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-workspaces-thin-client/

The new Amazon WorkSpaces Thin Client improves end-user and IT staff productivity with cost-effective, secure, easy-to-manage access to virtual desktops. The devices are preconfigured and shipped directly to the end user, ready to deploy, connect, and use.

Here’s my testing setup:

The Thin Client is a small cube that connects directly to a monitor, keyboard, mouse, and other USB peripherals such as headsets, microphones, and cameras. With the optional hub it can also drive a second monitor. The administrator can create environments that give users access to Amazon WorkSpaces, Amazon WorkSpaces Web, or Amazon AppStream 2.0, with multiple options for managing user identities and credentials using Active Directory.

Thin Clients in action
As a very long-time user of Amazon WorkSpaces via a thin client, I am thrilled to be able to tell you about this device and the administrative service behind it. While my priority is the ease with which I can switch from client to client while maintaining my working context (running apps, browser tabs, and so forth), administrators will find it attractive for other reasons. For example:

Cost – The device itself is low cost ($195 in the United States), far less expensive than a laptop and the associated operating system. Because the working environments are centrally configured and administered, there’s less work to be done in the field, leading to further cost savings. Further, the devices are far simpler than laptops, with less parts to break, wear out, or replace.

Security – The devices are shipped with a secure “secret” that is used to establish a trust relationship with the administrative service. There’s no data storage on the device, and it cannot host rogue apps that could attempt to exfiltrate data. It also helps to reduce risk of data leakage should a worker leave their job without returning their employer-supplied laptop.

Ease of Management – Administrators can easily create new environments for users or groups of users, distribute activation codes to them, and manage the environment via the AWS Management Console. They can set schedules for software updates and patches, verify compliance, and manage users over time.

Ease of Use – Users can unpack and connect the devices in minutes, enter their activation codes, log in to their virtual desktop environment, and start to work right away. They don’t have to take responsibility for installing software patches or updates, and can focus on their jobs.

There are lots of great use cases for these devices! First, there are situations where there’s a long-term need for regular access: call centers, task workers, training centers, and so forth. Second, there are other situations, where there’s a transient or short-term need for access: registration systems at large events, call centers stood up on a temporary basis for a special event or an emergency, disaster response, and the like. Given that some employees do not return laptops to their employers when they leave their job, providing them with inexpensive devices that do not have local storage makes a lot of sense.

Let’s walk through the process of getting set up, first as an administrator and then as a user.

Getting started as an administrator
The first step is to order some devices and have them shipped to my users, along with any desired peripherals.

Next, in my position as administrator, I open the Amazon WorkSpaces Thin Client Console, and click Get started:

Each Amazon WorkSpaces Thin Client environment provides access to a specific virtual desktop service (WorkSpaces, WorkSpaces Web, or Amazon AppStream 2.0). I click Create environment to move ahead:

I give my environment a name, indicate that I want patches applied automatically, and select WorkSpaces Web:

Next, I click Create WorkSpaces Web portal and go through those steps (not shown, basically choosing a VPC and two or more subnets, a security group, and an optional Private Certificate Authority):

I refresh, and my new portal is visible. I select it, enter a tag for tracking, and click Create environment:

My environment is ready right away. I keep a copy of the activation code (aci3a5yj) at hand to use later in the post:

I am using AWS Identity Center as my identity provider. I already set up my first user, and assigned myself to the MyWebPortal app (the precise steps that you take to do this will vary depending on your choice of identity provider):

Finally, as my last step in this process in my role as administrator, I share the activation code with my users (that would be me, in this case).

Getting started as a user
In my role as a user I return to my testing setup, power-on, go through a couple of quick steps to select my keyboard and connect to my home Wi-Fi, and enter my activation code:

Then I sign in using my AWS Identity Center user name and password:

And my WorkSpace is ready to use:

Administrator tools
As an administrator, I can manage environments, devices, and device software updates from the Thin Client Console. For example, I can review the list of devices that I manage:

Things to know
Here are a couple of things that are important to know:

Regions – The Thin Client Console is available in the US East (N. Virginia), US West (Oregon), Asia Pacific (Mumbai), Canada (Central), and Europe (Frankfurt, Ireland, Ireland, London) Regions.

Device Sales – The Amazon WorkSpaces Thin Clients are available in the United States now, with availability in other countries in early 2024.

Pricing – Devices are priced at $195, or $280 with an optional hub that allows you to use a second monitor. There’s a $6 per month fee to manage, maintain, and monitor each device, and you also pay for the underlying virtual desktop service.

Learn more
Visit the WorkSpaces Thin Client web page and Amazon Business Marketplace to learn more.

Jeff;

Introducing Amazon EC2 high memory U7i Instances for large in-memory databases (preview)

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/introducing-amazon-ec2-high-memory-u7i-instances-for-large-in-memory-databases-preview/

The new U7i instances are designed to support large, in-memory databases including SAP HANA, Oracle, and SQL Server. Powered by custom fourth generation Intel Xeon Scalable Processors (Sapphire Rapids), the instances are now available in multiple AWS regions in preview form, in the US West (Oregon), Asia Pacific (Seoul), and Europe (Frankfurt) AWS Regions, as follows:

Instance Name vCPUs
Memory (DDR5)
EBS Bandwidth
Network Bandwidth
u7in-16tb.224xlarge 896 16,384 GiB 100 Gbps 100 Gbps
u7in-24tb.224xlarge 896 24,576 GiB 100 Gbps 100 Gbps
u7in-32tb.224xlarge 896 32,768 GiB 100 Gbps 100 Gbps

We are also working on a smaller instance:

Instance Name vCPUs
Memory (DDR5)
EBS Bandwidth
Network Bandwidth
u7i-12tb.224xlarge 896 12,288 GiB 60 Gbps 100 Gbps

Here’s what 32 TiB of memory looks like:

And here are the 896 vCPUs (and lots of other info):

When compared to the first generation of High Memory instances, the U7i instances offer up to 125% more compute performance and up to 120% more memory performance. They also provide 2.5x as much EBS bandwidth, giving you the ability to hydrate in-memory databases at a rate of up to 44 terabytes per hour.

Each U7i instance supports attachment of up to 128 General Purpose (gp2 and gp3) or Provisioned IOPS (io1 and io2 Block Express) EBS volumes. Each io2 Block Express volume can be as big as 64 TiB and can deliver up to 256K IOPS at up to 32 Gbps, making them a great match for the U7i instance.

On the network side, the instances support ENA Express and deliver up to 25 Gbps of bandwidth per network flow.

Supported operating systems include Red Hat Enterprise Linux and SUSE Enterprise Linux Server.

Join the Preview
If you are ready to put the U7i instances to the test in your environment, join the preview.

Jeff;

Use anomaly detection with AWS Glue to improve data quality (preview)

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/use-anomaly-detection-with-aws-glue-to-improve-data-quality-preview/

We are launching a preview of a new AWS Glue Data Quality feature that will help to improve your data quality by using machine learning to detect statistical anomalies and unusual patterns. You get deep insights into data quality issues, data quality scores, and recommendations for rules that you can use to continuously monitor for anomalies, all without having to write any code.

Data quality counts
AWS customers already build data integration pipelines to extract and transform data. They set up data quality rules to ensure that the resulting data is of high quality and can be used to make accurate business decisions. In many cases, these rules assess the data based on criteria that were chosen and locked in at a specific point in time, reflecting the current state of the business. However, as the business environment changes and the properties of the data shift, the rules are not always reviewed and updated.

For example, a rule could be set to verify that daily sales are at least ten thousand dollars for an early-stage business. As the business succeeds and grows, the rule should be checked and updated from time to time, but in practice this rarely happens. As a result, if there’s an unexpected drop in sales, the outdated rule does not activate, and no one is happy.

Anomaly detection in action
To detect unusual patterns and to gain deeper insights into data, organizations try to create their own adaptive systems or turn to costly commercial solutions that require specific technical skills and specialized business knowledge.

To address this widespread challenge, Glue Data Quality now makes use of machine learning (ML).

Once activated, this cool new addition to Glue Data Quality gathers statistics as fresh data arrives, using ML and dynamic thresholds to learn from past patterns while looking outliers and unusual data patterns. This process produces observations and also visualizes trends so that you can quickly gain a better understanding of the anomaly.

You will also get rule recommendations as part of the Observations, and you can easily and progressively add them to your data pipelines. Rules can enforce an action such as stopping your data pipelines. In the past, you could only write static rules. Now, you can write Dynamic rules that have auto-adjusting thresholds and AnomalyDetection Rules that grasp recurring patterns and spot deviations. When you use rules as part of data pipelines, they can stop the data flow so that a data engineer can review, fix and resume.

To use anomaly detection, I add an Evaluate Data Quality node to my job:

I select the node and click Add analyzer to choose a statistic and the columns:

Glue Data Quality learns from the data to recognize patterns and then generates observations that will be shown in the Data quality tab:

And a visualization:

After I review the observations I add new rules. The first one sets adaptive thresholds that check the row count is between the smallest of the last 10 runs and the largest of the last 20 runs. The second one looks for unusual patters, for example RowCount being abnormally high on weekends:

Join the preview
This new capability is available in preview in the following AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland). To learn more, read Data Quality Anomaly Detection]].

Stay tuned for a detailed blog post when this feature launches!

Learn more

Data Quality Anomaly Detection

Jeff;