Tag Archives: news

Detect runtime security threats in Amazon ECS and AWS Fargate, new in Amazon GuardDuty

2023-11-27 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/introducing-amazon-guardduty-ecs-runtime-monitoring-including-aws-fargate/

Today, we’re announcing Amazon GuardDuty ECS Runtime Monitoring to help detect potential runtime security issues in Amazon Elastic Container Service (Amazon ECS) clusters running on both AWS Fargate and Amazon Elastic Compute Cloud (Amazon EC2).

GuardDuty combines machine learning (ML), anomaly detection, network monitoring, and malicious file discovery against various AWS data sources. When threats are detected, GuardDuty generates security findings and automatically sends them to AWS Security Hub, Amazon EventBridge, and Amazon Detective. These integrations help centralize monitoring for AWS and partner services, initiate automated responses, and launch security investigations.

GuardDuty ECS Runtime Monitoring helps detect runtime events such as file access, process execution, and network connections that might indicate runtime threats. It checks hundreds of threat vectors and indicators and can produce over 30 different finding types. For example, it can detect attempts of privilege escalation, activity generated by crypto miners or malware, or activity suggesting reconnaissance by an attacker. This is in addition to GuardDuty‘s primary detection categories.

GuardDuty ECS Runtime Monitoring uses a managed and lightweight security agent that adds visibility into individual container runtime behaviors. When using AWS Fargate, there is no need for you to install, configure, manage, or update the agent. We take care of that for you. This simplifies the management of your clusters and reduces the risk of leaving some tasks without monitoring. It also helps to improve your security posture and pass regulatory compliance and certification for runtime threats.

GuardDuty ECS Runtime Monitoring findings are visible directly in the console. You can configure GuardDuty to also send its findings to multiple AWS services or to third-party monitoring systems connected to your security operations center (SOC).

With this launch, Amazon Detective now receives security findings from GuardDuty ECS Runtime Monitoring and includes them in its collection of data for analysis and investigations. Detective helps to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities. It collects log data from AWS resources and uses machine learning, statistical analysis, and graph theory to build a linked set of data that enables you to easily conduct security investigations.

Configure GuardDuty ECS Runtime Monitoring on AWS Fargate
For this demo, I choose to show the experience provided for AWS Fargate. When using Amazon ECS, you must ensure your EC2 instances have the GuardDuty agent installed. You can install the agent manually, bake it into your AMI, or use GuardDuty‘s provided AWS Systems Manager document to install it (go to Systems Manager in the console, select Documents, and then search for GuardDuty). The documentation has more details about installing the agent on EC2 instances.

When operating from a GuardDuty administrator account, I can enable GuardDuty ECS Runtime Monitoring at the organization level to monitor all ECS clusters in all organizations’ AWS accounts.

In this demo, I use the AWS Management Console to enable Runtime Monitoring. Enabling GuardDuty ECS Runtime Monitoring in the console has an effect on all your clusters.

When I want GuardDuty to automatically deploy the GuardDuty ECS Runtime Monitoring agent on Fargate, I enable GuardDuty agent management. To exclude individual clusters from automatic management, I can tag them with GuardDutyManaged=false. I make sure I tag my clusters before enabling ECS Runtime Monitoring in the console. When I don’t want to use the automatic management option, I can leave the option disabled and selectively choose the clusters to monitor with the tag GuardDutyManaged=true.

The Amazon ECS or AWS Fargate cluster administrator must have authorization to manage tags on the clusters.

The IAM TaskExecutionRole you attach to tasks must have permissions to download the GuardDuty agent from a private ECR repository. This is done automatically when you use the AmazonECSTaskExecutionRolePolicy managed IAM policy.

Here is my view of the console when the Runtime Monitoring and agent management are enabled.

I can track the deployment of the security agent by assessing the Coverage statistics across all the ECS clusters.

Once monitoring is enabled, there is nothing else to do. Let’s see what findings it detects on my simple demo cluster.

Check out GuardDuty ECS runtime security findings
When GuardDuty ECS Runtime Monitoring detects potential threats, they appear in a list like this one.

I select a specific finding to view more details about it.

Things to know
By default, a Fargate task is immutable. GuardDuty won’t deploy the agent to monitor containers on existing tasks. If you want to monitor containers for already running tasks, you must stop and start the tasks after enabling GuardDuty ECS Runtime Monitoring. Similarly, when using Amazon ECS services, you must force a new deployment to ensure tasks are restarted with the agent. As I mentioned already, be sure the tasks have IAM permissions to download the GuardDuty monitoring agent from Amazon ECR.

We designed the GuardDuty agent to have little impact on performance, but you should plan for it in your Fargate task sizing calculations.

When you choose automatic agent management, GuardDuty also creates a VPC endpoint to allow the agent to communicate with GuardDuty APIs. When—just like me—you create your cluster with a CDK or CloudFormation script with the intention to delete the cluster after a period of time (for example, in a continuous integration scenario), bear in mind that the VPC endpoint must be deleted manually to allow CloudFormation to delete your stack.

Pricing and availability
You can now use GuardDuty ECS Runtime Monitoring on AWS Fargate and Amazon EC2 instances. For a full list of Regions where GuardDuty ECS Runtime Monitoring is available, visit our Region-specific feature availability page.

You can try GuardDuty ECS Runtime Monitoring for free for 30 days. When you enable GuardDuty for the first time, you have to explicitly enable GuardDuty ECS Runtime Monitoring. At the end of the trial period, we charge you per vCPU per hour of the monitoring agents. The GuardDuty pricing page has all the details.

Get insights about the threats to your container and enable GuardDuty ECS Runtime Monitoring today.

— seb

Introducing Amazon EC2 high memory U7i Instances for large in-memory databases (preview)

2023-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/introducing-amazon-ec2-high-memory-u7i-instances-for-large-in-memory-databases-preview/

The new U7i instances are designed to support large, in-memory databases including SAP HANA, Oracle, and SQL Server. Powered by custom fourth generation Intel Xeon Scalable Processors (Sapphire Rapids), the instances are now available in multiple AWS regions in preview form, in the US West (Oregon), Asia Pacific (Seoul), and Europe (Frankfurt) AWS Regions, as follows:

Instance Name	vCPUs	Memory (DDR5)	EBS Bandwidth	Network Bandwidth
u7in-16tb.224xlarge	896	16,384 GiB	100 Gbps	100 Gbps
u7in-24tb.224xlarge	896	24,576 GiB	100 Gbps	100 Gbps
u7in-32tb.224xlarge	896	32,768 GiB	100 Gbps	100 Gbps

We are also working on a smaller instance:

Instance Name	vCPUs	Memory (DDR5)	EBS Bandwidth	Network Bandwidth
u7i-12tb.224xlarge	896	12,288 GiB	60 Gbps	100 Gbps

Here’s what 32 TiB of memory looks like:

And here are the 896 vCPUs (and lots of other info):

When compared to the first generation of High Memory instances, the U7i instances offer up to 125% more compute performance and up to 120% more memory performance. They also provide 2.5x as much EBS bandwidth, giving you the ability to hydrate in-memory databases at a rate of up to 44 terabytes per hour.

Each U7i instance supports attachment of up to 128 General Purpose (gp2 and gp3) or Provisioned IOPS (io1 and io2 Block Express) EBS volumes. Each io2 Block Express volume can be as big as 64 TiB and can deliver up to 256K IOPS at up to 32 Gbps, making them a great match for the U7i instance.

On the network side, the instances support ENA Express and deliver up to 25 Gbps of bandwidth per network flow.

Supported operating systems include Red Hat Enterprise Linux and SUSE Enterprise Linux Server.

Join the Preview
If you are ready to put the U7i instances to the test in your environment, join the preview.

— Jeff;

Amazon Detective adds new capabilities to accelerate and improve your cloud security investigations

2023-11-27 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/amazon-detective-adds-investigations-and-finding-group-summaries-to-help-you-investigate-security-findings/

Today, Amazon Detective adds four new capabilities to help you save time and strengthen your security operations.

First, Detective investigations for IAM help security analysts investigate AWS Identity and Access Management (IAM) objects, such as users and roles, for indicators of compromise (IoCs) to determine potential involvement in known tactics from the MITRE ATT&CK framework. These automatic investigations are available in the Detective section of the AWS Management Console and through a new API to automate your analysis or incident response or to send these findings to other systems, such as AWS Security Hub or your SIEM.

Second, Detective finding group summaries uses generative artificial intelligence (AI) to enrich its investigations. It automatically analyzes finding groups and provides insights in natural language to accelerate security investigations. It provides a plain language title based on the analysis of the finding group with relevant summarized insights, such as describing the activity that initiated the event and its impact, if any. Finding group summaries handles the heavy lifting of analyzing the finding group built across multiple AWS data sources, making it easier and faster to investigate unusual or suspicious activity.

In addition to these two new capabilities that I describe in this post, Detective adds another two capabilities not covered here:

Detective now supports security investigations for threats detected by Amazon GuardDuty ECS Runtime Monitoring.
Detective now integrates with Amazon Security Lake, enabling security analysts to query and retrieve logs stored in Security Lake.

Amazon Detective makes it easier to analyze, investigate, and quickly identify the root cause of security findings or suspicious activities. Detective uses machine learning (ML), statistical analysis, and graph theory to help you visualize and conduct faster and more efficient security investigations. Detective automatically collects logs data and events from sources like AWS CloudTrail logs, Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, Amazon GuardDuty findings, Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, and AWS security findings. Detective maintains up to a year of aggregated data for analysis and investigations.

Cloud security professionals often find threat hunting and incident investigations to be resource-intensive and time-consuming. They must manually gather and analyze data from various sources to identify potential IAM-related threats. IAM investigations are particularly challenging due to dynamic cloud permissions and credentials. Analysts need to piece together data from different systems, including audit logs, entitlement reports, and CloudTrail events, which can be dispersed. Cloud permissions are often granted on-demand or through automation scripts, making authorization changes hard to track. Reconstructing activity timelines and identifying irregular entitlements can take hours or days, depending on complexity. Limited visibility into legacy systems and incomplete logs further complicates IAM investigations, making it difficult to obtain a definitive understanding of unauthorized access.

Detective investigations for IAM triage findings and surface only the most critical, suspicious issues, allowing security analysts to focus on high-level investigations. It automatically analyzes resources in your AWS environment to identify potential indicators of compromise or suspicious activity using machine learning and threat intelligence. This allows analysts to identify patterns and comprehend which resources are impacted by security events, offering a proactive approach to threat identification and mitigation.

The investigations are not only available in the console; you can use the new StartInvestigation API to automate a remediation workflow or collect information about all IP involved or AWS resources compromised. You can also use the API to feed the data to other systems to build a consolidated view of your security posture.

Finding group summaries evaluates the connections between security events across an environment and provides insights in natural language that link related threats, compromised resources, and malicious actor behavior. This narrative offers security analysts a comprehensive overview of security incidents that goes beyond individual service reports. By grouping and contextualizing data from multiple sources, finding group summaries identifies threats that might go unnoticed when insights are isolated. This approach improve the speed and efficiency of investigations and responses. Security analysts can utilize finding group summaries to gain a holistic understanding of security events and their interrelationships, helping them make informed decisions regarding containment and remediation.

Let’s see these two capabilities in action
In this demo, I start with Detective investigations for IAM in the Detective section of the console. The Detective dashboard shows me the number of investigations done and the number of IAM roles and users involved in suspicious activities.

From there, I drill down the list of investigations.

And I select one specific investigation to get the details. There is a summary first.

I scroll down the page to see what IP addresses are involved and for what type of activities. This example shows me a physical impossibility: the same IP was used in a short time from two different places, Australia and Japan.

The most interesting section of the page, in my opinion, is the mappings to tactics, techniques, and procedures (TTP). All TTPs are classified according to their severity. The console shows the techniques and actions used. When selecting a specific TTP, I can see the details in the right pane. In this example, the suspicious IP address has been involved in more than 2,000 failed attempts to change the trusted policy of an IAM role.

Finally, I navigate to the Indicators tab to see the list of indicators.

On the other side, finding group summaries is available under Finding groups. I select a finding group to receive a natural language explanation of the findings and risks involved.

Pricing and availability
These two new capabilities are now available to all AWS customers.

Detective investigations for IAM is available in all AWS Regions where Detective is available. Finding group summaries is available in five AWS Regions: US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore, Tokyo), and Europe (Frankfurt).

Learn all the details about Amazon Detective and get started today.

— seb

Top announcements of AWS re:Invent 2023

2023-11-27 AWS Editorial Team

Post Syndicated from AWS Editorial Team original https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2023/

Welcome to Las Vegas, where excitement fills the air for the premier AWS event of the year – re:Invent 2023! Happening Nov. 27 – Dec. 1, re:Invent 2023 offers keynotes, training, Innovation Talks, AWS Builder Labs, and much more to inspire you on your cloud journey.

Can’t make it in person? Tune in to the live streams of the keynotes, and check back here as we provide daily updates on the most exciting product launches.

If you don’t want to miss a thing, here are a few more ways to keep in touch:

(This post was last updated: 5:05 p.m. PST, Nov. 26, 2023.)

Use natural language to query Amazon CloudWatch logs and metrics (preview)
To make it easy to interact with your operational data, Amazon CloudWatch is introducing natural language query generation for Logs and Metrics Insights.

Increase collaboration and securely share cloud knowledge with AWS re:Post Private
re:Post Private includes content tailored specifically for your organization’s use cases, along with private discussion and collaboration forums for the members of your organization and your AWS account team.

Use anomaly detection with AWS Glue to improve data quality (preview)
This new feature will help to improve your data quality by using machine learning to detect statistical anomalies and unusual patterns.

Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities
With this new feature, you can now offload client authentication to Application Load Balancer, ensuring only trusted clients communicate with backend applications.

Check your AWS Free Tier usage programmatically with a new API
You can use the API directly with the AWS Command Line Interface or integrate it into an application with the AWS SDKs.

Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics
You can now consolidate metrics from your hybrid, multicloud, and on-premises data sources using Amazon CloudWatch and process them in a consistent, unified fashion.

Announcing cross-region data replication for Amazon WorkSpaces
Snapshots are taken every 12 hours, replicated to the desired destination region, and are used to provide a recovery point objective of 12-24 hours.

Amazon Transcribe Call Analytics adds new generative AI-powered call summaries (preview)
Powered by Amazon Bedrock, this feature helps businesses improve customer experience, and agent and supervisor productivity by automatically summarizing customer service calls.

Build generative AI apps using AWS Step Functions and Amazon Bedrock
Step Functions provides two new optimized API actions for Amazon Bedrock: InvokeModel and CreateModelCustomizationJob.

New Cost Optimization Hub centralizes recommended actions to save you money
This new AWS Billing and Cost Management feature makes it easy for you to identify, filter, aggregate, and quantify savings for AWS cost optimization recommendations.

Amazon CloudWatch Logs now offers automated pattern analytics and anomaly detection
Amazon CloudWatch can now automatically recognize and cluster patterns among log records, extract noteworthy content and trends, and notify you of anomalies using advanced machine learning algorithms.

Amazon Managed Service for Prometheus collector provides agentless metric collection for Amazon EKS
This new capability discovers and collects Prometheus metrics from Amazon Elastic Kubernetes Service (Amazon EKS) automatically and without an agent.

Optimize your storage costs for rarely-accessed files with Amazon EFS Archive
We’ve added a new storage class for Amazon Elastic File System optimized for long-lived data that is rarely accessed.

New Amazon CloudWatch log class for infrequent access logs at a reduced price
This new log class offers a tailored set of capabilities at a lower cost for infrequently accessed logs, enabling customers to consolidate all their logs in one place in a cost-effective manner.

Use natural language to query Amazon CloudWatch logs and metrics (preview)

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/use-natural-language-to-query-amazon-cloudwatch-logs-and-metrics-preview/

To make it easy to interact with your operational data, Amazon CloudWatch is introducing today natural language query generation for Logs and Metrics Insights. With this capability, powered by generative artificial intelligence (AI), you can describe in English the insights you are looking for, and a Logs or Metrics Insights query will be automatically generated.

This feature provides three main capabilities for CloudWatch Logs and Metrics Insights:

Generate new queries from a description or a question to help you get started easily.
Query explanation to help you learn the language including more advanced features.
Refine existing queries using guided iterations.

Let’s see how these work in practice with a few examples. I’ll cover logs first and then metrics.

Generate CloudWatch Logs Insights queries with natural language
In the CloudWatch console, I select Log Insights in the Logs section. I then select the log group of an AWS Lambda function that I want to investigate.

I choose the Query generator button to open a new Prompt field where I enter what I need using natural language:

Tell me the duration of the 10 slowest invocations

Then, I choose Generate new query. The following Log Insights query is automatically generated:

fields @timestamp, @requestId, @message, @logStream, @duration 
| filter @type = "REPORT" and @duration > 1000
| sort @duration desc
| limit 10

I choose Run query to see the results.

I find that now there’s too much information in the output. I prefer to see only the data I need, so I enter the following sentence in the Prompt and choose Update query.

Show only timestamps and latency

The query is updated based on my input and only the timestamp and duration are returned:

fields @timestamp, @duration 
| filter @type = "REPORT" and @duration > 1000
| sort @duration desc
| limit 10

I run the updated query and get a result that is easier for me to read.

Now, I want to know if there are any errors in the log. I enter this sentence in the Prompt and generate a new query:

Count the number of ERROR messages

As requested, the generated query is counting the messages that contain the ERROR string:

fields @message
| filter @message like /ERROR/
| stats count()

I run the query and find out that there are more errors than I expected. I need more information.

I use this prompt to update the query and get a better distribution of the errors:

Show the errors per hour

The updated query uses the bin() function to group the result in one hour intervals.

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) by bin(1h)

Let’s see a more advanced query about memory usage. I select the log groups of a few Lambda functions and type:

Show invocations with the most over-provisioned memory grouped by log stream

Before generating the query, I choose the gear icon to toggle the options to include my prompt and an explanation as comment. Here’s the result (I split the explanation over multiple lines for readability):

# Show invocations with the most over-provisioned memory grouped by log stream

fields @logStream, @memorySize/1000/1000 as memoryMB, @maxMemoryUsed/1000/1000 as maxMemoryUsedMB, (@memorySize/1000/1000 - @maxMemoryUsed/1000/1000) as overProvisionedMB 
| stats max(overProvisionedMB) as maxOverProvisionedMB by @logStream 
| sort maxOverProvisionedMB desc

# This query finds the amount of over-provisioned memory for each log stream by
# calculating the difference between the provisioned and maximum memory used.
# It then groups the results by log stream and calculates the maximum
# over-provisioned memory for each log stream. Finally, it sorts the results
# in descending order by the maximum over-provisioned memory to show
# the log streams with the most over-provisioned memory.

Now, I have the information I need to understand these errors. On the other side, I also have EC2 workloads. How are those instances running? Let’s look at some metrics.

Generate CloudWatch Metrics Insights queries with natural language
In the CloudWatch console, I select All metrics in the Metrics section. Then, in the Query tab, I use the Editor. If you prefer, the Query generator is available also in the Builder.

I choose Query generator like before. Then, I enter what I need using plain English:

Which 10 EC2 instances have the highest CPU utilization?

I choose Generate new query and get a result using the Metrics Insights syntax.

SELECT AVG("CPUUtilization")
FROM SCHEMA("AWS/EC2", InstanceId)
GROUP BY InstanceId
ORDER BY AVG() DESC
LIMIT 10

To see the graph, I choose Run.

Well, it looks like my EC2 instances are not doing much. This result shows how those instances are using the CPU, but what about storage? I enter this in the prompt and choose Update query:

How about the most EBS writes?

The updated query replaces the average CPU utilization with the sum of bytes written to all EBS volumes attached to the instance. It keeps the limit to only show the top 10 results.

SELECT SUM("EBSWriteBytes")
FROM SCHEMA("AWS/EC2", InstanceId)
GROUP BY InstanceId
ORDER BY SUM() DESC
LIMIT 10

I run the query and, by looking at the result, I have a better understanding of how storage is being used by my EC2 instances.

Try entering some requests and run the generated queries over your logs and metrics to see how this works with your data.

Things to know
Amazon CloudWatch natural language query generation for logs and metrics is available in preview in the US East (N. Virginia) and US West (Oregon) AWS Regions.

There is no additional cost for using natural language query generation during the preview. You only pay for the cost of running the queries according to CloudWatch pricing.

Generated queries are produced by generative AI and dependent on factors including the data selected and available in your account. For these reasons, your results may vary.

When generating a query, you can include your original request and an explanation of the query as comments. To do so, choose the gear icon in the bottom right corner of the query edit window and toggle those options.

This new capability can help you generate and update queries for logs and metrics, saving you time and effort. This approach allows engineering teams to scale their operations without worrying about specific data knowledge or query expertise.

Use natural language to analyze your logs and metrics with Amazon CloudWatch.

— Danilo

Increase collaboration and securely share cloud knowledge with AWS re:Post Private

2023-11-26 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/increase-collaboration-and-securely-share-cloud-knowledge-with-aws-repost-private/

Today we’re launching AWS re:Post Private, a fully managed knowledge service to accelerate cloud adoption, improve productivity, and drive innovation. re:Post Private allows organizations to increase collaboration and access knowledge resources built for your cloud community. It includes curated collections of technical content and training materials from AWS. The content is tailored specifically for your organization’s use cases, along with private discussion and collaboration forums for the members of your organization and your AWS account team.

As its name implies, you can think of it as a private version of AWS re:Post, with private content and access limited to people that belong to your organization and your AWS Account team.

Organizations of all sizes and verticals are increasingly moving their operations to the cloud. To ensure cloud adoption success, organizations must have the right skills and structure in place. The optimal way to achieve this is by setting up a centralized cloud center of excellence (CCOE). A CCOE is a centralized governance function for the organization and acts in a consultative role for central IT, business-unit IT, and cloud service consumers in the business. According to Gartner, a CCOE has three pillars: governance, brokerage, and community. The community pillar establishes the cloud community of practice (COP) that brings together stakeholders and facilitates cloud collaboration. It helps organizations adapt themselves for cloud adoption by promoting COP member interaction and facilitating cloud-related training and skills development.

AWS re:Post Private facilitates the creation, structure, and management of an internal cloud community of practice. It allows you to build a custom knowledge base that is searchable, reusable, and scalable. It allows community members to post private questions and answers and publish articles. It combines the benefits of traditional forums, such as community discussion and collaboration, with the benefits of an integrated information experience.

AWS re:Post Private is a fully managed service: there is no need to operate complex knowledge management and collaboration technologies or to develop custom solutions.

AWS re:Post Private also facilitates your interactions with AWS Support. You can create a support case directly from your private re:Post, and you can convert case resolution to reusable knowledge visible to all in your organization.

You choose in which AWS Region re:Post Private stores your data and who has access. All data at rest and in transit is encrypted using industry-standard algorithms. Your administrator chooses between using AWS-managed encryption keys or keys you manage and control.

Your organization’s Technical Account Managers are automatically added to your private re:Post. You can select other persons to invite among your organization and AWS teams, such as your AWS Solutions Architect. Only your private re:Post administrators need an AWS account. All other users can federate from your organization’s identity provider, such as Microsoft Active Directory.

Let’s see how to create a re:Post Private
To get started with AWS re:Post Private, as an administrator, I point my browser to the re:Post section of the AWS Management Console. I select Create private re:Post and enter the information needed to create a private re:Post for my organization, my team, or my project.

I can choose the Data encryption parameters and whether or not I enable Service access for Support case integration. When I’m ready, I select Create this re:Post.

Once the private re:Post is created, I can grant access to users and groups. User and group information comes from AWS IAM Identity Center and your identity provider. Invited users receive an email inviting them to connect to the private re:Post and create their profile.

That’s pretty much it for the administrator part. Once the private re:Post is created, I receive an endpoint name that I can share with the rest of my organization.

Let’s see how to use re:Post Private
As a member of the organization, I navigate to re:Post Private using the link I received from the administrator. I authenticate with the usual identity service of my organization, and I am redirected to the re:Post Private landing page.

On the top menu, I can select a tab to view the contents for Questions, Community Articles, Selections, Tags, Topics, Community Groups, or My Dashboard. This should be familiar if you already use the public knowledge service AWS re:Post that adopted a similar structure.

Further down on the page, I see the popular topics and the top contributors in my organization.I also have access to Questions and Community Groups. I can search the available content by keyword, tags, author, and so on.

Pricing and availability
You can create your organization’s AWS re:Post Private in the following AWS Regions: US West (Oregon) and Europe (Frankfurt).

AWS re:Post Private is available to customers having an AWS Enterprise or Enterprise On-Ramp support plan. re:Post Private offers a free tier that allows you to explore and try out standard capabilities for six months. There is no limit on the number of users in the free tier, and content storage is limited to 10 GB. When you reach the free storage limit, the plan is converted to the paid standard tier.

With AWS re:Post Private Standard tier, you only pay for what you use. We charge based on the number of users per month. Please visit the re:Post Private pricing page for more information.

Get started today and activate AWS re:Post Private for your organization.

— seb

Use anomaly detection with AWS Glue to improve data quality (preview)

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/use-anomaly-detection-with-aws-glue-to-improve-data-quality-preview/

We are launching a preview of a new AWS Glue Data Quality feature that will help to improve your data quality by using machine learning to detect statistical anomalies and unusual patterns. You get deep insights into data quality issues, data quality scores, and recommendations for rules that you can use to continuously monitor for anomalies, all without having to write any code.

Data quality counts
AWS customers already build data integration pipelines to extract and transform data. They set up data quality rules to ensure that the resulting data is of high quality and can be used to make accurate business decisions. In many cases, these rules assess the data based on criteria that were chosen and locked in at a specific point in time, reflecting the current state of the business. However, as the business environment changes and the properties of the data shift, the rules are not always reviewed and updated.

For example, a rule could be set to verify that daily sales are at least ten thousand dollars for an early-stage business. As the business succeeds and grows, the rule should be checked and updated from time to time, but in practice this rarely happens. As a result, if there’s an unexpected drop in sales, the outdated rule does not activate, and no one is happy.

Anomaly detection in action
To detect unusual patterns and to gain deeper insights into data, organizations try to create their own adaptive systems or turn to costly commercial solutions that require specific technical skills and specialized business knowledge.

To address this widespread challenge, Glue Data Quality now makes use of machine learning (ML).

Once activated, this cool new addition to Glue Data Quality gathers statistics as fresh data arrives, using ML and dynamic thresholds to learn from past patterns while looking outliers and unusual data patterns. This process produces observations and also visualizes trends so that you can quickly gain a better understanding of the anomaly.

You will also get rule recommendations as part of the Observations, and you can easily and progressively add them to your data pipelines. Rules can enforce an action such as stopping your data pipelines. In the past, you could only write static rules. Now, you can write Dynamic rules that have auto-adjusting thresholds and AnomalyDetection Rules that grasp recurring patterns and spot deviations. When you use rules as part of data pipelines, they can stop the data flow so that a data engineer can review, fix and resume.

To use anomaly detection, I add an Evaluate Data Quality node to my job:

I select the node and click Add analyzer to choose a statistic and the columns:

Glue Data Quality learns from the data to recognize patterns and then generates observations that will be shown in the Data quality tab:

And a visualization:

After I review the observations I add new rules. The first one sets adaptive thresholds that check the row count is between the smallest of the last 10 runs and the largest of the last 20 runs. The second one looks for unusual patters, for example RowCount being abnormally high on weekends:

Join the preview
This new capability is available in preview in the following AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland). To learn more, read Data Quality Anomaly Detection]].

Stay tuned for a detailed blog post when this feature launches!

Learn more

Data Quality Anomaly Detection

— Jeff;

Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities

2023-11-26 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/mutual-authentication-for-application-load-balancer-to-reliably-verify-certificate-based-client-identities/

Today, we are announcing support for mutually authenticating clients that present X509 certificates to Application Load Balancer. With this new feature, you can now offload client authentication to the load balancer, ensuring only trusted clients communicate with their backend applications. This new capability is built on S2N, AWS’s open source Transport Layer Security (TLS) implementation that provides strong encryption and protections against zero-day vulnerabilities, which developers can trust.

Mutual authentication (mTLS) is commonly used for business-to-business (B2B) applications such as online banking, automobile, or gaming devices to authenticate devices using digital certificates. Companies typically use it with a private certificate authority (CA) to authenticate their clients before granting access to data and services.

Customers have implemented mutual authentication using self-created or third-party solutions that require additional time and management overhead. These customers spend their engineering resources to build the functionality into their backend, update their code to keep up with the latest security patches, and invest heavily in infrastructure to create and rotate certificates.

With mutual authentication on Application Load Balancer, you have a fully managed, scalable, and cost-effective solution that enables you to use your developer resources to focus on other critical projects. Your ALB will authenticate clients with revocation checks and pass client certificate information to the target, which can be used for authorization by applications.

Getting started with mutual authentication on ALB
To enable mutual authentication on ALB, choose Create Application Load Balancer by the ALB wizard on Amazon EC2 console. When you select HTTPS in the Listeners and routing section, you can see more settings such as security policy, default server certificate, and a new client certificate handling option to support mutual authentication.

With Mutual authentication (mTLS) enabled, you can configure how listeners handle requests that present client certificates. This includes how your Application Load Balancer authenticates certificates and the amount of certificate metadata that is sent to your backend targets.

Mutual authentication has two options. The Passthrough option sends all the client certificate chains received from the client to your backend application using HTTP headers. The mTLS-enabled Application Load Balancer gets the client certificate in the handshake, establishes a TLS connection, and then sends whatever it gets in HTTPS headers to the target application. The application will need to verify the client certificate chain to authenticate the client.

With the Verify with trust store option, Application Load Balancer and client verify each other’s identity and establish a TLS connection to encrypt communication between them. We introduce a new trust store feature, and you can upload any CA bundle with roots and/or intermediate certificates generated by AWS Private Certificate Authority or any other third party CA as the source of trust to validate your client certificates.

It requires selecting an existing trust store or creating a new one. Trust stores contain your CAs, trusted certificates, and, optionally, certificate revocation lists (CRLs). The load balancer uses a trust store to perform mutual authentication with clients.

To use this option and create a new trust store, choose Trust Stores in the left menu of the Amazon EC2 console and choose Create trust store.

You can choose a CA certificate bundle in PEM format and, optionally, CRLs from your Amazon Simple Storage Service (Amazon S3) bucket. A CA certificate bundle is a group of CA certificates (root or intermediate) used by a trust store. CRLs can be used when a CA revokes client certificates that have been compromised, and you need to reject those revoked certificates. You can replace a CA bundle, and add or remove CRLs from the trust store after creation.

You can use the AWS Command Line Interface (AWS CLI) with new APIs such as create-trust-store to upload CA information, configure the mutual-authentication-mode on the Application Load Balancer listener, and send user certificate information to targets.

$ aws elbv2 create-trust-store --name my-tls-name \
    --ca-certificates-bundle-s3-bucket channy-certs \
    --ca-certificates-bundle-s3-key Certificates.pem \
    --ca-certificates-bundle-s3-object-version <version>
>> arn:aws:elasticloadbalancing:root:file1
$ aws elbv2 create-listener --load balancer-arn <value> \
    --protocol HTTPS \
    --port 443 \
    --mutual-authentication Mode=verify,
      TrustStoreArn=<arn:aws:elasticloadbalancing:root:file1>

If you already have your own private CA, such as AWS Private CA, third-party CA, or self-signed CA, you can upload their CA bundle or CRLs to the Application Load Balancer trust store to enable mutual authentication.

To test the mutual authentication on Application Load Balancer, follow the step-by-step instructions to make a self-signed CA bundle and client certificate using OpenSSL, upload them to the Amazon S3 bucket, and use them with an ELB trust store.

You can use curl with the --key and --cert parameters to send the client certificate as part of the request:

$ curl --key my_client.key --cert my_client.pem https://api.yourdomain.com

Mutual authentication can fail if a client presents an invalid or expired certificate, fails to present a certificate, cannot find a trust chain, or if any links in the trust chain have expired, or the certificate is on the revocation list.

Application Load Balancer will close the connections whenever it fails to authenticate a client and will record new connection logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the client’s IP address, handshake latency, TLS cipher used, and client certificate details. You can use these connection logs to analyze request patterns and troubleshoot issues.

To learn more, see Mutual authentication on Application Load Balancer in the AWS documentation.

Now available
Mutual authentication on Application Load Balancer is now available in all commercial AWS Regions where Application Load Balancer is available, except China. With no upfront costs or commitments required, you only pay for what you use. To learn more, see the Elastic Load Balancing pricing page.

Give it a try now and send feedback to AWS re:Post for Amazon EC2 or through your usual AWS Support contacts.

Learn more:
Application Load Balancer product page

— Channy

Check your AWS Free Tier usage programmatically with a new API

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/check-your-aws-free-tier-usage-programmatically-with-a-new-api/

Starting today, you can check your usage of the AWS Free Tier using the new AWS Free Tier API. You can use the API directly with the AWS Command Line Interface (AWS CLI) or integrate it into an application with the AWS SDKs.

The AWS Free Tier program provides you with the ability to explore and try out AWS services free of charge up to specified limits for each service. The AWS Free Tier includes three different types of offerings:

Always free offers allow customers to use a service for free up to specified limits as long as they are an AWS customer.
12 months free offers allow customers to use a service for free up to specified limits for one year from the date the account was activated.
Short-term trials are free to use for a specified period or up to a one-time limit, depending on the service.

Once you begin turning on AWS resources and interacting with AWS services that offer a free tier, you need to keep track of your progress toward the free tier limit so that you know when to expect to switch to pay-as-you-go pricing.

There are a few ways you can keep track of your AWS Free Tier usage:

The usage alerts in the Billing preferences of the AWS Billing and Cost Management console are enabled by default (unless the account was created via AWS Organizations) and send you emails when you exceed 85 percent of the Free Tier limit for each service.
You can create a zero-spend or a monthly-cost budget in the Budgets section of the Billing and Cost Management console. Using templates, it just requires a couple of clicks and the email address to notify.
The Free Tier page in the Billing and Cost Management console tells you the service, the type of offer, the current usage, and the forecasted usage for each offer in the current billing period.
The new GetFreeTierUsage API provides the same information in the Free Tier page with a structured format that you can use programmatically.

Let’s see how this new API works in practice.

Using the AWS Free Tier API with the AWS CLI
I got access to a new account created in the last few months. Here, I use the AWS Command Line Interface (AWS CLI) to call the GetFreeTierUsage API.

aws freetier get-free-tier-usage

The response is a JSON document that contains a description of the current usage for each offer that is applicable to this account during this billing period. For simplicity, I only show a few offers here.

{
    "freeTierUsages": [
        {
            "service": "Amazon Simple Queue Service",
            "operation": "",
            "usageType": "Requests",
            "region": "global",
            "actualUsageAmount": 294387.0,
            "forecastedUsageAmount": 679354.6153846154,
            "limit": 1000000.0,
            "unit": "Requests",
            "description": "1000000.0 Requests are always free per month as part of AWS Free Usage Tier (Global-Requests)",
            "freeTierType": "Always Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "",
            "usageType": "EBS:VolumeUsage",
            "region": "global",
            "actualUsageAmount": 9.0,
            "forecastedUsageAmount": 33.0,
            "limit": 30.0,
            "unit": "GB-Mo",
            "description": "30.0 GB-Mo for free for 12 months as part of AWS Free Usage Tier (Global-EBS:VolumeUsage)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances:0002",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 476.0,
            "forecastedUsageAmount": 851.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 225.0,
            "forecastedUsageAmount": 485.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Redshift",
            "operation": "RunComputeNode:0001",
            "usageType": "Node:dc2.large",
            "region": "global",
            "actualUsageAmount": 367.0,
            "forecastedUsageAmount": 735.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free per month during a short-term trial as part of AWS Free Usage Tier (Global-Node:dc2.large)",
            "freeTierType": "Free Trial"
        },
        ...
    ]
}

In the freeTierUsages list, I find some of the most common offers:

Two compute offers for Amazon Elastic Compute Cloud (Amazon EC2).
- The offer with operation RunInstances:0002 is for Windows.
- The offer with operation RunInstances is for Linux.
- The value of the operation property is the same as the platform details and usage operation displayed on the Instances or AMIs pages in the Amazon EC2 console. For more information, see the AMI billing information fields in the Amazon EC2 User Guide.
One storage offer for Amazon Elastic Block Store (Amazon EBS) volumes. This and the two Amazon EC2 compute offers have freeTierType equal to 12 Months Free.
An Always Free offer for Amazon Simple Queue Service (Amazon SQS).
A Free Trial (short-term) offer for Amazon Redshift.

Let’s have a look at some properties of these offers:

description gives a readable explanation of what the offer is about.
freeTierType tells the type of offer: Always Free, 12 Months Free, or Free Trial (short-term).
unit describes the unit used to measure usage for the offer. For example, Hrs (hours) for EC2 instances, GB-Mo (GB per month) for EBS volumes, Requests for Amazon SQS, and so on.

Three interesting properties are the limit of the offer (limit), the actual usage amount of the offer (actualUsageAmount), and the forecasted usage amount (forecastedUsageAmount) at the end of the billing period (the current month). They are all based on the unit used by the offer. For example, the Windows and Linux compute offers each have a limit of 750 hours per month. For the storage offer, the limit is 30 GB per month. For Amazon SQS, the limit of the offer is one million requests per month.

Details on the limits and services provided for free are detailed in each card on the AWS Free Tier page and on the pricing page of each service. The actual and forecast usage amounts provided by the AWS Free Tier API are estimated up to three times per day, similar to AWS Cost and Usage Reports.

If the forecast usage is greater than the limit for the offer, I should expect to switch to pay-as-you-go pricing before the end of the billing period if I continue to use the service in the same way. Actual usage is no longer tracked by the GetFreeTierUsage API once the limit is reached. This means that the actual usage amount cannot be greater than its limit. If that’s the case, the corresponding offer is not returned by the API.

For example, I look for the offers for which the forecast is greater than the limit using the --query option of the AWS CLI:

aws freetier get-free-tier-usage --query 'freeTierUsages[?forecastedUsageAmount > limit]'

{
    "freeTierUsages": [
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "",
            "usageType": "EBS:VolumeUsage",
            "region": "global",
            "actualUsageAmount": 9.0,
            "forecastedUsageAmount": 33.0,
            "limit": 30.0,
            "unit": "GB-Mo",
            "description": "30.0 GB-Mo for free for 12 months as part of AWS Free Usage Tier (Global-EBS:VolumeUsage)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances:0002",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 476.0,
            "forecastedUsageAmount": 851.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        }
    ]
}

According to this result, if I want to stay within the Free Tier limits, I can check how I use EBS volumes and Amazon EC2 compute with Windows.

For example, I am currently using 476 hours out of the 750 available in a month for Windows EC2 instances. At this pace, I am forecasted to cross the limit and reach about 851 hours. If I am concerned by the costs, I can switch off my Windows instances when not in use or during the night.

Things to know
Previously, the Free Tier API was not publicly available and was used internally for the Free Tier page in the AWS Billing console, where you can find the same data. We hope that making the GetFreeTierUsage API publicly available can help you have fun with AWS, have better use of the AWS Free Tier offers, and be aware of what is free and what to do when you get close to or over a limit.

Using this information, you can build custom reporting that meets your business needs. For example, if you want to avoid compute costs, you can programmatically stop or hibernate your EC2 instances or set the size of an EC2 Auto Scaling group to zero. You can use any of the AWS SDKs to create a web app or integrate this data in a monitoring solution.

More generally, you can send additional emails or notifications (for example, using Amazon SES or Amazon SNS) when the usage of an offer is close to its limit. This can help you get the maximum benefit of an offer without incurring additional costs. You can also do this with AWS Budgets if you set a usage budget amount to the Free Tier limit.

If an offer is no longer applicable to this account (for example, because it expired at the end of the previous month), the corresponding item is not included in the list. If you save the results from previous invocations of the API, you can compare the list of offers with those reported during the previous billing cycle to see which offers have recently expired.

To learn more about keeping track of your AWS Free Tier usage, we created these three 10-minute courses on AWS Skill Builder, an online learning center where you can learn from AWS experts and build cloud skills online:

— Danilo

Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-use-amazon-cloudwatch-to-consolidate-hybrid-multi-cloud-and-on-premises-metrics/

You can now consolidate metrics from your hybrid, multicloud, and on-premises data sources using Amazon CloudWatch and process them in a consistent, unified fashion. You can query, visualize, and alarm on any and all of the metrics, regardless of their source. In addition to giving you a unified view, this new feature will help you to identify trends and issues that span multiple parts and aspects of your infrastructure.

When I first heard about this new feature, I thought, “Wait, I can do that with PutMetricData, what’s the big deal?” Quite a bit, as it turns out. PutMetricData stores the metrics in CloudWatch, but this cool new feature fetches them on demand, directly from the source.

Instead of storing data, you select and configure connectors that pull data from Amazon Managed Service for Prometheus, generic Prometheus, Amazon OpenSearch Service, Amazon RDS for MySQL, Amazon RDS for PostgreSQL, CSV files stored in Amazon Simple Storage Service (Amazon S3), and Microsoft Azure Monitor. Each connector is a AWS Lambda function that is deployed from a AWS CloudFormation template. CloudWatch invokes the appropriate Lambda functions as needed and makes use of the returned metrics immediately — they are not buffered or kept around.

Creating and using connectors
To get started I open the CloudWatch Console, click All metrics, and activate the Multi source query tab, then I click Create and manage data sources:

And then I do it again:

Then I choose a data source type:

CloudWatch will then prompt me for the details that it needs to create and set up the connector for my data source. For example, if I select Amazon RDS – MySQL, I give my data source a name, choose the RDS database instance, and specify the connection info:

When I click Create data source, a Lambda function, a Lambda Permission, an IAM role, a Secrets Manager Secret, a Log Group, and a AWS CloudFormation Stack will be created in my account:

Then, when I am ready to reference the data source and make use of the metrics that it provides, I enter a SQL query that returns timestamps and values for the metric:

Inside the Lambda function
The code for the Custom – getting started template is short, simple, and easy to understand. It implements handlers for two events:

DescribeGetMetricData – This handler returns a string that includes the name of the connector, default values for the arguments to the other handler, and a text description in Markdown format that is displayed in the custom data source query builder in the CloudWatch console.

GetMetricData – This handler returns a metric name, 1-dimensional array of timestamps and metric values, all for a time range that is provided as arguments to the handler.

If you spend a few minutes examining this code you should be able to see how to write functions to connect to your own data sources.

Things to know
Here are a couple of things to keep in mind about this powerful new feature:

Regions – You can create and use data connectors in all commercial AWS Regions; a connector that is running in one region can connect to and retrieve data from services and endpoints in other regions and other AWS accounts.

Pricing – There is no extra charge for the connectors. You pay for the invocations of the Lambda functions and for any other AWS infrastructure that you create.

— Jeff;

Announcing cross-region data replication for Amazon WorkSpaces

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/cross-region-data-replication-for-amazon-workspaces/

You can now use cross-region data replication to provide business continuity for your Amazon WorkSpaces users. Snapshots are taken every 12 hours, replicated to the desired destination region, and are used to provide a recovery point objective (RPO) of 12-24 hours.

Multi-Region Resilience Review
In her 2022 post Advancing business continuity with Amazon WorkSpaces Multi-Region Resilience, my colleague Ariel introduced you to the initial version of this feature and showed you how to use it to set up standby virtual desktops available for your users. After it has been set up, users log in with Amazon WorkSpaces registration codes that include fully qualified domain names (FQDNs). If the WorkSpaces in the primary region are unavailable, the users are redirected to standby WorkSpaces in the secondary region.

The standby WorkSpaces are available for a small, fixed monthly fee for infrastructure and storage, with a low flat rate change for each hour of usage during the month. Together, this feature and this business model make it easy and economical for you to maintain a standby deployment.

Cross-Region Data Replication
Today we are making this feature even more powerful by adding one-way cross-region data replication. Applications, documents, and other resources stored on the primary WorkSpace are snapshotted every 12 hours and copied to the region hosting the secondary WorkSpace. You get an additional layer of redundancy, enhanced data protection, and can minimize productivity that would otherwise be lost to disruptions. This is particularly helpful if users have installed and configured applications on top of the base image since they won’t have to repeat these steps on the secondary WorkSpace.

Here’s how it all works:

Normal Operation – During normal operation, the users in your WorkSpaces fleet are using the primary region. EBS snapshots of the system (C:) and data (D:) drives are created every 12 hours. Multi-Region Resilience runs in the secondary region and checks for fresh snapshots regularly. When it finds them, it initiates a copy to the secondary region. As the copies arrive in the secondary region they are used to update the secondary WorkSpace.

Failover Detection – As part of the setup process, you will follow Configure your DNS service and setup DNS routing policies to set up DNS routing policies and optional Amazon Route 53 health checks to manage cross-Region redirection.

Failover – If a large-scale event (LSE) affects the primary region and the primary WorkSpace, the failover detection that I just described goes in to effect. When users try to reconnect, they are redirected to the secondary region, the latest snapshots are used to launch a WorkSpace for them, and they are back up and running with access to data and apps that are between 12 and 24 hours old.

Failback – At the conclusion of the LSE, the users manually back up any data that they have created on the secondary WorkSpace and log out of it. Then they log in again, and this time they will be directed to the primary region and WorkSpace, where they can restore their backups and continue to work.

Getting Set Up
As a WorkSpaces administrator, I start by locating the desired primary WorkSpace:

I select it and choose Create Standby WorkSpaces from the Actions menu:

I select the desired region for the secondary WorkSpace and click Next:

Then I choose the right directory in the region, and again click Next:

If the primary WorkSpace is encrypted, I must enter the ARN of the KMS key in the secondary region (or, even better, use a multi-Region key). I check Enable data replication and confirm that I am authorizing an additional monthly charge:

On the next page I review my choices and click Create to initiate the creation of the secondary WorkSpace in the region that I selected.

As mentioned earlier I also need to set up Multi-Region Resilience. This includes setting up a domain name to use as a WorkSpaces registration code, setting up Route 53 health checks, and using them to power routing policies.

Things to Know
Here are a couple of important things to know about Cross-Region Data Replication:

Directories – You can use a self-managed Active Directory, AWS Managed AD or AD Connector configured as described in this post. Simple AD is not supported.

Snapshots – The first-ever EBS snapshot for a particular data volume is full, and subsequent snapshots are incremental. As a result, the first replication for a given WorkSpace will likely take longer than subsequent ones. Snapshots are initiated on a schedule that is internal to WorkSpaces and you cannot control the timing.

Encryption – You can use this feature with Encrypted WorkSpaces as long as you use the same AWS Key Management Service (AWS KMS) keys in the primary and secondary regions. You can also use multi-Region keys.

Bundles – You can use the Windows 10 and Windows 11 bundles, and you can also BYOL.

Accounts – Every AWS account has a fixed limit on the number of pending EBS snapshots. This may affect your ability to use this feature with large fleets of WorkSpaces.

Pricing – You pay a fixed monthly fee based on the amount of storage configured for each primary WorkSpace. See the Amazon WorkSpaces Pricing page for more information.

— Jeff;

New Cost Optimization Hub centralizes recommended actions to save you money

2023-11-26 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/new-cost-optimization-hub-to-find-all-recommended-actions-in-one-place-for-saving-you-money/

Today, we are announcing Cost Optimization Hub, a new AWS Billing and Cost Management feature that makes it easy for you to identify, filter, aggregate, and quantify savings for AWS cost optimization recommendations.

With the new Cost Optimization Hub, you can interactively query cost optimization recommendations such as idle resource detection, resource rightsizing, and purchasing options across multiple AWS Regions and AWS accounts in your organizations without any data aggregation and processing. You can find out how much you’ll save if you implement those recommendations and easily compare and prioritize recommendations by savings.

Andy Jassy, CEO of Amazon, told shareholders, “We’re trying to build customer relationships (and a business) that outlast all of us, and as a result, our AWS sales and support teams are spending much of their time helping customers optimize their AWS spend so they can better weather this uncertain economy” in his 2022 Letter to Shareholders.

Cost Optimization Hub gathers all cost-optimizing recommended actions across AWS Cloud Financial Management (CFM) services, including AWS Cost Explorer and AWS Compute Optimizer, in one place. It incorporates customer-specific pricing and discounts into these recommendations, and it deduplicates findings and savings to give a consolidated view of your cost optimization opportunities.

If you are a FinOps team or infrastructure management team member who wants to understand your cost optimization opportunities in aggregate, such as which AWS accounts or AWS Regions have the most cost optimization opportunities, you should start with Cost Optimization Hub.

You can easily analyze cost optimization opportunities with built-in filters and grouping options. For example, after understanding which AWS account has the most cost optimization opportunities, you can identify the top cost optimization strategies, such as stopping idle resources, rightsizing, and Graviton migration. If you identify which AWS Region has the highest number of rightsizing opportunities, you can get a list of rightsizing recommendations for the Region. It will redirect you to the Compute Optimizer console through deep linking to validate the details, such as the projected CPU utilization if you implement the change.

Getting started with Cost Optimization Hub
To get started, choose Cost Optimization Hub in the left navigation menu of the AWS Billing and Cost Management Console. You can opt in by selecting Enable. There is a 24-hour wait time for Cost Optimization Hub to populate data initially and data will be refreshed daily afterward.

After opt-in, you can see the dashboard of cost optimization recommendations by AWS account, AWS Region, and tag key. If you want to see the list of resources available for optimization, choose View opportunities.

Cost Optimization Hub supports six types of cost-optimizing recommended actions, including:

Stop – Stop idle or unused resources to save up to 100 percent of the resources’ cost.
Rightsize – Move to a smaller Amazon EC2 instance type, Amazon EBS volume, AWS Lambda memory size, or AWS Fargate task size
Upgrade – Move to a later-generation product, such as moving from EBS io1 volume type to io2.
Graviton migration – Move from EC2 instance types with x86-based processors to EC2 instance types with AWS Graviton-based processors to save costs.
Purchase Savings Plans – Purchase Compute Savings Plans, EC2 Instance Savings Plans, and Amazon SageMaker Savings Plans
Purchase Reserved Instances – Purchase Amazon EC2, Amazon RDS, Amazon DynamoDB, Amazon ElastiCache, and Amazon Redshift Reserved Instances.

You can see the resource type, top recommended action, and estimated monthly savings. You can also filter the list by AWS account, AWS Region, implementation effort, and tag key as the group-by dimension.

You also can classify each recommendation as “Is resources restart needed” or “Is rollback possible.” If you specify Is resources restart needed=No as the filter, you can only see recommendations that don’t require you to restart your resources, such as EBS volume recommendations. Similarly, If you specify Is rollback possible=Yes as the filter, you can only see recommendations that can be rolled back.

If you select a specific source, for example, right-sizing EC2 instance, you can view details and connect to Amazon EC2 and the AWS Compute Optimizer console. Note that estimated monthly savings is a quick approximation of future savings. The actual savings you will realize are dependent on your future AWS usage patterns.

You can also interactively query through AWS Command Line Interface (AWS CLI) and AWS SDKs. Here’s a sample query to find the recommendations about deleting and rightsizing resources:

$ aws cost-optimization-hub list-recommendations

The preceding query gives you the following results:

{
   "items":[
      {
         "recommendationId":"MDA2MDI1ODQ1MTA1XzQ5MzNhYzZlLWZmYTUtNGI2ZC04YzBkLTAxYWE3Y2JlNjNlYg==",
         "accountId":"006025845105",
         "region":"Global",
         "resourceId":"006025845105_ComputeSavingsPlans",
         "currentResourceType":"ComputeSavingsPlans",
         "recommendedResourceType":"ComputeSavingsPlans",
         "estimatedMonthlySavings":1506.591472696,
         "estimatedSavingsPercentage":55.46400024,
         "estimatedMonthlyCost":2716.341169146,
         "currencyCode":"USD",
         "implementationEffort":"VeryLow",
         "restartNeeded":false,
         "actionType":"PurchaseSavingsPlans",
         "rollbackPossible":false,
         "recommendedResourceSummary":"$1.628/hour with three years term",
         "lastRefreshTimestamp":"2023-10-23T16:54:13-07:00",
         "recommendationLookbackPeriodInDays":30,
         "source":"CostExplorer"
      },
      {
         "recommendationId":"MDA2MDI1ODQ1MTA1XzhiZTRlNTczLTE0MDctNGIzOS05MmY3LTdmN2EzOTU2Y2ZkYw==",
         "accountId":"006025845105",
         "region":"us-east-1",
         "resourceId":"arn:aws:lambda:us-east-1:006025845105:function:Lambda-recommendation-testing:$LATEST",
         "resourceArn":"arn:aws:lambda:us-east-1:006025845105:function:Lambda-recommendation-testing:$LATEST",
         "currentResourceType":"LambdaFunction",
         "recommendedResourceType":"LambdaFunction",
         "estimatedMonthlySavings":3.1682091425308054e-06,
         "estimatedSavingsPercentage":1.936368871741565,
         "estimatedMonthlyCost":0.00016044778307703665,
         "currencyCode":"USD",
         "implementationEffort":"Low",
         "restartNeeded":false,
         "actionType":"Rightsize",
         "rollbackPossible":true,
         "currentResourceSummary":"128 MB memory",
         "recommendedResourceSummary":"160 MB memory",
         "lastRefreshTimestamp":"2023-10-24T04:07:35.364000-07:00",
         "recommendationLookbackPeriodInDays":14,
         "source":"ComputeOptimizer"
      }
   ]
}

For more information about new Cost Optimization Hub APIs, see the Cost Optimization Hub API documentation.

Now available
Cost Optimization Hub is now generally available for all customers. There is no additional charge for this new capability. You can now get started and view cost optimization recommendations across all AWS Regions.

To learn more, see the Cost Optimization Hub page and send feedback to AWS re:Post for Cost Optimization or through your usual AWS Support contacts.

— Channy

Amazon Managed Service for Prometheus collector provides agentless metric collection for Amazon EKS

2023-11-26 Donnie Prakoso

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/amazon-managed-service-for-prometheus-collector-provides-agentless-metric-collection-for-amazon-eks/

Today, I’m happy to announce a new capability, Amazon Managed Service for Prometheus collector, to automatically and agentlessly discover and collect Prometheus metrics from Amazon Elastic Kubernetes Service (Amazon EKS). Amazon Managed Service for Prometheus collector consists of a scraper that discovers and collects metrics from Amazon EKS applications and infrastructure without needing to run any collectors in-cluster.

This new capability provides fully managed Prometheus-compatible monitoring and alerting with Amazon Managed Service for Prometheus. One of the significant benefits is that the collector is fully managed, automatically right-sized, and scaled for your use case. This means you don’t have to run any compute for collectors to collect the available metrics. This helps you optimize metric collection costs to monitor your applications and infrastructure running on EKS.

With this launch, Amazon Managed Service for Prometheus now supports two major modes of Prometheus metrics collection: AWS managed collection, a fully managed and agentless collector, and customer managed collection.

Getting started with Amazon Managed Service for Prometheus Collector
Let’s take a look at how to use AWS managed collectors to ingest metrics using this new capability into a workspace in Amazon Managed Service for Prometheus. Then, we will evaluate the collected metrics in Amazon Managed Service for Grafana.

When you create a new EKS cluster using the Amazon EKS console, you now have the option to enable AWS managed collector by selecting Send Prometheus metrics to Amazon Managed Service for Prometheus. In the Destination section, you can also create a new workspace or select your existing Amazon Managed Service for Prometheus workspace. You can learn more about how to create a workspace by following the getting started guide.

Then, you have the flexibility to define your scraper configuration using the editor or upload your existing configuration. The scraper configuration controls how you would like the scraper to discover and collect metrics. To see possible values you can configure, please visit the Prometheus Configuration page.

Once you’ve finished the EKS cluster creation, you can go to the Observability tab on your cluster page to see the list of scrapers running in your EKS cluster.

The next step is to configure your EKS cluster to allow the scraper to access metrics. You can find the steps and information on Configuring your Amazon EKS cluster.

Once your EKS cluster is properly configured, the collector will automatically discover metrics from your EKS cluster and nodes. To visualize the metrics, you can use Amazon Managed Grafana integrated with your Prometheus workspace. Visit the Set up Amazon Managed Grafana for use with Amazon Managed Service for Prometheus page to learn more.

The following is a screenshot of metrics ingested by the collectors and visualized in an Amazon Managed Grafana workspace. From here, you can run a simple query to get the metrics that you need.

Using AWS CLI and APIs
Besides using the Amazon EKS console, you can also use the APIs or AWS Command Line Interface (AWS CLI) to add an AWS managed collector. This approach is useful if you want to add an AWS managed collector into an existing EKS cluster or make some modifications to the existing collector configuration.

To create a scraper, you can run the following command:

aws amp create-scraper \ 
       --source eksConfiguration="{clusterArn=<EKS-CLUSTER-ARN>,securityGroupIds=[<SG-SECURITY-GROUP-ID>],subnetIds=[<SUBNET-ID>]}" \ 
       --scrape-configuration configurationBlob=<BASE64-CONFIGURATION-BLOB> \ 
       --destination=ampConfiguration={workspaceArn="<WORKSPACE_ARN>"}

You can get most of the parameter values from the respective AWS console, such as your EKS cluster ARN and your Amazon Managed Service for Prometheus workspace ARN. Other than that, you also need to define the scraper configuration defined as configurationBlob.

Once you’ve defined the scraper configuration, you need to encode the configuration file into base64 encoding before passing the API call. The following is the command that I use in my Linux development machine to encode sample-configuration.yml into base64 and copy it onto the clipboard.

$ base64 sample-configuration.yml | pbcopy

Now Available
The Amazon Managed Service for Prometheus collector capability is now available to all AWS customers in all AWS Regions where Amazon Managed Service for Prometheus is supported.

Learn more:

Amazon Managed Service for Prometheus product page
AWS managed collectors documentation page

Happy building!
— Donnie

Optimize your storage costs for rarely-accessed files with Amazon EFS Archive

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/optimize-your-storage-costs-for-rarely-accessed-files-with-amazon-efs-archive/

Today, we are introducing EFS Archive, a new storage class for Amazon Elastic File System (Amazon EFS) optimized for long-lived data that is rarely accessed.

With this launch, Amazon EFS supports three Regional storage classes:

EFS Standard – Powered by SSD storage and designed to deliver submillisecond latency for active data.
EFS Infrequent Access (EFS IA) – Cost-optimized for data accessed only a few times a quarter, and that doesn’t need the submillisecond latencies of EFS Standard.
EFS Archive – Cost-optimized for long-lived data accessed a few times a year or less and offering similar performance to EFS IA.

All Regional storage classes deliver gigabytes-per-second throughput and hundreds of thousands of IOPS performance and are designed for eleven nines of durability.

You don’t need to manually pick and choose a storage class for your file systems because EFS lifecycle management can automatically migrate files across storage classes based on their access patterns. This allows you to have a single shared file system that contains files processed in very different ways: from active latency-sensitive to cold rarely-accessed data.

Many datasets have subsets of data that are valuable for generating insights but aren’t often used. With EFS Archive, you can store rarely accessed data cost-effectively while keeping it in the same shared file system as other data. This simplified storage approach allows end users and applications to collaborate on large shared datasets in one place, making it easier and quicker to set up and scale analytics workloads.

Using EFS Archive, you can optimize costs for workloads with large file-based datasets that contain a mix of active and inactive data such as user shares, machine learning (ML) training datasets, SaaS applications, and data retained for regulatory compliance like financial transactions and medical records.

Let’s see how this works in practice.

Using EFS Archive storage
To use the new EFS Archive storage class, I need to configure lifecycle management for the file system. In the Amazon EFS console, I select one of my file systems and choose Edit. To use EFS Archive storage, the file system Throughput mode must be Elastic. Elastic Throughput is the recommended choice for most workloads because it is designed to provide applications with as much throughput as they need with pay-as-you-use pricing.

Now, I configure Lifecycle management to transition files into EFS IA or EFS Archive based on my workload’s access patterns.

My workloads rarely use files older than one month. Files older than a quarter are not used by normal activities but need to be kept for a longer time. Based on these considerations, I select to automatically transition files to EFS IA after 30 days and to EFS Archive after 90 days since the last access. These are the default settings for new file systems.

When one of my old files is accessed, it’s usually an indicator that is being used in a new analysis, so it’ll become active again for some period. For this reason, I use the option to transition files back to Standard storage on their first access in IA or Archive storage.

I save changes, and that’s it! This file system will now automatically use different storage classes based on how files are being processed by my applications.

Things to know
EFS Archive is available today in all AWS Regions where Amazon EFS is offered, excluding those based in China.

To offer a more cost-optimized experience for colder, rarely-accessed files, EFS Archive offers 50 percent lower storage cost than EFS IA with a three times higher request charge when data is accessed. For more information, see Amazon EFS pricing.

You can use EFS Archive with existing file systems by configuring the file system lifecycle policies. New file systems are created by default with a lifecycle policy that automatically transitions files to EFS IA after 30 days and to EFS Archive after 90 days since the last access.

Optimize your storage costs by configuring lifecycle management for your Amazon EFS file systems.

— Danilo

New Amazon CloudWatch log class for infrequent access logs at a reduced price

2023-11-26 Marcia Villalba

Post Syndicated from Marcia Villalba original https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-log-class-for-infrequent-access-logs-at-a-reduced-price/

Amazon CloudWatch Logs announces today a new log class called Infrequent Access. This new log class offers a tailored set of capabilities at a lower cost for infrequently accessed logs, enabling customers to consolidate all their logs in one place in a cost-effective manner.

As customers’ applications continue to scale and grow, so does the volume of logs generated. To limit the increase of logging costs, many customers are forced to make hard trade-offs. For example, some customers limit the logs generated by their applications, which can hinder the visibility of the application, or choose a different solution for different log types, which adds complexity and inefficiencies in managing different logging solutions. For instance, customers may send logs needed for real-time analytics and alerting to CloudWatch Logs and send more detailed logs needed for debugging and troubleshooting to a lower-cost solution that doesn’t have as many features as CloudWatch. In the end, these workarounds can impact the observability of the application, because customers need to navigate across multiple solutions to see their logs.

The Infrequent Access log class allows you to build a holistic observability solution using CloudWatch by centralizing all your logs in one place to ingest, query, and store your logs in a cost-efficient way. Infrequent Access is 50 percent lower per GB ingestion price than Standard log class. It provides a tailored set of capabilities for customers that don’t need advanced features like Live Tail, metric extraction, alarming, or data protection that the Standard log class provides. With Infrequent Access, you can still get the benefits of fully managed ingestion, storage, and the ability to deep dive using CloudWatch Logs Insights.

The following table shows a side-by-side comparison of the features that the new Infrequent Access and the Standard log classes have.

Feature	Infrequent Access log class	Standard log class
Fully managed ingestion and storage	Available	Available
Cross-account	Available	Available
Encryption with KMS	Available	Available
Logs Insights	Available	Available
Subscription filters / Export to S3	Not available	Available
GetLogEvents / FilterLogEvents	Not available	Available
Contributor, Container, and Lambda Insights	Not available	Available
Metric filter and alerting	Not available	Available
Data protection	Not available	Available
Embedded metric format (EMF)	Not available	Available
Live Tail	Not available	Available

When to use the new Infrequent Access log class
Use the Infrequent Access log class when you have a new workload that doesn’t require advanced features provided by the Standard log class. One important consideration is that when you create a log group with a specific log class, you cannot change that log group log class afterward.

The Infrequent Access log class is suitable for debug logs or web server logs because they are quite verbose and rarely require any of the advanced functionality that the Standard log class provides.

Another good workload for the Infrequent Access log class is an Internet of Things (IoT) fleet sending detailed logs that are only accessed for after the fact forensic analysis after the event. In addition, the Infrequent Access log class is a good choice for workloads where logs need to be stored for compliance because they will be queried infrequently.

Getting started
To get started using the new Infrequent Access log class, create a new log group in the CloudWatch Logs console and select the new Infrequent Access log class. You can create logs groups with the new Infrequent Access log class not only from the AWS Management Console but also from the AWS Command Line Interface (AWS CLI), AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), and AWS SDKs.

Once you have the new log group created, you can start using it in your workloads. For this example, I will configure a web application to send debug logs to this log group. After a while of the web application executes for a while, you can go back to the log group, where you see a new log stream.

When you select a log stream, you will be directed to CloudWatch Logs Insights.

Using the same familiar CloudWatch Logs Insights experience you get with Standard Class, you can create queries and search those logs to find relevant information, and you can analyze all the logs quickly in one place.

Available now
The new Infrequent Access log class is now available in all AWS Regions except the China and GovCloud Regions. You can start using it and enjoy a more cost-effective way to collect, store, and analyze your logs in a fully managed experience.

To learn more about the new log class, you can check the CloudWatch Logs user guide dedicated page for the Infrequent Access log class.

– Marcia

November 2023 Top500 New Systems Analysis

2023-11-25 Patrick Kennedy

Post Syndicated from Patrick Kennedy original https://www.servethehome.com/november-2023-top500-new-systems-analysis-intel-nvidia-amd-arm-nec-fujitsu/

The November 2023 Top500 new systems showed changes in CPUs, GPUs, and interconnects. It now feels like there are many missing AI systems

The post November 2023 Top500 New Systems Analysis appeared first on ServeTheHome.

Introducing Amazon CloudFront KeyValueStore: A low-latency datastore for CloudFront Functions

2023-11-21 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/introducing-amazon-cloudfront-keyvaluestore-a-low-latency-datastore-for-cloudfront-functions/

Amazon CloudFront allows you to securely deliver static and dynamic content with low latency and high transfer speeds. With CloudFront Functions, you can perform latency-sensitive customizations for millions of requests per second. For example, you can use CloudFront Functions to modify headers, normalize cache keys, rewrite URLs, or authorize requests.

Today, we are introducing CloudFront KeyValueStore, a secure global low-latency key value datastore that allows read access from within CloudFront Functions, enabling advanced customizable logic at the CloudFront edge locations.

Previously, you had to embed configuration data inside the function code. For example, data for determining if a URL should be redirected and which URL to redirect the viewer to. When embedding configuration data with the function code, every small change in configuration requires a code change and a redeployment of the function code. Updating and deploying code for every new lookup addition introduces the risk of making inadvertent changes to code. Also, the maximum function size is 10 KB, making it difficult for many use cases to fit all the data within the code.

With CloudFront KeyValueStore, you can now update the data associated with a function and the function code independently from each other. This simplifies function code and makes it easy to update data without the need to deploy code changes.

Let’s see how this works in practice.

Creating a CloudFront key value store
In the CloudFront console, I choose Functions from the navigation pane. In the KeyValueStores tab, I choose Create KeyValueStore.

Here, I have the option to import key value pairs from a JSON file in an Amazon Simple Storage Service (Amazon S3) bucket. I am not doing that now because I want to start with no keys. I enter a name and description and complete the creation of the key value store.

When the key value store has been created, I choose Edit in the Key value pairs section and then Add pair. I type hello for the key and Hello World for the value and save the changes. I can add more keys and values, but one key is enough for now.

When I update a key value store, changes are propagated to all CloudFront edge locations in a few seconds so that it can be used with low latency by the functions that are associated with the key value store. Let’s see how that works.

Using CloudFront KeyValueStore from CloudFront Functions
In the CloudFront console, I choose Functions in the navigation pane and then Create function. I type a name for the function, select the cloudfront-js-2.0 runtime, and complete the creation of the function. Then, I use the new option to associate the key value store with this function.

I copy the key value store ID from the console to use it in the following function code:

import cf from 'cloudfront';

const kvsId = '<KEY_VALUE_STORE_ID>';

// This fails if the key value store is not associated with the function
const kvsHandle = cf.kvs(kvsId);

async function handler(event) {
    // Use the first part of the pathname as key, for example http(s)://domain/<key>/something/else
    const key = event.request.uri.split('/')[1]
    let value = "Not found" // Default value
    try {
        value = await kvsHandle.get(key);
    } catch (err) {
        console.log(`Kvs key lookup failed for ${key}: ${err}`);
    }
    var response = {
        statusCode: 200,
        statusDescription: 'OK',
        body: {
            encoding: 'text',
            data: `Key: ${key} Value: ${value}\n`
        }
    };
    return response;
}

This function uses the first part of the path of the request as key and responds with the name of the key and its value.

I save the changes and publish the function. In the Publish tab of the function, I associate the function with a CloudFront distribution that I created before. I use the Viewer Request event type and Default (*) cache behavior to intercept all requests to the distribution.

In the console, I go back to the list of functions and wait for the function to be deployed. Then, I use curl from the command line to download content from the distribution and test the result of the function.

First, I try with a couple of paths that invoke the function and look up the key I created before (hello):

curl https://distribution-domain.cloudfront.net/hello
Key: hello Value: Hello World

curl https://distribution-domain.cloudfront.net/hello/world
Key: hello Value: Hello World

It works! Then, I try with a different path to see that the default value I use in the code is returned when the key is not found.

curl https://distribution-domain.cloudfront.net/hi
Key: hi Value: Not found

Now that this first example works, let’s try something more advanced and useful.

Rewriting the URL using configuration data in CloudFront KeyValueStore
Let’s build a function that uses the content of the URL in the HTTP request to look up in a key value store the custom path that CloudFront should use to make the actual request. This function can help manage the multiple services that are part of a website.

For example, I want to update the blog platform I use for my website. The old blog has origin path /blog-v1 while the new blog has origin path /blog-v2.

At first, I am still using the old blog. In the CloudFormation console, I add the blog key to the key value store with value blog-v1.

Then, I create the following function and associate it with the distribution using Viewer Request event and Default (*) cache behavior to intercept all requests to the distribution.

import cf from 'cloudfront';

const kvsId = "<KEY_VALUE_STORE_ID>";

// This fails if the key value store is not associated with the function
const kvsHandle = cf.kvs(kvsId);

async function handler(event) {
    const request = event.request;
    // Use the first segment of the pathname as key
    // For example http(s)://domain/<key>/something/else
    const pathSegments = request.uri.split('/')
    const key = pathSegments[1]
    try {
        // Replace the first path of the pathname with the value of the key
        // For example http(s)://domain/<value>/something/else
        pathSegments[1] = await kvsHandle.get(key);
        const newUri = pathSegments.join('/');
        console.log(`${request.uri} -> ${newUri}`)
        request.uri = newUri;
    } catch (err) {
        // No change to the pathname if the key is not found
        console.log(`${request.uri} | ${err}`);
    }
    return request;
}

Now, when I type blog at the beginning of the URL path, the request will actually go to the blog-v1 path. CloudFront will make the HTTP request to the old blog because blog-v1 is the origin path used by the old blog.

For example, if I type https://distribution-domain.cloudfront.net/blog/index.html in a browser, I see the old blog (V1).

In the console, I update the blog key with value blog-v2. I access the same URL after a few seconds, and now I reach the new blog (V2).

As you can see, the public URL is the same, but the content has changed. More generally, this function assumes that URLs do not change between the two blog versions.

I can now add more keys for the different services that are part of my website (blog, support, help, commerce, and so on) and set their values to use the correct URL path for each of them. When I add a new version for one of them (for example, I migrate to a new commerce platform), I can configure a new origin and update the corresponding key to use the new origin path.

This is just an example of the flexibility you get when you separate configuration data from code. If you are already using CloudFront Functions, you can simplify your code by using CloudFront KeyValueStore.

Things to know
CloudFront KeyValueStore is available today in all edge locations globally. With CloudFront KeyValueStore, you pay only for what you use based on the read/write operations from the public API and the read operations from within CloudFront Functions. For more information, see CloudFront pricing.

You can manage a key value store using the AWS Management Console, AWS Command Line Interface (AWS CLI), and AWS SDKs. AWS CloudFormation support is coming soon. The maximum size of a key value store is 5 MB, and you can associate a single key value store to each function. The maximum size of a key is 512 bytes. Values can be up to 1KB in size. When creating a key value store, you can import key/value data during creation using a source file on Amazon S3 with this JSON structure:

{
  "data":[
    {
      "key":"key1",
      "value":"val1"
    },
    {
      "key":"key2",
      "value":"val2"
    }
  ]
}

Importing key/value data at creation can help automate the setup of a new environment (such as test or dev) and easily replicate the configuration from one environment to another (such as preproduction to production).

Simplify the way you add custom logic at the edge using CloudFront KeyValueStore.

— Danilo

AWS Weekly Roundup – EC2 DL2q instances, PartyRock, Amplify’s 6th birthday, and more – November 20, 2023

2023-11-20 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-ec2-dl2q-instances-partyrock-amplifys-6th-birthday-and-more-november-20-2023/

Last week I saw an astonishing 160+ new service launches. There were so many updates that we decided to publish a weekly roundup again. This continues the same innovative pace of the previous week as we are getting closer to AWS re:Invent 2023.

Our News Blog team is also finalizing new blog posts for re:Invent to introduce awesome launches with service teams for your reading pleasure. Jeff Barr shared The Road to AWS re:Invent 2023 to explain our blogging journey and process. Please stay tuned in the next week!

Last week’s launches
Here are some of the launches that caught my attention last week:

Amazon EC2 DL2q instances – New DL2q instances are powered by Qualcomm AI 100 Standard accelerators and are the first to feature Qualcomm’s AI technology in the public cloud. With eight Qualcomm AI 100 Standard accelerators and 128 GiB of total accelerator memory, you can run popular generative artificial intelligence (AI) applications and extend to edge devices across smartphones, autonomous driving, personal compute, and extended reality headsets to develop and validate these AI workloads before deploying.

PartyRock for Amazon Bedrock – We introduced PartyRock, a fun and intuitive hands-on, generative AI app-building playground powered by Amazon Bedrock. You can experiment, learn all about prompt engineering, build mini-apps, and share them with your friends—all without writing any code or creating an AWS account.

You also can now access the Meta Llama 2 Chat 13B foundation model and Cohere Command Light, Embed English, and multilingual models for Amazon Bedrock.

AWS Amplify celebrates its sixth birthday – We announced six new launches; a new documentation site, support for Next.js 14 with our hosting and JavaScript library, added custom token providers and an automatic React Native social sign-in update to Amplify Auth, new ChangePassword and DeleteUser account settings components, and updated all Amplify UI packages to use new Amplify JavaScript v6. You can also use wildcard subdomains when using a custom domain with your Amplify application deployed to AWS Amplify Hosting.

Amplify docs site UI

Also check out other News Blog posts about major launches published in the past week:

Other AWS service launches
Here are some other bundled feature launches per AWS service:

Amazon Athena – You can use a new cost-based optimizer (CBO) to enhance query performance based on table and column statistics, collected by AWS Glue Data Catalog and Athena JDBC 3.x driver, a new alternative that supports almost all authentication plugins. You can also use Amazon EMR Studio to develop and run interactive queries on Amazon Athena.

Amazon CloudWatch – You can use a new CloudWatch metric called EBS Stalled I/O Check to monitor the health of your Amazon EBS volumes, the regular expression for Amazon CloudWatch Logs Live Tail filter pattern syntax to search and match relevant log events, observability of SAP Sybase ASE database in CloudWatch Application Insights, and up to two stats commands in a Log Insights query to perform aggregations on the results.

Amazon CodeCatalyst – You can connect to a Amazon Virtual Private Cloud (Amazon VPC) from CodeCatalyst Workflows, provision infrastructure using Terraform within CodeCatalyst Workflows, access CodeCatalyst with your workforce identities configured in IAM Identity Center, and create teams made up of members of the CodeCatalyst space.

Amazon Connect – You can use a pre-built queue performance dashboard and Contact Lens conversational analytics dashboard to view and compare real-time and historical aggregated queue performance. You can use quick responses for chats, previously written formats such as typing in ‘/#greet’ to insert a personalized response, and scanning attachments to detect malware or other unwanted content.

AWS Glue – AWS Glue for Apache Spark added new six database connectors: Teradata, SAP HANA, Azure SQL, Azure Cosmos DB, Vertica, and MongoDB, as well as the native connectivity to Amazon OpenSearch Service.

AWS Lambda – You can see single pane view of metrics, logs, and traces in the AWS Lambda console and advanced logging controls to natively capture logs in JSON structured format. You can view the SAM template on the Lambda console and export the function’s configuration to AWS Application Composer. AWS Lambda also supports Java 21 and NodeJS 20 versions built on the new Amazon Linux 2023 runtime.

AWS Local Zones in Dallas – You can enable the new Local Zone in Dallas, Texas, us-east-1-dfw-2a, with Amazon EC2 C6i, M6i, R6i, C6gn, and M6g instances and Amazon EBS volume types gp2, gp3, io1, sc1, and st1. You can also access Amazon ECS, Amazon EKS, Application Load Balancer, and AWS Direct Connect in this new Local Zone to support a broad set of workloads at the edge.

Amazon Managed Streaming for Apache Kafka (Amazon MSK) – You can standardize access control to Kafka resources using AWS Identity and Access Management (IAM) and build Kafka clients for Amazon MSK Serverless written in all programming languages. These are open source client helper libraries and code samples for popular languages, including Java, Python, Go, and JavaScript. Also, Amazon MSK now supports an enhanced version of Apache Kafka 3.6.0 that offers generally available Tiered Storage and automatically sends you storage capacity alerts when you are at risk of exhausting your storage.

Amazon OpenSearch Service Ingestion – You can migrate your data from Elasticsearch version 7.x clusters to the latest versions of Amazon OpenSearch Service and use persistent buffering to protect the durability of incoming data.

Amazon RDS –Amazon RDS for MySQL now supports creating active-active clusters using the Group Replication plugin, upgrading MySQL 5.7 snapshots to MySQL 8.0, and Innovation Release version of MySQL 8.1.

Amazon RDS Custom for SQL Server extends point-in-time recovery support for up to 1,000 databases, supports Service Master Key Retention to use transparent data encryption (TDE), table- and column-level encryption, DBMail and linked servers, and use SQL Server Developer edition with the bring your own media (BYOM).

Additionally, Amazon RDS Multi-AZ deployments with two readable standbys now supports minor version upgrades and system maintenance updates with typically less than one second of downtime when using Amazon RDS Proxy.

AWS Partner Central – You can use an improved user experience in AWS Partner Central to build and promote your offerings and the new Investments tab in the Partner Analytics Dashboard to gain actionable insights. You can now link accounts and associated users between Partner Central and AWS Marketplace and use an enhanced co-sell experience with APN Customer Engagements (ACE) manager.

Amazon QuickSight – You can programmatically manage user access and custom permissions support for roles to restrict QuickSight functionality to the QuickSight account for IAM Identity Center and Active Directory using APIs. You can also use shared restricted folders, a Contributor role and support for data source asset types in folders and the Custom Week Start feature, an addition designed to enhance the data analysis experience for customers across diverse industries and social contexts.

AWS Trusted Advisor – You can use new APIs to programmatically access Trusted Advisor best practices checks, recommendations, and prioritized recommendations and 37 new Amazon RDS checks that provide best practices guidance by analyzing DB instance configuration, usage, and performance data.

There’s a lot more launch news that I haven’t covered. See AWS What’s New for more details.

See you virtually in AWS re:Invent
Next week we’ll hear the latest from AWS, learn from experts, and connect with the global cloud community in Las Vegas. If you come, check out the agenda, session catalog, and attendee guides before your departure.

If you’re not able to attend re:Invent in person this year, we’re offering the option to livestream our Keynotes and Innovation Talks. With the registration for online pass, you will have access to on-demand keynote, Innovation Talks, and selected breakout sessions after the event.

– Channy

New – Long-Form voices for Amazon Polly

2023-11-16 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-long-form-voices-for-amazon-polly/

We are launching three new voices for Polly. Powered by a new long-form engine, the voices are natural and expressive, with appropriate pauses, emphasis, and tone.

New Voices
The new long-form voices are perfect for blog posts, news articles, training videos, and marketing content. The underlying Machine Learning model extracts meaning from the text, learning about speech segments, prosody (the pattern of rhythm and pauses), intonation, and other aspects of expressive speech, allowing the synthesized audio to express emotions, especially in dialogs. The new long-form engine uses a deep learning text-to-speech (TTS) model trained to acquire a contextual understanding of the text that allows it to express prosody in an appropriate way. This allows the intention of the story to drive the vocal performance and create the correct emphasis, pauses, and tones of a realistic human voice.

Here are the new voices:

Name	Locale	Gender	Language	Sample
Danielle	en_US	Female	English (US)
Gregory	en_US	Male	English (US)
Ruth	en_US	Female	English (US)

Using the New Voices
You can access the new voices using the AWS Management Console, AWS Command Line Interface (AWS CLI), or the AWS SDKs. Using the CLI, I start by listing the voices that use the new long-form engine:

$ aws --region us-east-1 polly describe-voices --output json \
  | jq -r '.Voices[] | select(.SupportedEngines | index("long-form")) | .Name'
Danielle
Gregory
Ruth

I can pick one, or I can try all of them:

for v in `aws polly describe-voices --output json \
          | jq -r '.Voices[] | select(.SupportedEngines | index("long-form")) | .Name'`; do
    Text="Hello my name is $v and I can read blog posts, articles, \
and other long-form content for you. I am the best\!"
    aws polly synthesize-speech --output-format 'mp3' \
    --text "$Text" --voice-id $v $v.mp3 --engine long-form; \
    aws s3 cp $v.mp3 s3://jbarr-voices; \
done

My shell script had a small quoting bug, but the resulting audio was too funny not to include!

Programmatically, you can reproduce my example by writing code that calls the DescribeVoices and SynthesizeSpeech functions.

Things to Know
Here are some interesting things that you should know about the new voices:

Pricing – Long-form voices are priced at $100 per million characters or Speech Marks requests. Check out the Amazon Polly Pricing page to learn more.

Engines & Voices – Some of the voices that I listed above can be used with more than one engine. For example, the Danielle voice can be used with the new long-form engine and the existing neural engine.

Regions – The new engine and voices are available in the US East (N. Virginia) Region.

Check out the new voices, build something awesome, and let me know what you think!

— Jeff;

Build AI apps with PartyRock and Amazon Bedrock

2023-11-16 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/build-ai-apps-with-partyrock-and-amazon-bedrock/

If you are ready to learn more about Generative AI while having fun and building cool stuff, check out PartyRock.aws . You can experiment, learn all about prompt engineering, build mini-apps, and share them with your friends — all without writing any code or creating an AWS account. You can also start with an app that has been shared with you and remix it to further enhance and customize it.

Using PartyRock
To get started, I visit https://partyrock.aws/ , click Sign in, and log in using my Apple, Amazon, or Google account:

After authenticating, I am on the front page of PartyRock. I can review some sample apps, or I can click Build your own app to get started:

I can enter a description of the app that I want to build and use PartyRock’s Generative AI to get a running start, or I can built it myself on a widget-by-widget basis:

I am knee-deep in blog posts for AWS re:Invent right now. While most of my colleagues patiently wait for their draft to be ready, a few of them are impatient and keep asking me (this is the adult version of “Are we there yet?”). While I try to maintain a sense of humor about this, I sometimes get just a bit snarky with them. Let’s see if PartyRock can help. I enter my prompt and click Generate App:

My app (Snarky Patient Blogger) is ready within a few seconds, and I enter some input to see if the output has sufficient snark for my needs:

That looks great, so let’s take it apart and see how it works!

The application (Snarky Patient Blogger) has two widgets: User Input and Snarky Response. I click the Edit icon on the first widget, and see that it has a title, loading text, and a default value. The title allows widgets to reference each other by name:

This simple widget encapsulates a call to the Amazon Bedrock InvokeModel function. The widget specifies the use of the Claude v2 model, and a simple prompt, which references the User Input widget. I can experiment by changing either one, saving the change, and waiting a second or two for the result. For example, changing the model to Claude Instant gives me a slightly different response:

Now I want a visual representation of the reply. I’ll use a Text Generation widget to find the most important nouns in the response, and an Image Generation widget to visualize the results. I add the first widget and use a simple prompt:

I test it by clicking the Retry icon and the output looks perfect:

I add the Image Generation widget, and fiddle with the prompt a bit. After a minute or two I have what I want:

The finished app looks like this:

Once I am happy with the app, I can Make public and Share:

The finished app is at https://partyrock.aws/u/jeffbarr/E-FXPUkO7/Snarky-Patient-Blogger and you are welcome to play with it. You can also log in and click Remix to use it as the starting point for an even better app of your own.

But Wait, There’s More
As is usually the case with my blog posts, I did not have room to show you every last feature in detail. Here are a few that I skipped:

Empty App – I used the App Builder in my example, but I can also choose Start from an Empty App, select my widgets, and set them up as desired:

Remix – I can start with an existing app (mine or another public one) and Remix it to customize or enhance it:

Chatbot Widget – I can interact with my app using a prompt as a starting point:

@ Referencing – I can use the “@” to reference other widgets by name while I am building my app:

Advanced Settings – Some of the widgets offer advanced settings. For example, the Text Generation widget gives me the option to control the Temperature and Top P parameters to the model:

Backstage – The PartyRock Backstage lets me see my apps and my cumulative consumption of PartyRock credits:

Things to Know
Here are a couple of things that you should know about PartyRock:

Pricing – For a limited time, AWS offers new PartyRock users a free trial without the need to provide a credit card or sign up for an AWS account, so that you can begin learning fundamental skills without the worry of incurring costs. You can track your credit consumption in the Backstage, as shown above. Credit usage is calculated based on your input tokens, output tokens, and generated images; for full details read the Billing and Support section of the PartyRock FAQ.

Model Access – We plan to give you access to additional models over time.

In the Works – We are working on even more widgets and features, so stay tuned for more information.

Learning Resources – To learn more, check out these resources:

Party Time
The next step is up to you. Log in to PartyRock, create something cool, and share it with everyone you know. Let me know what you come up with!

— Jeff;

Noise

Tag Archives: news

Detect runtime security threats in Amazon ECS and AWS Fargate, new in Amazon GuardDuty

Introducing Amazon EC2 high memory U7i Instances for large in-memory databases (preview)

Amazon Detective adds new capabilities to accelerate and improve your cloud security investigations

Top announcements of AWS re:Invent 2023

Use natural language to query Amazon CloudWatch logs and metrics (preview)

Increase collaboration and securely share cloud knowledge with AWS re:Post Private

Use anomaly detection with AWS Glue to improve data quality (preview)

Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities

Check your AWS Free Tier usage programmatically with a new API

Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics

Announcing cross-region data replication for Amazon WorkSpaces

New Cost Optimization Hub centralizes recommended actions to save you money

Amazon Managed Service for Prometheus collector provides agentless metric collection for Amazon EKS

Optimize your storage costs for rarely-accessed files with Amazon EFS Archive

New Amazon CloudWatch log class for infrequent access logs at a reduced price

November 2023 Top500 New Systems Analysis

Introducing Amazon CloudFront KeyValueStore: A low-latency datastore for CloudFront Functions

AWS Weekly Roundup – EC2 DL2q instances, PartyRock, Amplify’s 6th birthday, and more – November 20, 2023

New – Long-Form voices for Amazon Polly

Build AI apps with PartyRock and Amazon Bedrock

The collective thoughts of the interwebz