Tag Archives: news

Automatic restore testing and validation now available in AWS Backup

2023-11-27 Veliswa Boya

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/automatic-restore-testing-and-validation-is-now-available-in-aws-backup/

Performing automatic game day testing of all your critical resources is an important step in determining that you are prepared to respond to ransomware or any data loss event. This gives you the opportunity to take appropriate corrective actions based on the results and monitor results such as success or failure from these tests. Ultimately, you will be able to ascertain if the restore times meet your expected organization’s recovery time objective (RTO) goals, helping you develop improved recovery strategies.

Today, we’re announcing restore testing, a new capability in AWS Backup that allows you to perform restore testing of your AWS resources across storage, compute, and databases. With this feature, you can automate the entire restore testing process and avoid surprises later by determining now whether you can successfully recover using your backups in the event of a data loss such as ransomware. As an additional option, to demonstrate compliance with your organizational and regulatory data governance requirements, you can use the restore job results.

How it works
Restore testing in AWS Backup supports restore testing of resources for which the recovery points are created by AWS Backup, and the following services are supported: Amazon Elastic Block Store (Amazon EBS), Amazon Elastic Compute Cloud (Amazon EC2), Amazon Aurora, Amazon Relational Database Service (Amazon RDS), Amazon Elastic File Store (Amazon EFS), Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, Amazon FSx, Amazon DocumentDB, and Amazon Neptune. You can get started with restore testing from the AWS Backup console, AWS CLI, or AWS SDK.

Earlier, I created EC2 instances and a backup of these instances. Then, I created my restore testing plan in the AWS Backup console.

In this General section, I enter the name of the plan, a test frequency, a Start time, and a Start within. Start time sets the time for the test to begin, for example, if you have a daily test frequency set, you specify what time the plan will run each day. Start within is the period of time in which the restore test is designated to begin. AWS Backup makes a best effort to commence all designated restore jobs during the Start within time window. You have a choice to keep this very minimal or very large based on your preference.

In the Recovery point selection section, I specify the vaults that the recovery points should come from, and a timeframe of eligible recovery points as part of this restore testing plan. I left the criteria for a recovery point at the default selection. I also didn’t opt to include recovery points generated by point-in-time recovery (PITR) in this restore testing plan.

Tagging is optional so for the purposes of this test I didn’t add a tag. I was then finished with setup, and it was time for me to choose Create restore testing plan to proceed with creating this restore testing plan.

Once the restore testing plan has been created, it is time to assign resources. I start by specifying the IAM role that AWS Backup will assume when running the restore test. In terms of retention period before cleanup, I kept the default selection of deleting the restored resources immediately, to optimize costs. Alternatively, by specifying a retention period I could have also configured to integrate my own tests (for example, AWS Lambda) using Amazon EventBridge (CloudWatch Events) and send back validation status using the new PutRestoreValidationResult API so that it is reported in the restore job.

I have EC2 instances that I created and backed up earlier, and I specify that this plan is for Amazon EC2 resource types. I include all protected resources of this EC2 resource type in the selection scope. I have very few resources, so I didn’t add the optional tags.

I opted to use the default instance type for the restore. I also didn’t specify any additional parameters. It’s then time to choose Assign resources.

Once the resources have been assigned, all information related to the restore testing plan will be presented in a summarized form where you’ll be able to see when the restore testing jobs have executed.

Once I have enough restores performed over time, I can also view the Restore time history for every resource restored from the Protected resources tab.

Now available
Restore testing in AWS Backup is available in all AWS Regions where AWS Backup is available except AWS China Regions, AWS GovCloud (US), and Israel (Tel Aviv).
To learn more, visit the AWS Backup user guide. You can submit your questions to AWS re:Post for AWS Backup or through your usual AWS Support contacts.

— Veliswa

Amazon CodeWhisperer offers new AI-powered code remediation, IaC support, and integration with Visual Studio

2023-11-27 Irshad Buchh

Post Syndicated from Irshad Buchh original https://aws.amazon.com/blogs/aws/amazon-codewhisperer-offers-new-ai-powered-code-remediation-iac-support-and-integration-with-visual-studio/

Today, we’re announcing the general availability of artificial intelligence (AI)-powered code remediation and infrastructure as code (IaC) support for Amazon CodeWhisperer, an AI-powered productivity tool for the IDE and command line. Amazon CodeWhisperer is also now available in Visual Studio, in preview. These new enhancements to Amazon CodeWhisperer help to enable faster and more efficient software development by offloading undifferentiated work and delivering more automation, security, efficiency, and accelerated code delivery for customers, and provides this support in more places where developers love to work.

AI-powered code remediation – Since its launch, Amazon CodeWhisperer has identified hard-to-find security vulnerabilities with built-in security scans. It now provides generative AI-powered code suggestions to help remediate identified security and code quality issues. Built-in security scanning is designed to detect issues such as exposed credentials and log injection. Generative AI-powered code suggestions are designed to remediate the identified vulnerabilities, and are tailored to your application code so that you can quickly accept fixes with confidence. When a security scan is completed in CodeWhisperer, you are presented with code suggestions that you can simply accept to close the identified vulnerabilities quickly. Generative AI-powered code suggestions speed up the process of addressing security issues, so you can focus on higher-value work instead of manually reviewing code line by line to find the correct solution. You do not need to perform any additional setup in Amazon CodeWhisperer to start using this capability.

Security scanning is available for Java, Python, JavaScript, and now available for TypeScript, C#, AWS CloudFormation (YAML, JSON), AWS CDK (TypeScript, Python), and HashiCorp Terraform (HCL). Code suggestions to remediate vulnerabilities are currently available for code written in Java, Python, and JavaScript.

Infrastructure as code (IaC) – Amazon CodeWhisperer announces support for IaC, now encompassing AWS CloudFormation (YAML, JSON), AWS CDK (Typescript, Python), and HashiCorp Terraform (HCL). This update enhances the efficiency of IaC script development, allowing developers and DevOps teams to write infrastructure code seamlessly. With support for multiple IaC languages, CodeWhisperer promotes collaboration and consistency across diverse teams. This marks a significant advancement in cloud infrastructure development, offering a more streamlined and productive coding experience for users.

Visual Studio – Amazon CodeWhisperer is now available in Visual Studio 2022 (preview). Developers can build applications faster with real-time code suggestions for C#. Get started with the Individual Tier for free by installing the AWS Toolkit extension and signing in with an AWS Builder ID.

CodeWhisperer also helps developers code responsibly by flagging code suggestions that may resemble publicly available code. CodeWhisperer will provide the repository URL and license when code similar to public code.

Finally, Amazon CodeWhisperer recently previewed (11/20) a new time-saving capability for the command line interface. Now, Amazon CodeWhisperer adds typeahead code completions and inline documentation for hundreds of popular CLIs like Git, npm, AWS CLI, and Docker. It also adds the ability for you to translate natural language to shell code. For more details, read Introducing Amazon CodeWhisperer for command line.

Learn more
Amazon CodeWhisperer

Go build!

— Irshad

FlexGroup Volume Management for Amazon FSx for NetApp ONTAP is now available

2023-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/flexgroup-volume-management-for-amazon-fsx-for-netapp-ontap-is-now-available/

You can now create, manage, and back up your Amazon FSx for NetApp ONTAP FlexGroup volumes using the AWS Management Console, the Amazon FSx CLI, and the AWS SDK. FlexGroups can be as large as 20 petabytes and offer greater performance for demanding workloads. Before this launch you could only create them using the ONTAP CLI and the ONTAP REST API (these options remain available). Also new to this launch is the ability to create Amazon FSx backups of your FlexGroup volumes.

FlexVol and FlexGroup
FSx for ONTAP supports two volume styles:

FlexVol – Support for up to 300 TiB of storage, making these volumes a good fit for general-purpose workloads.

FlexGroup – Support for up to 20 PiB of storage and billions of files per volume, making these volumes a good fit for more demanding electronic design automation (EDA), seismic analysis, and software build/test workloads.

Using FlexGroups
I will use the AWS Management Console to create a new file system. I select Amazon FSx for NetApp ONTAP, and click Next:

I select Standard create, enter a name for my file system (FS-Jeff-1), and select Single-AZ as the deployment type:

I can use the recommended throughput capacity, or I can specify it explicitly:

As you can surmise from the values above, the throughput is determined by the number of high availability (HA) pairs that will be used to host your file system. A single-AZ file system can be hosted on up to 6 such pairs; a multi-AZ file system must reside on a single pair. To learn more about these options visit New – Scale-out file systems for Amazon FSx for NetApp ONTAP.

After making my selections for Network & security, Encryption, and Default storage virtual machine configuration, I select the FlexGroup volume style, assign a name to the initial volume, and either accept the recommended number of constituents or specify it myself:

On the next page I review my choices and click Create file system:

The creation process is a good time for a lunch break. When I return the initial volume (Vol1) of my file system is ready to use. I can create additional FlexVol or FlexGroup volumes as needed:

Things to Know
Here are a couple of things to keep in mind about FlexGroup volumes:

Constituents – Although each FlexGroup volume can have as many as 200 constituents, we recommend 8 per HA pair. Given the 300 TiB per-constituent size limit, this allows you to create volumes with up to 2.4 PiB of storage per HA pair. ONTAP will balance your files across constituents automatically.

File Counts – If you are using NFSv3 and expect to store many billions of files on a FlexGroup volume, be sure to enable 64-bit identifiers on the storage virtual machine associated with the file system.

Backups – Starting today you can also create backups of FlexGroup volumes, giving you the same fully-managed built-in options that you already have for FlexVol volumes.

NetApp System Manager – You can use the ONTAP CLI and the browser-based NetApp System Manager to perform advanced operations on your ONTAP file systems, storage virtual machines, and volumes. The management endpoint and administrator credentials are available on the File system details page:

Regions – Both volume styles are available in all AWS Regions where Amazon FSx for NetApp ONTAP is supported.

Pricing – You pay for the SSD storage, SSD IOPS, and throughput capacity that you provision, with separate charges for capacity pool usage, backups, and SnapLock licensing; see the Amazon FSx for NetApp ONTAP Pricing page to learn more.

— Jeff;

New – Scale-out file systems for Amazon FSx for NetApp ONTAP

2023-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-scale-out-file-systems-for-amazon-fsx-for-netapp-ontap/

You can now create Amazon FSx for NetApp ONTAP file systems that are up to 9x faster than even before. As is already the case with this service, the file systems are fully managed, with latency to primary storage at the sub-millisecond level and latency to the capacity pool in the tens of milliseconds. This new level of performance will allow you to bring even more of your most demanding electronic design automation (EDA), visual effects (VFX), and statistical computing workloads (to name just a few) to the cloud.

Existing FSx for ONTAP scale-up file systems are powered by a single pair of servers in an active/passive high availability (HA) configuration, and can support a maximum of 4 GBps of throughput and 192 TiB of SSD storage. The server pair can be deployed across distinct fault domains of a single Availability Zone, or in two separate Availability Zones to provide continuous availability even when an Availability Zone is unavailable.

With today’s launch you can now create scale-out FSx for ONTAP file systems that are powered by two to six HA pairs. Here are the specs for the scale-up and scale-out file systems (these are all listed as “up to” since you can specify your desired values for each one when you create your file system):

Deployment Type	Read Throughput	Write Throughput	SSD IOPS	SSD Storage	Availability Zones
Scale-up	Up to 4 GBps	Up to 1.8 GBps	Up to 160K	Up to 192 TiB	Single or Multiple
Scale-out	Up to 36 GBps	Up to 6.6 GBps	Up to 1.2M	Up to 1 PiB	Single

The amount of throughput that you specify for your file system will determine the actual server configuration as follows:

Specified Throughput	Deployment Type	HA Pairs	Throughput (Per Server)	SSD Storage (Per Server)	SSD IOPS (Per Server)
4 GBps or less	Scale-up	Single	Up to 4 GBps Read Up to 1.1 GBps Write (Single-AZ) Up to 1.8 GBps Write (Multi-AZ)	1-192 TiB	Up to 160K
More than 4 GBps	Scale-out	Up to 6	Up to 6 GBps Read Up to 1.1 GBps Write	1-512 TiB	Up to 200K

To learn more about your choices, visit Amazon FSx for NetApp ONTAP performance.

Creating a Scale-Out File System
I can create a scale-out file system using the AWS Management Console, AWS Command Line Interface (AWS CLI), or by writing code that calls the Amazon FSx CreateFileSystem function. I will use the console, and start by choosing Amazon FSx for NetApp ONTAP:

I choose Standard create, enter a name, select a Single-AZ deployment, and enter my desired SSD storage capacity. I can accepted the recommended throughput capacity, choose a value from the dropdown, or enter a value and see my options (more on that in a sec):

The dropdown has a helpful magical feature. If I type my desired throughput capacity it will show me one or more options:

The console will sometimes show me several options for the same amount of desired throughput capacity. Here are some guidelines to help you make a choice that will be a good fit for your workload:

Low Throughput – If you choose an option that provides 4 GBps or less, you will be running on a single HA pair. This is the simplest option to choose if you don’t need a high degree of throughput.

High Throughput and/or High Storage – Maximum throughput scales with the number of HA pairs that you provision. Also, choosing an option with more pairs will maximize your headroom for future growth in provisioned storage.

I make selections and enter values for the remaining options as usual, and click Next to review my settings. I check to make sure that I have made good choices for any of the attributes that can’t be edited after creation, and click Create file system.

I take a break to see what’s happening upstairs, feed my dog, and when I return my new 2 TB file system is ready to go:

I can increase the storage capacity and I can change provisioned IOPS as frequently as every 6 hours:

At the moment, the provisioned throughput capacity cannot be changed after creation if the file system uses more than one HA pair.

Available Now
Scale-out file systems are available in the US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Sydney), and Europe (Ireland) Regions and you can start to create and use them today.

Learn more

— Jeff;

Introducing shared VPC support for Amazon FSx for NetApp ONTAP

2023-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/introducing-shared-vpc-support-for-amazon-fsx-for-netapp-ontap/

You can now create Multi-AZ FSx for ONTAP file systems in VPCs that have been shared with you by other accounts in the same AWS Organization. This highly requested feature enables a clean separation of duties between network administrators and storage administrators, and makes it possible to create storage that’s durable, highly available, and accessible from multiple VPCs.

Shared VPC support
Before today’s launch, you had the ability to create Single-AZ FSx for ONTAP file systems in subnets that were shared with you by another AWS account, as well as both Single – and Multi-AZ file systems in subnets that you own.

With today’s launch you can now do the same for file systems in multiple Availability Zones. Multi-AZ FSx for ONTAP file systems offer even higher availability than Single-AZ file systems, and are a great way to address and support large-scale enterprise storage needs. This new support for shared VPCs gives enterprises, many of which make use of multiple VPCs for technical and organizational reasons, to use FSx for ONTAP in Multi-AZ deployments, while allowing network administrators and storage administrators to work independently.

This is easy to set up, but you do need to make sure that there are no IP address conflicts between subnets that are not shared between VPCs. I don’t have an AWS Organization set up, so I will hand-wave through part of this process. As a network administrator (the owner account), I use the AWS Resource Access Manager (RAM) to share the appropriate subnets of my VPC with the desired participant accounts in my Organization:

Then I (or the administrators for those accounts) accept the resource shares.

Next, I use the new FSx for ONTAP Settings to enable route table updates from participant accounts, and click Submit (this gives the FSx ONTAP service permission to modify route table entries in shared subnets on behalf of participant accounts):

At this point, the storage administrators for the participant accounts can create Multi-AZ FSx for ONTAP file systems in the subnets that have been shared with them by the owner accounts.

There is no additional charge for this feature and it is available in all AWS Regions where FSx for ONTAP is supported.

— Jeff;

Announcing on-demand data replication for Amazon FSx for OpenZFS

2023-11-27 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/on-demand-data-replication-for-amazon-fsx-for-openzfs/

Today we’re adding to Amazon FSx for OpenZFS the capability to send a snapshot from a file system to another file system in your account.

You can trigger the copy with one single API call or CLI command, and we take care of the rest. You don’t need to use commands like rsync and monitor the state of the transfer. The service takes care of the copy on your behalf. It manages potential network interruptions and retries automatically until the transfer completes. It transfers data incrementally at block level using OpenZFS’s native send and receive capabilities.

This new capability helps you to maintain agility by, for example, allowing quicker and easier creation of testing and development environments, and performance improvements by simplifying the management of read replicas to provide scale-out performance.

Amazon FSx for OpenZFS is a fully managed file storage service that lets you launch, run, and scale fully managed file systems built on the open source OpenZFS file system. FSx for OpenZFS makes it easy to migrate your on-premises ZFS file servers without changing your applications or how you manage data and to build new high-performance, data-intensive applications on the cloud.

Snapshots are one of the most powerful features of ZFS file systems. A snapshot is a read-only copy of a file system or volume. Snapshots can be created almost instantly and initially consume no additional disk space within the storage pool. When a snapshot is created, its space is initially shared between the snapshot and the file system and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot. The snapshot consumes incremental disk space by continuing to reference the old data and so prevents the space from being freed. Snapshots can be rolled back on-demand and almost instantly, even on very large file systems. Snapshots can also be cloned to form new volumes.

Snapshots are block-level copies. They are more efficient to transfer than traditional file-level copies, where the system must sometimes traverse millions of files to detect the ones that changed. Transferring an incremental snapshot is also more efficient than transferring an incremental file-based copy because snapshots are incremental at block level. They only contain blocks modified since the last snapshot.

On-demand replication of ZFS snapshots allows the transfer of terabytes of data using the native send and receive capability of OpenZFS without having to worry about the underlying infrastructure. We detect and manage network interruptions and other types of errors for you, making it easier for you to replicate data across file systems.

There are two main use cases where you might want to use this new capability.

Developers and quality assurance (QA) engineers might send on-demand snapshots to development and testing environments. It allows them to work with production data, ensuring accurate testing and development outcomes. The use of recent snapshots as consistent starting points for testing enhances the efficiency of the development and testing processes.

Data engineers might use on-demand replication to run parallel experiments on a dataset. Imagine your application processes a large dataset. You want to run multiple versions of your data processing algorithm on the same base dataset to find the best tuning for your use case. With on-demand data replication, you can create multiple identical copies of your file system and run each experiment in parallel.

Let’s see how it works
To prepare this demo, I use the FSx for OpenZFS section of the AWS Management Console. First, I create two Amazon FSx for OpenZFS volumes. Then, I mount the two file systems on one Amazon Linux instance (/zfs-filesystem1 and /zfs-filesystem2). I prepare a file on the first volume, and I expect to find the same file on the second volume after an on-demand replication.

To synchronize data between my two volumes, I navigate to the snapshot section of the console. Then I select Copy snapshot and update volume. I also have the option to copy the snapshot to a new ZFS volume.

On the Copy snapshot and update volume page, I select the destination File system and Volume. I also confirm the source snapshot. I choose the Source snapshot copy strategy, either requesting a full copy or an incremental copy. When ready, I select Update.

After a while—how long depends on the amount of data to transfer—I observe a new snapshot listed on the destination volume. In my demo scenario, it just takes a few seconds.

I return to my Linux instance and list the content available in my second mount point /zfs-snapshot. I am happy to see my cow ASCII art on the second file system .

Alternatively, I can automate on-demand transfers using the new FSx APIs: CopySnapshotAndUpdateVolume and CopySnapshotAndCreateVolume.

To set up an ongoing periodic replication, I use the provided CloudFormation template to create an automated replication schedule. When deployed, the system periodically takes a snapshot of the volume on the source file system and performs an incremental replication to a volume on the destination file system. For example, I could schedule replication to a development file system to happen once every 15 minutes for testing purposes.

Pricing and availability
This new capability is available in all AWS Regions where FSx for OpenZFS is available.

It comes at no additional cost. AWS charges the usual fees for network data transfer between Availability Zones.

You pay standard FSx for OpenZFS charges for the amount of storage used by the remote file system.

The new on-demand replication for Amazon FSx for OpenZFS allows you to efficiently transfer incremental file system snapshots to a new volume on your account. It allows developers and QA engineers to work with copies of production data and data engineers to run parallel experiments on datases.

Now go build and configure your first on-demand replication today!

— seb

IAM Access Analyzer updates: Find unused access, check policies before deployment

2023-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/iam-access-analyzer-updates-find-unused-access-check-policies-before-deployment/

We are launching two new features for AWS Identity and Access Management (IAM) Access Analyzer today:

Unused Access Analyzer – A new analyzer that continuously monitors roles and users looking for permissions that are granted but not actually used. Central security teams can take advantage of a dashboard view that will help them to find the accounts that can most benefit from a review of unused permissions, roles, and IAM users.

Custom Policy Checks – Validation that newly authored policies do not grant additional (and perhaps unintended) permissions. You can exercise tighter control over your IAM policies and accelerate the process of moving AWS applications from development to production by adding automated policy reviews to your CI/CD pipelines and custom policy tools.

Let’s take a look at today’s launches!

Unused Access Analyzer
You can already create an analyzer that monitors for external access. With today’s launch you can create one that looks for access permissions that are either overly generous or that have fallen into disuse. This includes unused IAM roles, unused access keys for IAM users, unused passwords for IAM users, and unused services and actions for active IAM roles and users.

After reviewing the findings generated by an organization-wide or account-specific analyzer, you can take action by removing permissions that you don’t need. You can create analyzers and analyze findings from the AWS Management Console, CLI, or API. Let’s start with the IAM Console. I click Analyzers and settings in the left-side navigation:

I can see my current analyzers (none, in this case). I click Create analyzer to proceed:

I specify Unused access analysis, leave the default tracking period of 90 days as-is, and opt to check my account rather than my Organization, then I click Create analyzer:

My analyzer is created, and I check back a little while later to see what it finds. My findings were available within a minute, but this will vary. Here are some of the findings:

As you can see, I have lots of unused IAM roles and permissions (clearly I am a bad Role model). I can click on a Finding to learn more:

If this is a role that I need, I can click Archive to remove it from the list of active findings. I can also create archive rules that will do the same for similar findings:

The external access analyzer works in a similar way, and is a perfect place to start when you are new to Access Analyzer and are ready to find and remove extra permissions:

The dashboard gives me an overview of all active findings:

If I create an analyzer and specify my Organization as the Zone of trust, I can also view a list that shows the accounts that have the largest number of active findings:

This feature is also available from the command line. I can create a new analyzer like this:

$ aws access-analyzer create-analyzer --type ACCOUNT_UNUSED_ACCESS \
  --analyzer-name OneWeek \
  --configuration '{"unusedAccess" : {"unusedAccessAge" : 90}}'
----------------------------------------------------------------------------
|                              CreateAnalyzer                              |
+-----+--------------------------------------------------------------------+
|  arn|  arn:aws:access-analyzer:us-east-1:348414629041:analyzer/OneWeek   |
+-----+--------------------------------------------------------------------+

I can list the findings, perhaps all I want is the actual resource Ids to start:

$  aws access-analyzer list-findings-v2 \
  --analyzer-arn  arn:aws:access-analyzer:us-east-1:123456789012:analyzer/OneWeek 
  --output json |
 jq -r '.findings[] | .resource'

arn:aws:iam::123456789012:role/MobileHub_Service_Role
arn:aws:iam::123456789012:role/EKSClusterRole
arn:aws:iam::123456789012:role/service-role/AWSDataSyncS3BucketAccess-jbarr-data
arn:aws:iam::123456789012:role/rds-monitoring-role
arn:aws:iam::123456789012:role/IsengardRoleForDependencyAssuranceIamAnalyzer
arn:aws:iam::123456789012:role/service-role/s3crr_role_for_rep-src_to_rep-dest
arn:aws:iam::123456789012:role/service-role/AWSDeepRacerServiceRole
...

I can archive findings by Id:

$ aws access-analyzer update-findings  \
  --analyzer-arn arn:aws:access-analyzer:us-east-1:123456789012:analyzer/OneWeek 
  --status ARCHIVED --ids "f0492061-8638-48ac-b91a-f0583cc839bf"

And I can perform the same operations using the IAM Access Analyzer API.

This feature is priced based on the number of IAM roles analyzed each month and is available in all AWS Regions where IAM is available.

Custom Policy Checks
You can now validate that IAM policies adhere to your security standards ahead of deployments and proactively detect non-conformant updates to policies. This will help you to innovate more quickly, move apps from development to production more efficiently, and to have confidence that any changes you make represent your intent.

Let’s start with my allow-all-ssm policy:

For illustrative purposes, I edit it to add S3 access:

Then I click Check for new access, confirm that I understand that a charge will be made, and click Check policy:

The automated reasoning verifies the policy and tells me that I did enable new access. If that was my intent I click Next to proceed, otherwise I rethink my changes to the policy:

This is a very simple and contrived example, but I am confident that you can see how useful and valuable this can be to your security efforts. You can also access this from the CLI (check-no-new-access) and API (CheckNoNewAccess).

There’s also another command and function that is designed to be used in your CI/CD pipelines, AWS CloudFormation hooks, and custom policy tools. check-access-not-granted and CheckAccessNotGranted accept a policy document and a permission such as s3:Get*, and check to make sure that the policy does not grant the permission. You could use this, for example, to make sure that a policy which specifies that Security Hub should be disabled cannot be deployed. This will help you to move from development to production with the confidence that your policies adhere to your organization’s security standards.

This feature is priced based on the number of checks that are performed each month and is available in all AWS commercial and AWS GovCloud Regions.

Learn more
AWS Identity and Access Management (IAM) Access Analyzer

— Jeff;

Amazon EKS Pod Identity simplifies IAM permissions for applications on Amazon EKS clusters

2023-11-27 Donnie Prakoso

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/amazon-eks-pod-identity-simplifies-iam-permissions-for-applications-on-amazon-eks-clusters/

Starting today, you can use Amazon EKS Pod Identity to simplify your applications that access AWS services. This enhancement provides you with a seamless and easy to configure experience that lets you define required IAM permissions for your applications in Amazon Elastic Kubernetes Service (Amazon EKS) clusters so you can connect with AWS services outside the cluster.

Amazon EKS Pod Identity helps you solve growing challenges for managing permissions across many of your EKS clusters.

Simplifying experience with Amazon EKS Pod Identity
In 2019, we introduced IAM roles for service accounts (IRSA). IRSA lets you associate an IAM role with a Kubernetes service account. This helps you to implement the principle of least privilege by giving pods only the permissions they need. This approach prioritizes pods in IAM and helps developers configure applications with fine-grained permissions that enable the least privileged access to AWS services.

Now, with Amazon EKS Pod Identity, it’s even easier to configure and automate granting AWS permissions to Kubernetes identities. As the cluster administrator, you no longer need to switch between Amazon EKS and IAM services to authenticate your applications to all AWS resources.

The overall workflow to start using Amazon EKS Pod Identity can be summarized in a few simple steps:

Step 1: Create an IAM role with required permissions for your application and specify pods.eks.amazonaws.com as the service principal in its trust policy.
Step 2: Install Amazon EKS Pod Identity Agent add-on using the Amazon EKS console or AWS Command Line Interface (AWS CLI).
Step 3: Map the role to a service account directly in the Amazon EKS console, APIs, or AWS CLI.

Once it’s done, any new pods that use that service account will automatically be configured to receive IAM credentials.

Let’s get started
Let me show you how you can get started with EKS Pod Identity. For the demo in this post, I need to configure permission for a simple API running in my Amazon EKS cluster, which will return the list of files in my Amazon Simple Storage Service (Amazon S3) bucket.

First, I need to create an IAM role to provide the required permissions so my applications can run properly. In my case, I need to configure permissions to access my S3 bucket.

Next, on the same IAM role, I need to configure its trust policy and configure the principal to pods.eks.amazonaws.com. The following is the IAM template that I use:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

At this stage, my IAM role is ready, and now we need to configure the Amazon EKS Pod Identity Agent in my cluster. For this article, I’m using my existing EKS cluster. If you want to learn how to do that, visit Getting started with Amazon EKS.

Moving on, I navigate to the Amazon EKS dashboard and then select my EKS cluster.

In my EKS cluster page, I need to select the Add-ons tab and then choose Get more add-ons.

Then, I need to add the Amazon EKS Pod Identity Agent add-on.

On the next page, I can add additional configuration if needed. In this case, I leave the default configuration and choose Next.

Then, I just need to review my add-on configuration and choose Create.

After a few minutes, the Amazon EKS Pod Identity Agent add-on is active for my cluster.

Once I have Amazon EKS Pod Identity in my cluster, I need to associate the IAM role to my Kubernetes pods.

I need to navigate to the Access tab in my EKS cluster. On the Pod Identity associations section, I select Create Pod Identity association to map my IAM role to Kubernetes pods.

Here, I use the IAM role that I created in the beginning. I also need to define my Kubernetes namespace and service account. If they don’t exist yet, I can type in the name of the namespace and service account. If they already exist, I can select them from the dropdown. Then, I choose Create.

Those are all the steps I need to do to configure IAM permissions for my applications running on Amazon EKS with EKS Pod Identity. Now, I can see my IAM role is listed in Pod Identity associations.

When I test my API running on Amazon EKS, it runs as expected and returns the list of files in my S3 bucket.

curl -X https://<API-URL> -H "Accept: application/json" 

{
   "files": [
         "test-file-1.md",
         "test-file-2.md"
    ]        
}

I found that Amazon EKS Pod Identity simplifies the experience of managing IAM roles for my applications running on Amazon EKS. I can easily reuse IAM roles across multiple EKS clusters without needing to update the role trust policy each time a new cluster is created.

New AWS APIs to configure EKS Pod Identity
You also have the flexibility to configure Amazon EKS Pod Identity for your cluster using AWS CLI. Amazon EKS Pod Identity provides a new set of APIs that you can use.

For example, I can use aws eks create-addon to install the Amazon EKS Pod Identity Agent add-on into my cluster. Here’s the AWS CLI command:

$ aws eks create-addon \
--cluster-name <CLUSTER_NAME> \
--addon-name eks-pod-identity-agent \
--addon-version v1.0.0-eksbuild.1

{
    "addon": {
    "addonName": "eks-pod-identity-agent",
    "clusterName": "<CLUSTER_NAME>",
    "status": "CREATING",
    "addonVersion": "v1.0.0-eksbuild.1",
    "health": {
        "issues": []
        },
    "addonArn": "<ARN>",
    "createdAt": 1697734297.597,
    "modifiedAt": 1697734297.612,
    "tags": {}
    }
}

Another example of what you can do with AWS APIs is to map the IAM role into your Kubernetes pods.

$ aws eks create-pod-identity-association \
  --cluster-name <CLUSTER_NAME> \
  --namespace <NAMESPACE> \
  --service-account <SERVICE_ACCOUNT_NAME> \
  --role-arn <IAM_ROLE_ARN>

Things to know

Availability – Amazon EKS Pod Identity is available in all AWS Regions supported by Amazon EKS, except the AWS GovCloud (US-East), AWS GovCloud (US-West), China (Beijing, operated by Sinnet), and China (Ningxia, operated by NWCD).

Pricing – Amazon EKS Pod Identity is available at no charge.

Supported Amazon EKS cluster – Amazon EKS Pod Identity supports Kubernetes running version 1.24 and above in Amazon EKS. You can see EKS Pod Identity cluster versions for more information.

Supported AWS SDK versions – You need to update your application to use the latest AWS SDK versions. Check out AWS developer tools to find out how to install and update your AWS SDK.

Get started today and visit EKS Pod Identities documentation page to learn more about how to simplify IAM management for your applications.

Happy building!
— Donnie

New Amazon WorkSpaces Thin Client provides cost-effective, secure access to virtual desktops

2023-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-workspaces-thin-client/

The new Amazon WorkSpaces Thin Client improves end-user and IT staff productivity with cost-effective, secure, easy-to-manage access to virtual desktops. The devices are preconfigured and shipped directly to the end user, ready to deploy, connect, and use.

Here’s my testing setup:

The Thin Client is a small cube that connects directly to a monitor, keyboard, mouse, and other USB peripherals such as headsets, microphones, and cameras. With the optional hub it can also drive a second monitor. The administrator can create environments that give users access to Amazon WorkSpaces, Amazon WorkSpaces Web, or Amazon AppStream 2.0, with multiple options for managing user identities and credentials using Active Directory.

Thin Clients in action
As a very long-time user of Amazon WorkSpaces via a thin client, I am thrilled to be able to tell you about this device and the administrative service behind it. While my priority is the ease with which I can switch from client to client while maintaining my working context (running apps, browser tabs, and so forth), administrators will find it attractive for other reasons. For example:

Cost – The device itself is low cost ($195 in the United States), far less expensive than a laptop and the associated operating system. Because the working environments are centrally configured and administered, there’s less work to be done in the field, leading to further cost savings. Further, the devices are far simpler than laptops, with less parts to break, wear out, or replace.

Security – The devices are shipped with a secure “secret” that is used to establish a trust relationship with the administrative service. There’s no data storage on the device, and it cannot host rogue apps that could attempt to exfiltrate data. It also helps to reduce risk of data leakage should a worker leave their job without returning their employer-supplied laptop.

Ease of Management – Administrators can easily create new environments for users or groups of users, distribute activation codes to them, and manage the environment via the AWS Management Console. They can set schedules for software updates and patches, verify compliance, and manage users over time.

Ease of Use – Users can unpack and connect the devices in minutes, enter their activation codes, log in to their virtual desktop environment, and start to work right away. They don’t have to take responsibility for installing software patches or updates, and can focus on their jobs.

There are lots of great use cases for these devices! First, there are situations where there’s a long-term need for regular access: call centers, task workers, training centers, and so forth. Second, there are other situations, where there’s a transient or short-term need for access: registration systems at large events, call centers stood up on a temporary basis for a special event or an emergency, disaster response, and the like. Given that some employees do not return laptops to their employers when they leave their job, providing them with inexpensive devices that do not have local storage makes a lot of sense.

Let’s walk through the process of getting set up, first as an administrator and then as a user.

Getting started as an administrator
The first step is to order some devices and have them shipped to my users, along with any desired peripherals.

Next, in my position as administrator, I open the Amazon WorkSpaces Thin Client Console, and click Get started:

Each Amazon WorkSpaces Thin Client environment provides access to a specific virtual desktop service (WorkSpaces, WorkSpaces Web, or Amazon AppStream 2.0). I click Create environment to move ahead:

I give my environment a name, indicate that I want patches applied automatically, and select WorkSpaces Web:

Next, I click Create WorkSpaces Web portal and go through those steps (not shown, basically choosing a VPC and two or more subnets, a security group, and an optional Private Certificate Authority):

I refresh, and my new portal is visible. I select it, enter a tag for tracking, and click Create environment:

My environment is ready right away. I keep a copy of the activation code (aci3a5yj) at hand to use later in the post:

I am using AWS Identity Center as my identity provider. I already set up my first user, and assigned myself to the MyWebPortal app (the precise steps that you take to do this will vary depending on your choice of identity provider):

Finally, as my last step in this process in my role as administrator, I share the activation code with my users (that would be me, in this case).

Getting started as a user
In my role as a user I return to my testing setup, power-on, go through a couple of quick steps to select my keyboard and connect to my home Wi-Fi, and enter my activation code:

Then I sign in using my AWS Identity Center user name and password:

And my WorkSpace is ready to use:

Administrator tools
As an administrator, I can manage environments, devices, and device software updates from the Thin Client Console. For example, I can review the list of devices that I manage:

Things to know
Here are a couple of things that are important to know:

Regions – The Thin Client Console is available in the US East (N. Virginia), US West (Oregon), Asia Pacific (Mumbai), Canada (Central), and Europe (Frankfurt, Ireland, Ireland, London) Regions.

Device Sales – The Amazon WorkSpaces Thin Clients are available in the United States now, with availability in other countries in early 2024.

Pricing – Devices are priced at $195, or $280 with an optional hub that allows you to use a second monitor. There’s a $6 per month fee to manage, maintain, and monitor each device, and you also pay for the underlying virtual desktop service.

Learn more
Visit the WorkSpaces Thin Client web page and Amazon Business Marketplace to learn more.

— Jeff;

Detect runtime security threats in Amazon ECS and AWS Fargate, new in Amazon GuardDuty

2023-11-27 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/introducing-amazon-guardduty-ecs-runtime-monitoring-including-aws-fargate/

Today, we’re announcing Amazon GuardDuty ECS Runtime Monitoring to help detect potential runtime security issues in Amazon Elastic Container Service (Amazon ECS) clusters running on both AWS Fargate and Amazon Elastic Compute Cloud (Amazon EC2).

GuardDuty combines machine learning (ML), anomaly detection, network monitoring, and malicious file discovery against various AWS data sources. When threats are detected, GuardDuty generates security findings and automatically sends them to AWS Security Hub, Amazon EventBridge, and Amazon Detective. These integrations help centralize monitoring for AWS and partner services, initiate automated responses, and launch security investigations.

GuardDuty ECS Runtime Monitoring helps detect runtime events such as file access, process execution, and network connections that might indicate runtime threats. It checks hundreds of threat vectors and indicators and can produce over 30 different finding types. For example, it can detect attempts of privilege escalation, activity generated by crypto miners or malware, or activity suggesting reconnaissance by an attacker. This is in addition to GuardDuty‘s primary detection categories.

GuardDuty ECS Runtime Monitoring uses a managed and lightweight security agent that adds visibility into individual container runtime behaviors. When using AWS Fargate, there is no need for you to install, configure, manage, or update the agent. We take care of that for you. This simplifies the management of your clusters and reduces the risk of leaving some tasks without monitoring. It also helps to improve your security posture and pass regulatory compliance and certification for runtime threats.

GuardDuty ECS Runtime Monitoring findings are visible directly in the console. You can configure GuardDuty to also send its findings to multiple AWS services or to third-party monitoring systems connected to your security operations center (SOC).

With this launch, Amazon Detective now receives security findings from GuardDuty ECS Runtime Monitoring and includes them in its collection of data for analysis and investigations. Detective helps to analyze, investigate, and quickly identify the root cause of potential security issues or suspicious activities. It collects log data from AWS resources and uses machine learning, statistical analysis, and graph theory to build a linked set of data that enables you to easily conduct security investigations.

Configure GuardDuty ECS Runtime Monitoring on AWS Fargate
For this demo, I choose to show the experience provided for AWS Fargate. When using Amazon ECS, you must ensure your EC2 instances have the GuardDuty agent installed. You can install the agent manually, bake it into your AMI, or use GuardDuty‘s provided AWS Systems Manager document to install it (go to Systems Manager in the console, select Documents, and then search for GuardDuty). The documentation has more details about installing the agent on EC2 instances.

When operating from a GuardDuty administrator account, I can enable GuardDuty ECS Runtime Monitoring at the organization level to monitor all ECS clusters in all organizations’ AWS accounts.

In this demo, I use the AWS Management Console to enable Runtime Monitoring. Enabling GuardDuty ECS Runtime Monitoring in the console has an effect on all your clusters.

When I want GuardDuty to automatically deploy the GuardDuty ECS Runtime Monitoring agent on Fargate, I enable GuardDuty agent management. To exclude individual clusters from automatic management, I can tag them with GuardDutyManaged=false. I make sure I tag my clusters before enabling ECS Runtime Monitoring in the console. When I don’t want to use the automatic management option, I can leave the option disabled and selectively choose the clusters to monitor with the tag GuardDutyManaged=true.

The Amazon ECS or AWS Fargate cluster administrator must have authorization to manage tags on the clusters.

The IAM TaskExecutionRole you attach to tasks must have permissions to download the GuardDuty agent from a private ECR repository. This is done automatically when you use the AmazonECSTaskExecutionRolePolicy managed IAM policy.

Here is my view of the console when the Runtime Monitoring and agent management are enabled.

I can track the deployment of the security agent by assessing the Coverage statistics across all the ECS clusters.

Once monitoring is enabled, there is nothing else to do. Let’s see what findings it detects on my simple demo cluster.

Check out GuardDuty ECS runtime security findings
When GuardDuty ECS Runtime Monitoring detects potential threats, they appear in a list like this one.

I select a specific finding to view more details about it.

Things to know
By default, a Fargate task is immutable. GuardDuty won’t deploy the agent to monitor containers on existing tasks. If you want to monitor containers for already running tasks, you must stop and start the tasks after enabling GuardDuty ECS Runtime Monitoring. Similarly, when using Amazon ECS services, you must force a new deployment to ensure tasks are restarted with the agent. As I mentioned already, be sure the tasks have IAM permissions to download the GuardDuty monitoring agent from Amazon ECR.

We designed the GuardDuty agent to have little impact on performance, but you should plan for it in your Fargate task sizing calculations.

When you choose automatic agent management, GuardDuty also creates a VPC endpoint to allow the agent to communicate with GuardDuty APIs. When—just like me—you create your cluster with a CDK or CloudFormation script with the intention to delete the cluster after a period of time (for example, in a continuous integration scenario), bear in mind that the VPC endpoint must be deleted manually to allow CloudFormation to delete your stack.

Pricing and availability
You can now use GuardDuty ECS Runtime Monitoring on AWS Fargate and Amazon EC2 instances. For a full list of Regions where GuardDuty ECS Runtime Monitoring is available, visit our Region-specific feature availability page.

You can try GuardDuty ECS Runtime Monitoring for free for 30 days. When you enable GuardDuty for the first time, you have to explicitly enable GuardDuty ECS Runtime Monitoring. At the end of the trial period, we charge you per vCPU per hour of the monitoring agents. The GuardDuty pricing page has all the details.

Get insights about the threats to your container and enable GuardDuty ECS Runtime Monitoring today.

— seb

Introducing Amazon EC2 high memory U7i Instances for large in-memory databases (preview)

2023-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/introducing-amazon-ec2-high-memory-u7i-instances-for-large-in-memory-databases-preview/

The new U7i instances are designed to support large, in-memory databases including SAP HANA, Oracle, and SQL Server. Powered by custom fourth generation Intel Xeon Scalable Processors (Sapphire Rapids), the instances are now available in multiple AWS regions in preview form, in the US West (Oregon), Asia Pacific (Seoul), and Europe (Frankfurt) AWS Regions, as follows:

Instance Name	vCPUs	Memory (DDR5)	EBS Bandwidth	Network Bandwidth
u7in-16tb.224xlarge	896	16,384 GiB	100 Gbps	100 Gbps
u7in-24tb.224xlarge	896	24,576 GiB	100 Gbps	100 Gbps
u7in-32tb.224xlarge	896	32,768 GiB	100 Gbps	100 Gbps

We are also working on a smaller instance:

Instance Name	vCPUs	Memory (DDR5)	EBS Bandwidth	Network Bandwidth
u7i-12tb.224xlarge	896	12,288 GiB	60 Gbps	100 Gbps

Here’s what 32 TiB of memory looks like:

And here are the 896 vCPUs (and lots of other info):

When compared to the first generation of High Memory instances, the U7i instances offer up to 125% more compute performance and up to 120% more memory performance. They also provide 2.5x as much EBS bandwidth, giving you the ability to hydrate in-memory databases at a rate of up to 44 terabytes per hour.

Each U7i instance supports attachment of up to 128 General Purpose (gp2 and gp3) or Provisioned IOPS (io1 and io2 Block Express) EBS volumes. Each io2 Block Express volume can be as big as 64 TiB and can deliver up to 256K IOPS at up to 32 Gbps, making them a great match for the U7i instance.

On the network side, the instances support ENA Express and deliver up to 25 Gbps of bandwidth per network flow.

Supported operating systems include Red Hat Enterprise Linux and SUSE Enterprise Linux Server.

Join the Preview
If you are ready to put the U7i instances to the test in your environment, join the preview.

— Jeff;

Amazon Detective adds new capabilities to accelerate and improve your cloud security investigations

2023-11-27 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/amazon-detective-adds-investigations-and-finding-group-summaries-to-help-you-investigate-security-findings/

Today, Amazon Detective adds four new capabilities to help you save time and strengthen your security operations.

First, Detective investigations for IAM help security analysts investigate AWS Identity and Access Management (IAM) objects, such as users and roles, for indicators of compromise (IoCs) to determine potential involvement in known tactics from the MITRE ATT&CK framework. These automatic investigations are available in the Detective section of the AWS Management Console and through a new API to automate your analysis or incident response or to send these findings to other systems, such as AWS Security Hub or your SIEM.

Second, Detective finding group summaries uses generative artificial intelligence (AI) to enrich its investigations. It automatically analyzes finding groups and provides insights in natural language to accelerate security investigations. It provides a plain language title based on the analysis of the finding group with relevant summarized insights, such as describing the activity that initiated the event and its impact, if any. Finding group summaries handles the heavy lifting of analyzing the finding group built across multiple AWS data sources, making it easier and faster to investigate unusual or suspicious activity.

In addition to these two new capabilities that I describe in this post, Detective adds another two capabilities not covered here:

Detective now supports security investigations for threats detected by Amazon GuardDuty ECS Runtime Monitoring.
Detective now integrates with Amazon Security Lake, enabling security analysts to query and retrieve logs stored in Security Lake.

Amazon Detective makes it easier to analyze, investigate, and quickly identify the root cause of security findings or suspicious activities. Detective uses machine learning (ML), statistical analysis, and graph theory to help you visualize and conduct faster and more efficient security investigations. Detective automatically collects logs data and events from sources like AWS CloudTrail logs, Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, Amazon GuardDuty findings, Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, and AWS security findings. Detective maintains up to a year of aggregated data for analysis and investigations.

Cloud security professionals often find threat hunting and incident investigations to be resource-intensive and time-consuming. They must manually gather and analyze data from various sources to identify potential IAM-related threats. IAM investigations are particularly challenging due to dynamic cloud permissions and credentials. Analysts need to piece together data from different systems, including audit logs, entitlement reports, and CloudTrail events, which can be dispersed. Cloud permissions are often granted on-demand or through automation scripts, making authorization changes hard to track. Reconstructing activity timelines and identifying irregular entitlements can take hours or days, depending on complexity. Limited visibility into legacy systems and incomplete logs further complicates IAM investigations, making it difficult to obtain a definitive understanding of unauthorized access.

Detective investigations for IAM triage findings and surface only the most critical, suspicious issues, allowing security analysts to focus on high-level investigations. It automatically analyzes resources in your AWS environment to identify potential indicators of compromise or suspicious activity using machine learning and threat intelligence. This allows analysts to identify patterns and comprehend which resources are impacted by security events, offering a proactive approach to threat identification and mitigation.

The investigations are not only available in the console; you can use the new StartInvestigation API to automate a remediation workflow or collect information about all IP involved or AWS resources compromised. You can also use the API to feed the data to other systems to build a consolidated view of your security posture.

Finding group summaries evaluates the connections between security events across an environment and provides insights in natural language that link related threats, compromised resources, and malicious actor behavior. This narrative offers security analysts a comprehensive overview of security incidents that goes beyond individual service reports. By grouping and contextualizing data from multiple sources, finding group summaries identifies threats that might go unnoticed when insights are isolated. This approach improve the speed and efficiency of investigations and responses. Security analysts can utilize finding group summaries to gain a holistic understanding of security events and their interrelationships, helping them make informed decisions regarding containment and remediation.

Let’s see these two capabilities in action
In this demo, I start with Detective investigations for IAM in the Detective section of the console. The Detective dashboard shows me the number of investigations done and the number of IAM roles and users involved in suspicious activities.

From there, I drill down the list of investigations.

And I select one specific investigation to get the details. There is a summary first.

I scroll down the page to see what IP addresses are involved and for what type of activities. This example shows me a physical impossibility: the same IP was used in a short time from two different places, Australia and Japan.

The most interesting section of the page, in my opinion, is the mappings to tactics, techniques, and procedures (TTP). All TTPs are classified according to their severity. The console shows the techniques and actions used. When selecting a specific TTP, I can see the details in the right pane. In this example, the suspicious IP address has been involved in more than 2,000 failed attempts to change the trusted policy of an IAM role.

Finally, I navigate to the Indicators tab to see the list of indicators.

On the other side, finding group summaries is available under Finding groups. I select a finding group to receive a natural language explanation of the findings and risks involved.

Pricing and availability
These two new capabilities are now available to all AWS customers.

Detective investigations for IAM is available in all AWS Regions where Detective is available. Finding group summaries is available in five AWS Regions: US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore, Tokyo), and Europe (Frankfurt).

Learn all the details about Amazon Detective and get started today.

— seb

Top announcements of AWS re:Invent 2023

2023-11-27 AWS Editorial Team

Post Syndicated from AWS Editorial Team original https://aws.amazon.com/blogs/aws/top-announcements-of-aws-reinvent-2023/

Welcome to Las Vegas, where excitement fills the air for the premier AWS event of the year – re:Invent 2023! Happening Nov. 27 – Dec. 1, re:Invent 2023 offers keynotes, training, Innovation Talks, AWS Builder Labs, and much more to inspire you on your cloud journey.

Can’t make it in person? Tune in to the live streams of the keynotes, and check back here as we provide daily updates on the most exciting product launches.

If you don’t want to miss a thing, here are a few more ways to keep in touch:

(This post was last updated: 5:05 p.m. PST, Nov. 26, 2023.)

Use natural language to query Amazon CloudWatch logs and metrics (preview)
To make it easy to interact with your operational data, Amazon CloudWatch is introducing natural language query generation for Logs and Metrics Insights.

Increase collaboration and securely share cloud knowledge with AWS re:Post Private
re:Post Private includes content tailored specifically for your organization’s use cases, along with private discussion and collaboration forums for the members of your organization and your AWS account team.

Use anomaly detection with AWS Glue to improve data quality (preview)
This new feature will help to improve your data quality by using machine learning to detect statistical anomalies and unusual patterns.

Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities
With this new feature, you can now offload client authentication to Application Load Balancer, ensuring only trusted clients communicate with backend applications.

Check your AWS Free Tier usage programmatically with a new API
You can use the API directly with the AWS Command Line Interface or integrate it into an application with the AWS SDKs.

Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics
You can now consolidate metrics from your hybrid, multicloud, and on-premises data sources using Amazon CloudWatch and process them in a consistent, unified fashion.

Announcing cross-region data replication for Amazon WorkSpaces
Snapshots are taken every 12 hours, replicated to the desired destination region, and are used to provide a recovery point objective of 12-24 hours.

Amazon Transcribe Call Analytics adds new generative AI-powered call summaries (preview)
Powered by Amazon Bedrock, this feature helps businesses improve customer experience, and agent and supervisor productivity by automatically summarizing customer service calls.

Build generative AI apps using AWS Step Functions and Amazon Bedrock
Step Functions provides two new optimized API actions for Amazon Bedrock: InvokeModel and CreateModelCustomizationJob.

New Cost Optimization Hub centralizes recommended actions to save you money
This new AWS Billing and Cost Management feature makes it easy for you to identify, filter, aggregate, and quantify savings for AWS cost optimization recommendations.

Amazon CloudWatch Logs now offers automated pattern analytics and anomaly detection
Amazon CloudWatch can now automatically recognize and cluster patterns among log records, extract noteworthy content and trends, and notify you of anomalies using advanced machine learning algorithms.

Amazon Managed Service for Prometheus collector provides agentless metric collection for Amazon EKS
This new capability discovers and collects Prometheus metrics from Amazon Elastic Kubernetes Service (Amazon EKS) automatically and without an agent.

Optimize your storage costs for rarely-accessed files with Amazon EFS Archive
We’ve added a new storage class for Amazon Elastic File System optimized for long-lived data that is rarely accessed.

New Amazon CloudWatch log class for infrequent access logs at a reduced price
This new log class offers a tailored set of capabilities at a lower cost for infrequently accessed logs, enabling customers to consolidate all their logs in one place in a cost-effective manner.

Use natural language to query Amazon CloudWatch logs and metrics (preview)

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/use-natural-language-to-query-amazon-cloudwatch-logs-and-metrics-preview/

To make it easy to interact with your operational data, Amazon CloudWatch is introducing today natural language query generation for Logs and Metrics Insights. With this capability, powered by generative artificial intelligence (AI), you can describe in English the insights you are looking for, and a Logs or Metrics Insights query will be automatically generated.

This feature provides three main capabilities for CloudWatch Logs and Metrics Insights:

Generate new queries from a description or a question to help you get started easily.
Query explanation to help you learn the language including more advanced features.
Refine existing queries using guided iterations.

Let’s see how these work in practice with a few examples. I’ll cover logs first and then metrics.

Generate CloudWatch Logs Insights queries with natural language
In the CloudWatch console, I select Log Insights in the Logs section. I then select the log group of an AWS Lambda function that I want to investigate.

I choose the Query generator button to open a new Prompt field where I enter what I need using natural language:

Tell me the duration of the 10 slowest invocations

Then, I choose Generate new query. The following Log Insights query is automatically generated:

fields @timestamp, @requestId, @message, @logStream, @duration 
| filter @type = "REPORT" and @duration > 1000
| sort @duration desc
| limit 10

I choose Run query to see the results.

I find that now there’s too much information in the output. I prefer to see only the data I need, so I enter the following sentence in the Prompt and choose Update query.

Show only timestamps and latency

The query is updated based on my input and only the timestamp and duration are returned:

fields @timestamp, @duration 
| filter @type = "REPORT" and @duration > 1000
| sort @duration desc
| limit 10

I run the updated query and get a result that is easier for me to read.

Now, I want to know if there are any errors in the log. I enter this sentence in the Prompt and generate a new query:

Count the number of ERROR messages

As requested, the generated query is counting the messages that contain the ERROR string:

fields @message
| filter @message like /ERROR/
| stats count()

I run the query and find out that there are more errors than I expected. I need more information.

I use this prompt to update the query and get a better distribution of the errors:

Show the errors per hour

The updated query uses the bin() function to group the result in one hour intervals.

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) by bin(1h)

Let’s see a more advanced query about memory usage. I select the log groups of a few Lambda functions and type:

Show invocations with the most over-provisioned memory grouped by log stream

Before generating the query, I choose the gear icon to toggle the options to include my prompt and an explanation as comment. Here’s the result (I split the explanation over multiple lines for readability):

# Show invocations with the most over-provisioned memory grouped by log stream

fields @logStream, @memorySize/1000/1000 as memoryMB, @maxMemoryUsed/1000/1000 as maxMemoryUsedMB, (@memorySize/1000/1000 - @maxMemoryUsed/1000/1000) as overProvisionedMB 
| stats max(overProvisionedMB) as maxOverProvisionedMB by @logStream 
| sort maxOverProvisionedMB desc

# This query finds the amount of over-provisioned memory for each log stream by
# calculating the difference between the provisioned and maximum memory used.
# It then groups the results by log stream and calculates the maximum
# over-provisioned memory for each log stream. Finally, it sorts the results
# in descending order by the maximum over-provisioned memory to show
# the log streams with the most over-provisioned memory.

Now, I have the information I need to understand these errors. On the other side, I also have EC2 workloads. How are those instances running? Let’s look at some metrics.

Generate CloudWatch Metrics Insights queries with natural language
In the CloudWatch console, I select All metrics in the Metrics section. Then, in the Query tab, I use the Editor. If you prefer, the Query generator is available also in the Builder.

I choose Query generator like before. Then, I enter what I need using plain English:

Which 10 EC2 instances have the highest CPU utilization?

I choose Generate new query and get a result using the Metrics Insights syntax.

SELECT AVG("CPUUtilization")
FROM SCHEMA("AWS/EC2", InstanceId)
GROUP BY InstanceId
ORDER BY AVG() DESC
LIMIT 10

To see the graph, I choose Run.

Well, it looks like my EC2 instances are not doing much. This result shows how those instances are using the CPU, but what about storage? I enter this in the prompt and choose Update query:

How about the most EBS writes?

The updated query replaces the average CPU utilization with the sum of bytes written to all EBS volumes attached to the instance. It keeps the limit to only show the top 10 results.

SELECT SUM("EBSWriteBytes")
FROM SCHEMA("AWS/EC2", InstanceId)
GROUP BY InstanceId
ORDER BY SUM() DESC
LIMIT 10

I run the query and, by looking at the result, I have a better understanding of how storage is being used by my EC2 instances.

Try entering some requests and run the generated queries over your logs and metrics to see how this works with your data.

Things to know
Amazon CloudWatch natural language query generation for logs and metrics is available in preview in the US East (N. Virginia) and US West (Oregon) AWS Regions.

There is no additional cost for using natural language query generation during the preview. You only pay for the cost of running the queries according to CloudWatch pricing.

Generated queries are produced by generative AI and dependent on factors including the data selected and available in your account. For these reasons, your results may vary.

When generating a query, you can include your original request and an explanation of the query as comments. To do so, choose the gear icon in the bottom right corner of the query edit window and toggle those options.

This new capability can help you generate and update queries for logs and metrics, saving you time and effort. This approach allows engineering teams to scale their operations without worrying about specific data knowledge or query expertise.

Use natural language to analyze your logs and metrics with Amazon CloudWatch.

— Danilo

Increase collaboration and securely share cloud knowledge with AWS re:Post Private

2023-11-26 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/increase-collaboration-and-securely-share-cloud-knowledge-with-aws-repost-private/

Today we’re launching AWS re:Post Private, a fully managed knowledge service to accelerate cloud adoption, improve productivity, and drive innovation. re:Post Private allows organizations to increase collaboration and access knowledge resources built for your cloud community. It includes curated collections of technical content and training materials from AWS. The content is tailored specifically for your organization’s use cases, along with private discussion and collaboration forums for the members of your organization and your AWS account team.

As its name implies, you can think of it as a private version of AWS re:Post, with private content and access limited to people that belong to your organization and your AWS Account team.

Organizations of all sizes and verticals are increasingly moving their operations to the cloud. To ensure cloud adoption success, organizations must have the right skills and structure in place. The optimal way to achieve this is by setting up a centralized cloud center of excellence (CCOE). A CCOE is a centralized governance function for the organization and acts in a consultative role for central IT, business-unit IT, and cloud service consumers in the business. According to Gartner, a CCOE has three pillars: governance, brokerage, and community. The community pillar establishes the cloud community of practice (COP) that brings together stakeholders and facilitates cloud collaboration. It helps organizations adapt themselves for cloud adoption by promoting COP member interaction and facilitating cloud-related training and skills development.

AWS re:Post Private facilitates the creation, structure, and management of an internal cloud community of practice. It allows you to build a custom knowledge base that is searchable, reusable, and scalable. It allows community members to post private questions and answers and publish articles. It combines the benefits of traditional forums, such as community discussion and collaboration, with the benefits of an integrated information experience.

AWS re:Post Private is a fully managed service: there is no need to operate complex knowledge management and collaboration technologies or to develop custom solutions.

AWS re:Post Private also facilitates your interactions with AWS Support. You can create a support case directly from your private re:Post, and you can convert case resolution to reusable knowledge visible to all in your organization.

You choose in which AWS Region re:Post Private stores your data and who has access. All data at rest and in transit is encrypted using industry-standard algorithms. Your administrator chooses between using AWS-managed encryption keys or keys you manage and control.

Your organization’s Technical Account Managers are automatically added to your private re:Post. You can select other persons to invite among your organization and AWS teams, such as your AWS Solutions Architect. Only your private re:Post administrators need an AWS account. All other users can federate from your organization’s identity provider, such as Microsoft Active Directory.

Let’s see how to create a re:Post Private
To get started with AWS re:Post Private, as an administrator, I point my browser to the re:Post section of the AWS Management Console. I select Create private re:Post and enter the information needed to create a private re:Post for my organization, my team, or my project.

I can choose the Data encryption parameters and whether or not I enable Service access for Support case integration. When I’m ready, I select Create this re:Post.

Once the private re:Post is created, I can grant access to users and groups. User and group information comes from AWS IAM Identity Center and your identity provider. Invited users receive an email inviting them to connect to the private re:Post and create their profile.

That’s pretty much it for the administrator part. Once the private re:Post is created, I receive an endpoint name that I can share with the rest of my organization.

Let’s see how to use re:Post Private
As a member of the organization, I navigate to re:Post Private using the link I received from the administrator. I authenticate with the usual identity service of my organization, and I am redirected to the re:Post Private landing page.

On the top menu, I can select a tab to view the contents for Questions, Community Articles, Selections, Tags, Topics, Community Groups, or My Dashboard. This should be familiar if you already use the public knowledge service AWS re:Post that adopted a similar structure.

Further down on the page, I see the popular topics and the top contributors in my organization.I also have access to Questions and Community Groups. I can search the available content by keyword, tags, author, and so on.

Pricing and availability
You can create your organization’s AWS re:Post Private in the following AWS Regions: US West (Oregon) and Europe (Frankfurt).

AWS re:Post Private is available to customers having an AWS Enterprise or Enterprise On-Ramp support plan. re:Post Private offers a free tier that allows you to explore and try out standard capabilities for six months. There is no limit on the number of users in the free tier, and content storage is limited to 10 GB. When you reach the free storage limit, the plan is converted to the paid standard tier.

With AWS re:Post Private Standard tier, you only pay for what you use. We charge based on the number of users per month. Please visit the re:Post Private pricing page for more information.

Get started today and activate AWS re:Post Private for your organization.

— seb

Use anomaly detection with AWS Glue to improve data quality (preview)

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/use-anomaly-detection-with-aws-glue-to-improve-data-quality-preview/

We are launching a preview of a new AWS Glue Data Quality feature that will help to improve your data quality by using machine learning to detect statistical anomalies and unusual patterns. You get deep insights into data quality issues, data quality scores, and recommendations for rules that you can use to continuously monitor for anomalies, all without having to write any code.

Data quality counts
AWS customers already build data integration pipelines to extract and transform data. They set up data quality rules to ensure that the resulting data is of high quality and can be used to make accurate business decisions. In many cases, these rules assess the data based on criteria that were chosen and locked in at a specific point in time, reflecting the current state of the business. However, as the business environment changes and the properties of the data shift, the rules are not always reviewed and updated.

For example, a rule could be set to verify that daily sales are at least ten thousand dollars for an early-stage business. As the business succeeds and grows, the rule should be checked and updated from time to time, but in practice this rarely happens. As a result, if there’s an unexpected drop in sales, the outdated rule does not activate, and no one is happy.

Anomaly detection in action
To detect unusual patterns and to gain deeper insights into data, organizations try to create their own adaptive systems or turn to costly commercial solutions that require specific technical skills and specialized business knowledge.

To address this widespread challenge, Glue Data Quality now makes use of machine learning (ML).

Once activated, this cool new addition to Glue Data Quality gathers statistics as fresh data arrives, using ML and dynamic thresholds to learn from past patterns while looking outliers and unusual data patterns. This process produces observations and also visualizes trends so that you can quickly gain a better understanding of the anomaly.

You will also get rule recommendations as part of the Observations, and you can easily and progressively add them to your data pipelines. Rules can enforce an action such as stopping your data pipelines. In the past, you could only write static rules. Now, you can write Dynamic rules that have auto-adjusting thresholds and AnomalyDetection Rules that grasp recurring patterns and spot deviations. When you use rules as part of data pipelines, they can stop the data flow so that a data engineer can review, fix and resume.

To use anomaly detection, I add an Evaluate Data Quality node to my job:

I select the node and click Add analyzer to choose a statistic and the columns:

Glue Data Quality learns from the data to recognize patterns and then generates observations that will be shown in the Data quality tab:

And a visualization:

After I review the observations I add new rules. The first one sets adaptive thresholds that check the row count is between the smallest of the last 10 runs and the largest of the last 20 runs. The second one looks for unusual patters, for example RowCount being abnormally high on weekends:

Join the preview
This new capability is available in preview in the following AWS Regions: US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Ireland). To learn more, read Data Quality Anomaly Detection]].

Stay tuned for a detailed blog post when this feature launches!

Learn more

Data Quality Anomaly Detection

— Jeff;

Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities

2023-11-26 Channy Yun

Post Syndicated from Channy Yun original https://aws.amazon.com/blogs/aws/mutual-authentication-for-application-load-balancer-to-reliably-verify-certificate-based-client-identities/

Today, we are announcing support for mutually authenticating clients that present X509 certificates to Application Load Balancer. With this new feature, you can now offload client authentication to the load balancer, ensuring only trusted clients communicate with their backend applications. This new capability is built on S2N, AWS’s open source Transport Layer Security (TLS) implementation that provides strong encryption and protections against zero-day vulnerabilities, which developers can trust.

Mutual authentication (mTLS) is commonly used for business-to-business (B2B) applications such as online banking, automobile, or gaming devices to authenticate devices using digital certificates. Companies typically use it with a private certificate authority (CA) to authenticate their clients before granting access to data and services.

Customers have implemented mutual authentication using self-created or third-party solutions that require additional time and management overhead. These customers spend their engineering resources to build the functionality into their backend, update their code to keep up with the latest security patches, and invest heavily in infrastructure to create and rotate certificates.

With mutual authentication on Application Load Balancer, you have a fully managed, scalable, and cost-effective solution that enables you to use your developer resources to focus on other critical projects. Your ALB will authenticate clients with revocation checks and pass client certificate information to the target, which can be used for authorization by applications.

Getting started with mutual authentication on ALB
To enable mutual authentication on ALB, choose Create Application Load Balancer by the ALB wizard on Amazon EC2 console. When you select HTTPS in the Listeners and routing section, you can see more settings such as security policy, default server certificate, and a new client certificate handling option to support mutual authentication.

With Mutual authentication (mTLS) enabled, you can configure how listeners handle requests that present client certificates. This includes how your Application Load Balancer authenticates certificates and the amount of certificate metadata that is sent to your backend targets.

Mutual authentication has two options. The Passthrough option sends all the client certificate chains received from the client to your backend application using HTTP headers. The mTLS-enabled Application Load Balancer gets the client certificate in the handshake, establishes a TLS connection, and then sends whatever it gets in HTTPS headers to the target application. The application will need to verify the client certificate chain to authenticate the client.

With the Verify with trust store option, Application Load Balancer and client verify each other’s identity and establish a TLS connection to encrypt communication between them. We introduce a new trust store feature, and you can upload any CA bundle with roots and/or intermediate certificates generated by AWS Private Certificate Authority or any other third party CA as the source of trust to validate your client certificates.

It requires selecting an existing trust store or creating a new one. Trust stores contain your CAs, trusted certificates, and, optionally, certificate revocation lists (CRLs). The load balancer uses a trust store to perform mutual authentication with clients.

To use this option and create a new trust store, choose Trust Stores in the left menu of the Amazon EC2 console and choose Create trust store.

You can choose a CA certificate bundle in PEM format and, optionally, CRLs from your Amazon Simple Storage Service (Amazon S3) bucket. A CA certificate bundle is a group of CA certificates (root or intermediate) used by a trust store. CRLs can be used when a CA revokes client certificates that have been compromised, and you need to reject those revoked certificates. You can replace a CA bundle, and add or remove CRLs from the trust store after creation.

You can use the AWS Command Line Interface (AWS CLI) with new APIs such as create-trust-store to upload CA information, configure the mutual-authentication-mode on the Application Load Balancer listener, and send user certificate information to targets.

$ aws elbv2 create-trust-store --name my-tls-name \
    --ca-certificates-bundle-s3-bucket channy-certs \
    --ca-certificates-bundle-s3-key Certificates.pem \
    --ca-certificates-bundle-s3-object-version <version>
>> arn:aws:elasticloadbalancing:root:file1
$ aws elbv2 create-listener --load balancer-arn <value> \
    --protocol HTTPS \
    --port 443 \
    --mutual-authentication Mode=verify,
      TrustStoreArn=<arn:aws:elasticloadbalancing:root:file1>

If you already have your own private CA, such as AWS Private CA, third-party CA, or self-signed CA, you can upload their CA bundle or CRLs to the Application Load Balancer trust store to enable mutual authentication.

To test the mutual authentication on Application Load Balancer, follow the step-by-step instructions to make a self-signed CA bundle and client certificate using OpenSSL, upload them to the Amazon S3 bucket, and use them with an ELB trust store.

You can use curl with the --key and --cert parameters to send the client certificate as part of the request:

$ curl --key my_client.key --cert my_client.pem https://api.yourdomain.com

Mutual authentication can fail if a client presents an invalid or expired certificate, fails to present a certificate, cannot find a trust chain, or if any links in the trust chain have expired, or the certificate is on the revocation list.

Application Load Balancer will close the connections whenever it fails to authenticate a client and will record new connection logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the client’s IP address, handshake latency, TLS cipher used, and client certificate details. You can use these connection logs to analyze request patterns and troubleshoot issues.

To learn more, see Mutual authentication on Application Load Balancer in the AWS documentation.

Now available
Mutual authentication on Application Load Balancer is now available in all commercial AWS Regions where Application Load Balancer is available, except China. With no upfront costs or commitments required, you only pay for what you use. To learn more, see the Elastic Load Balancing pricing page.

Give it a try now and send feedback to AWS re:Post for Amazon EC2 or through your usual AWS Support contacts.

Learn more:
Application Load Balancer product page

— Channy

Check your AWS Free Tier usage programmatically with a new API

2023-11-26 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/check-your-aws-free-tier-usage-programmatically-with-a-new-api/

Starting today, you can check your usage of the AWS Free Tier using the new AWS Free Tier API. You can use the API directly with the AWS Command Line Interface (AWS CLI) or integrate it into an application with the AWS SDKs.

The AWS Free Tier program provides you with the ability to explore and try out AWS services free of charge up to specified limits for each service. The AWS Free Tier includes three different types of offerings:

Always free offers allow customers to use a service for free up to specified limits as long as they are an AWS customer.
12 months free offers allow customers to use a service for free up to specified limits for one year from the date the account was activated.
Short-term trials are free to use for a specified period or up to a one-time limit, depending on the service.

Once you begin turning on AWS resources and interacting with AWS services that offer a free tier, you need to keep track of your progress toward the free tier limit so that you know when to expect to switch to pay-as-you-go pricing.

There are a few ways you can keep track of your AWS Free Tier usage:

The usage alerts in the Billing preferences of the AWS Billing and Cost Management console are enabled by default (unless the account was created via AWS Organizations) and send you emails when you exceed 85 percent of the Free Tier limit for each service.
You can create a zero-spend or a monthly-cost budget in the Budgets section of the Billing and Cost Management console. Using templates, it just requires a couple of clicks and the email address to notify.
The Free Tier page in the Billing and Cost Management console tells you the service, the type of offer, the current usage, and the forecasted usage for each offer in the current billing period.
The new GetFreeTierUsage API provides the same information in the Free Tier page with a structured format that you can use programmatically.

Let’s see how this new API works in practice.

Using the AWS Free Tier API with the AWS CLI
I got access to a new account created in the last few months. Here, I use the AWS Command Line Interface (AWS CLI) to call the GetFreeTierUsage API.

aws freetier get-free-tier-usage

The response is a JSON document that contains a description of the current usage for each offer that is applicable to this account during this billing period. For simplicity, I only show a few offers here.

{
    "freeTierUsages": [
        {
            "service": "Amazon Simple Queue Service",
            "operation": "",
            "usageType": "Requests",
            "region": "global",
            "actualUsageAmount": 294387.0,
            "forecastedUsageAmount": 679354.6153846154,
            "limit": 1000000.0,
            "unit": "Requests",
            "description": "1000000.0 Requests are always free per month as part of AWS Free Usage Tier (Global-Requests)",
            "freeTierType": "Always Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "",
            "usageType": "EBS:VolumeUsage",
            "region": "global",
            "actualUsageAmount": 9.0,
            "forecastedUsageAmount": 33.0,
            "limit": 30.0,
            "unit": "GB-Mo",
            "description": "30.0 GB-Mo for free for 12 months as part of AWS Free Usage Tier (Global-EBS:VolumeUsage)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances:0002",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 476.0,
            "forecastedUsageAmount": 851.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 225.0,
            "forecastedUsageAmount": 485.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Redshift",
            "operation": "RunComputeNode:0001",
            "usageType": "Node:dc2.large",
            "region": "global",
            "actualUsageAmount": 367.0,
            "forecastedUsageAmount": 735.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free per month during a short-term trial as part of AWS Free Usage Tier (Global-Node:dc2.large)",
            "freeTierType": "Free Trial"
        },
        ...
    ]
}

In the freeTierUsages list, I find some of the most common offers:

Two compute offers for Amazon Elastic Compute Cloud (Amazon EC2).
- The offer with operation RunInstances:0002 is for Windows.
- The offer with operation RunInstances is for Linux.
- The value of the operation property is the same as the platform details and usage operation displayed on the Instances or AMIs pages in the Amazon EC2 console. For more information, see the AMI billing information fields in the Amazon EC2 User Guide.
One storage offer for Amazon Elastic Block Store (Amazon EBS) volumes. This and the two Amazon EC2 compute offers have freeTierType equal to 12 Months Free.
An Always Free offer for Amazon Simple Queue Service (Amazon SQS).
A Free Trial (short-term) offer for Amazon Redshift.

Let’s have a look at some properties of these offers:

description gives a readable explanation of what the offer is about.
freeTierType tells the type of offer: Always Free, 12 Months Free, or Free Trial (short-term).
unit describes the unit used to measure usage for the offer. For example, Hrs (hours) for EC2 instances, GB-Mo (GB per month) for EBS volumes, Requests for Amazon SQS, and so on.

Three interesting properties are the limit of the offer (limit), the actual usage amount of the offer (actualUsageAmount), and the forecasted usage amount (forecastedUsageAmount) at the end of the billing period (the current month). They are all based on the unit used by the offer. For example, the Windows and Linux compute offers each have a limit of 750 hours per month. For the storage offer, the limit is 30 GB per month. For Amazon SQS, the limit of the offer is one million requests per month.

Details on the limits and services provided for free are detailed in each card on the AWS Free Tier page and on the pricing page of each service. The actual and forecast usage amounts provided by the AWS Free Tier API are estimated up to three times per day, similar to AWS Cost and Usage Reports.

If the forecast usage is greater than the limit for the offer, I should expect to switch to pay-as-you-go pricing before the end of the billing period if I continue to use the service in the same way. Actual usage is no longer tracked by the GetFreeTierUsage API once the limit is reached. This means that the actual usage amount cannot be greater than its limit. If that’s the case, the corresponding offer is not returned by the API.

For example, I look for the offers for which the forecast is greater than the limit using the --query option of the AWS CLI:

aws freetier get-free-tier-usage --query 'freeTierUsages[?forecastedUsageAmount > limit]'

{
    "freeTierUsages": [
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "",
            "usageType": "EBS:VolumeUsage",
            "region": "global",
            "actualUsageAmount": 9.0,
            "forecastedUsageAmount": 33.0,
            "limit": 30.0,
            "unit": "GB-Mo",
            "description": "30.0 GB-Mo for free for 12 months as part of AWS Free Usage Tier (Global-EBS:VolumeUsage)",
            "freeTierType": "12 Months Free"
        },
        {
            "service": "Amazon Elastic Compute Cloud",
            "operation": "RunInstances:0002",
            "usageType": "BoxUsage:freetier.micro",
            "region": "global",
            "actualUsageAmount": 476.0,
            "forecastedUsageAmount": 851.0,
            "limit": 750.0,
            "unit": "Hrs",
            "description": "750.0 Hrs for free for 12 months as part of AWS Free Usage Tier (Global-BoxUsage:freetier.micro)",
            "freeTierType": "12 Months Free"
        }
    ]
}

According to this result, if I want to stay within the Free Tier limits, I can check how I use EBS volumes and Amazon EC2 compute with Windows.

For example, I am currently using 476 hours out of the 750 available in a month for Windows EC2 instances. At this pace, I am forecasted to cross the limit and reach about 851 hours. If I am concerned by the costs, I can switch off my Windows instances when not in use or during the night.

Things to know
Previously, the Free Tier API was not publicly available and was used internally for the Free Tier page in the AWS Billing console, where you can find the same data. We hope that making the GetFreeTierUsage API publicly available can help you have fun with AWS, have better use of the AWS Free Tier offers, and be aware of what is free and what to do when you get close to or over a limit.

Using this information, you can build custom reporting that meets your business needs. For example, if you want to avoid compute costs, you can programmatically stop or hibernate your EC2 instances or set the size of an EC2 Auto Scaling group to zero. You can use any of the AWS SDKs to create a web app or integrate this data in a monitoring solution.

More generally, you can send additional emails or notifications (for example, using Amazon SES or Amazon SNS) when the usage of an offer is close to its limit. This can help you get the maximum benefit of an offer without incurring additional costs. You can also do this with AWS Budgets if you set a usage budget amount to the Free Tier limit.

If an offer is no longer applicable to this account (for example, because it expired at the end of the previous month), the corresponding item is not included in the list. If you save the results from previous invocations of the API, you can compare the list of offers with those reported during the previous billing cycle to see which offers have recently expired.

To learn more about keeping track of your AWS Free Tier usage, we created these three 10-minute courses on AWS Skill Builder, an online learning center where you can learn from AWS experts and build cloud skills online:

— Danilo

Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-use-amazon-cloudwatch-to-consolidate-hybrid-multi-cloud-and-on-premises-metrics/

You can now consolidate metrics from your hybrid, multicloud, and on-premises data sources using Amazon CloudWatch and process them in a consistent, unified fashion. You can query, visualize, and alarm on any and all of the metrics, regardless of their source. In addition to giving you a unified view, this new feature will help you to identify trends and issues that span multiple parts and aspects of your infrastructure.

When I first heard about this new feature, I thought, “Wait, I can do that with PutMetricData, what’s the big deal?” Quite a bit, as it turns out. PutMetricData stores the metrics in CloudWatch, but this cool new feature fetches them on demand, directly from the source.

Instead of storing data, you select and configure connectors that pull data from Amazon Managed Service for Prometheus, generic Prometheus, Amazon OpenSearch Service, Amazon RDS for MySQL, Amazon RDS for PostgreSQL, CSV files stored in Amazon Simple Storage Service (Amazon S3), and Microsoft Azure Monitor. Each connector is a AWS Lambda function that is deployed from a AWS CloudFormation template. CloudWatch invokes the appropriate Lambda functions as needed and makes use of the returned metrics immediately — they are not buffered or kept around.

Creating and using connectors
To get started I open the CloudWatch Console, click All metrics, and activate the Multi source query tab, then I click Create and manage data sources:

And then I do it again:

Then I choose a data source type:

CloudWatch will then prompt me for the details that it needs to create and set up the connector for my data source. For example, if I select Amazon RDS – MySQL, I give my data source a name, choose the RDS database instance, and specify the connection info:

When I click Create data source, a Lambda function, a Lambda Permission, an IAM role, a Secrets Manager Secret, a Log Group, and a AWS CloudFormation Stack will be created in my account:

Then, when I am ready to reference the data source and make use of the metrics that it provides, I enter a SQL query that returns timestamps and values for the metric:

Inside the Lambda function
The code for the Custom – getting started template is short, simple, and easy to understand. It implements handlers for two events:

DescribeGetMetricData – This handler returns a string that includes the name of the connector, default values for the arguments to the other handler, and a text description in Markdown format that is displayed in the custom data source query builder in the CloudWatch console.

GetMetricData – This handler returns a metric name, 1-dimensional array of timestamps and metric values, all for a time range that is provided as arguments to the handler.

If you spend a few minutes examining this code you should be able to see how to write functions to connect to your own data sources.

Things to know
Here are a couple of things to keep in mind about this powerful new feature:

Regions – You can create and use data connectors in all commercial AWS Regions; a connector that is running in one region can connect to and retrieve data from services and endpoints in other regions and other AWS accounts.

Pricing – There is no extra charge for the connectors. You pay for the invocations of the Lambda functions and for any other AWS infrastructure that you create.

— Jeff;

Announcing cross-region data replication for Amazon WorkSpaces

2023-11-26 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/cross-region-data-replication-for-amazon-workspaces/

You can now use cross-region data replication to provide business continuity for your Amazon WorkSpaces users. Snapshots are taken every 12 hours, replicated to the desired destination region, and are used to provide a recovery point objective (RPO) of 12-24 hours.

Multi-Region Resilience Review
In her 2022 post Advancing business continuity with Amazon WorkSpaces Multi-Region Resilience, my colleague Ariel introduced you to the initial version of this feature and showed you how to use it to set up standby virtual desktops available for your users. After it has been set up, users log in with Amazon WorkSpaces registration codes that include fully qualified domain names (FQDNs). If the WorkSpaces in the primary region are unavailable, the users are redirected to standby WorkSpaces in the secondary region.

The standby WorkSpaces are available for a small, fixed monthly fee for infrastructure and storage, with a low flat rate change for each hour of usage during the month. Together, this feature and this business model make it easy and economical for you to maintain a standby deployment.

Cross-Region Data Replication
Today we are making this feature even more powerful by adding one-way cross-region data replication. Applications, documents, and other resources stored on the primary WorkSpace are snapshotted every 12 hours and copied to the region hosting the secondary WorkSpace. You get an additional layer of redundancy, enhanced data protection, and can minimize productivity that would otherwise be lost to disruptions. This is particularly helpful if users have installed and configured applications on top of the base image since they won’t have to repeat these steps on the secondary WorkSpace.

Here’s how it all works:

Normal Operation – During normal operation, the users in your WorkSpaces fleet are using the primary region. EBS snapshots of the system (C:) and data (D:) drives are created every 12 hours. Multi-Region Resilience runs in the secondary region and checks for fresh snapshots regularly. When it finds them, it initiates a copy to the secondary region. As the copies arrive in the secondary region they are used to update the secondary WorkSpace.

Failover Detection – As part of the setup process, you will follow Configure your DNS service and setup DNS routing policies to set up DNS routing policies and optional Amazon Route 53 health checks to manage cross-Region redirection.

Failover – If a large-scale event (LSE) affects the primary region and the primary WorkSpace, the failover detection that I just described goes in to effect. When users try to reconnect, they are redirected to the secondary region, the latest snapshots are used to launch a WorkSpace for them, and they are back up and running with access to data and apps that are between 12 and 24 hours old.

Failback – At the conclusion of the LSE, the users manually back up any data that they have created on the secondary WorkSpace and log out of it. Then they log in again, and this time they will be directed to the primary region and WorkSpace, where they can restore their backups and continue to work.

Getting Set Up
As a WorkSpaces administrator, I start by locating the desired primary WorkSpace:

I select it and choose Create Standby WorkSpaces from the Actions menu:

I select the desired region for the secondary WorkSpace and click Next:

Then I choose the right directory in the region, and again click Next:

If the primary WorkSpace is encrypted, I must enter the ARN of the KMS key in the secondary region (or, even better, use a multi-Region key). I check Enable data replication and confirm that I am authorizing an additional monthly charge:

On the next page I review my choices and click Create to initiate the creation of the secondary WorkSpace in the region that I selected.

As mentioned earlier I also need to set up Multi-Region Resilience. This includes setting up a domain name to use as a WorkSpaces registration code, setting up Route 53 health checks, and using them to power routing policies.

Things to Know
Here are a couple of important things to know about Cross-Region Data Replication:

Directories – You can use a self-managed Active Directory, AWS Managed AD or AD Connector configured as described in this post. Simple AD is not supported.

Snapshots – The first-ever EBS snapshot for a particular data volume is full, and subsequent snapshots are incremental. As a result, the first replication for a given WorkSpace will likely take longer than subsequent ones. Snapshots are initiated on a schedule that is internal to WorkSpaces and you cannot control the timing.

Encryption – You can use this feature with Encrypted WorkSpaces as long as you use the same AWS Key Management Service (AWS KMS) keys in the primary and secondary regions. You can also use multi-Region keys.

Bundles – You can use the Windows 10 and Windows 11 bundles, and you can also BYOL.

Accounts – Every AWS account has a fixed limit on the number of pending EBS snapshots. This may affect your ability to use this feature with large fleets of WorkSpaces.

Pricing – You pay a fixed monthly fee based on the amount of storage configured for each primary WorkSpace. See the Amazon WorkSpaces Pricing page for more information.

— Jeff;

Noise

Tag Archives: news

Automatic restore testing and validation now available in AWS Backup

Amazon CodeWhisperer offers new AI-powered code remediation, IaC support, and integration with Visual Studio

FlexGroup Volume Management for Amazon FSx for NetApp ONTAP is now available

New – Scale-out file systems for Amazon FSx for NetApp ONTAP

Introducing shared VPC support for Amazon FSx for NetApp ONTAP

Announcing on-demand data replication for Amazon FSx for OpenZFS

IAM Access Analyzer updates: Find unused access, check policies before deployment

Amazon EKS Pod Identity simplifies IAM permissions for applications on Amazon EKS clusters

New Amazon WorkSpaces Thin Client provides cost-effective, secure access to virtual desktops

Detect runtime security threats in Amazon ECS and AWS Fargate, new in Amazon GuardDuty

Introducing Amazon EC2 high memory U7i Instances for large in-memory databases (preview)

Amazon Detective adds new capabilities to accelerate and improve your cloud security investigations

Top announcements of AWS re:Invent 2023

Use natural language to query Amazon CloudWatch logs and metrics (preview)

Increase collaboration and securely share cloud knowledge with AWS re:Post Private

Use anomaly detection with AWS Glue to improve data quality (preview)

Mutual authentication for Application Load Balancer reliably verifies certificate-based client identities

Check your AWS Free Tier usage programmatically with a new API

Use Amazon CloudWatch to consolidate hybrid, multicloud, and on-premises metrics

Announcing cross-region data replication for Amazon WorkSpaces

The collective thoughts of the interwebz