Tag Archives: news

AWS Database Migration Service now automates time-intensive schema conversion tasks using generative AI

2024-12-01 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/aws-data-migration-service-improves-database-schema-conversion-with-generative-ai/

Starting today, AWS Database Migration Service Schema Conversion (AWS DMS SC) introduces a new capability to improve the database schema conversion experience by automatically converting up to 90 percent of schema objects from commercial databases to PostgreSQL migrations.

AWS DMS is a cloud service that makes it possible to migrate relational databases, data warehouses, NoSQL databases, and other types of data stores. You can use AWS DMS to migrate your data into the Amazon Web Services (AWS) Cloud or between combinations of cloud and on-premises setups.

Today, more than 1 million databases have been migrated using AWS Database Migration Service. AWS DMS helps you migrate your data from one database system to another. And, when migrating between different database engines, AWS DMS SC helps to convert the source database schema and procedures to the target database system.

However, although AWS DMS SC automates many steps in these migrations, certain complex database code elements still require manual intervention, which can extend migration timelines and add cost. This is particularly the case with proprietary system functions or procedures, and data type conversions, which don’t always have direct equivalents in PostgreSQL.

The new generative AI capability in AWS DMS SC is designed to address these challenges by automating some of the most time-intensive schema conversion tasks. Using large language models (LLMs) hosted on Amazon Bedrock, the new capability expands the existing conversion capabilities. It converts code snippets in the source database that were otherwise not supported by traditional rule-based techniques, including complex procedures and functions.

Generative AI–assisted code conversion helps to reduce migration costs and accelerate project timelines. Because AWS DMS SC automates more of the schema conversion process, you can focus on higher value tasks such as refining and optimizing your applications post-migration rather than manually resolving conversion gaps. Our beta customers have already experienced success with these AI-powered features in AWS DMS SC, achieving cost savings and faster migrations.

Let’s find out how it works
To demonstrate the ease of using this new generative AI capability, I’ll walk through the schema conversion process in AWS DMS SC. AWS DMS SC simplifies database migration by automatically converting my source database’s structure, including tables, views, stored procedures, functions, and more, to a format compatible with my target database. Any objects that can’t be automatically converted are flagged for manual attention.

I start with a self-managed commercial database running on Amazon Elastic Compute Cloud (Amazon EC2). I use the AWS Management Console to define the instance profile and the data providers. This is where I configure the replication instance network details, the database engine and its endpoint, the secret where the database password is securely stored, and more. I also create a migration project. These steps aren’t new, and you can refer to Accelerate your database migration journey using AWS DMS Schema Conversion in the AWS Database Blog to learn about the details.

After my project is created, I select it, and on the Schema conversion tab, I choose Launch schema conversion. It takes a couple of minutes to launch the conversion tool the first time.

AWS DMS SC with generative AI is an opt-in capability. I first activate the option. On the Settings tab, I turn on Enable Generative AI feature for conversion.

Before diving into the details of the conversion, I would like to get an overall assessment of the migration complexity. I select the schema I want to migrate. Then I select Assess in the menu.

After a few minutes, a high-level Summary is available. The Action items tab has more details. I choose Export results and choose PDF to receive a report to share with my colleagues. The report is generated and available from an S3 bucket.

The summary screen shows the percentage of Database storage objects and Database code objects that can be converted by the rule-based method. That’s 100% and 57% in this example. Let’s see how the generative AI-based conversion will change that.

The PDF contains an executive summary, various statistics about the number of objects to be migrated, the feasibility of conversion with generative AI, and the complexity of the migration.

By reading the report, I learn there is no blocker detected to migrate the stored procedures. I select the stored procedure I want to migrate (PRC_AIML_DEMO6). Then, I select the Actions menu on the source database (the left one) and choose Convert.

After a minute or two, I can read the original procedure code in the left pane and the proposed migrated version on the right panel.

The summary screen has been updated. Now, it shows that 100 percent of the code can be converted automatically.

I can edit the code and make changes as required. When I’m comfortable with the proposed new version, I select the Actions menu on the target database side (the right one) and choose Apply changes.

With this new generative AI capability, AWS DMS SC can automatically convert up to 90 percent of schema objects from commercial databases to PostgreSQL.

To support your compliance requirements, this capability is initially turned off, and you can enable it as needed. If you choose to use the generative AI features in AWS DMS SC, it will flexibly decide between traditional rule-based methods and generative AI based on the complexity of the objects being converted. Customers with strict policies against generative AI can continue to rely solely on the rule-based approach, with any unconverted or partially converted objects requiring manual adjustments.

Availability and pricing
This new capability is available today in the following AWS Regions: US East (Ohio, N. Virginia), US West (Oregon), and Europe (Frankfurt).

AWS DMS Schema Conversion with generative AI provides you with a faster migration pathway and helps you accelerate your transition to AWS.

To get started, visit the AWS DMS Schema Conversion documentation and learn how this generative AI capability can simplify your next database migration.

— seb

Simplify governance with declarative policies

2024-12-01 Esra Kayabali

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/simplify-governance-with-declarative-policies/

Today, I am happy to announce declarative policies, a new capability that helps you declare and enforce desired configuration for a given AWS Service at scale across your organization.

It is common for customers to create standards within their organizations for how cloud resources should be configured. For example, they might require blocking public access for Amazon EBS snapshots. They want these standards to be defined once centrally and enforced across all their accounts, including those that join the organization in the future. Additionally, whenever a cloud operator attempts to configure a resource in a way that does not meet the standard, they want that operator to receive a useful, actionable error message that explains how to remediate the configuration.

Declarative policies address these challenges by helping you to define and enforce desired configuration for AWS services with a few clicks or commands. You can select the configuration you want such as “block public access for VPCs” and AWS will automatically ensure that the desired state is enforced across your multi-account environment (or parts of it) once you attach the policy. This approach reduces the complexity of achieving the desired configuration. Once the configuration is set, it is maintained, even as new features or new APIs are added. Additionally, with declarative policies, administrators have visibility into the current state of service attributes across their environment, and – unlike access control policies, which cannot leak information to those without permissions – end users see custom error messages configured by their organization’s administrators, redirecting them to internal resources or support channels.

“ABSA Group operates in a heavily regulated environment and as we adopt more services, we use SCP policy exclusions to restrict actions and Config rules to detect violations. However, we must create an exception for every new API or feature. With declarative policies, we can simply set VPC Block Public Access to true and have peace of mind that no users, service-linked roles, or future APIs can facilitate public access in our AWS Organizations.” explains Vojtech Mencl, Lead Product Engineer at ABSA, a multinational banking and financial services conglomerate based in Johannesburg, South Africa.

“With custom error messages, we can easily redirect end users to an internal portal for more information on why their action failed. This drastically reduces the operational complexity for governance and accelerates our migration to AWS.” says Matt Draper, Principal Engineer at ABSA.

At this launch, declarative policies supports Amazon Elastic Compute Cloud (Amazon EC2), Amazon Virtual Private Cloud (Amazon VPC), and Amazon Elastic Block Store (Amazon EBS) services. Available service attributes include enforcing IMDSv2, allowing troubleshooting though serial console, allowed Amazon Machine Image (AMI) settings, and blocking public access for Amazon EBS snapshots, Amazon EC2 AMI, and VPC. When new accounts are added to an organization, they will inherit the declarative policy applied at the organization, organizational unit (OU) or account level.

You can create declarative policies through the AWS Organizations console, AWS Command Line Interface (AWS CLI), AWS CloudFormation, or through AWS Control Tower. Policies can be applied at the organization, OU, or account level. When attached, declarative policies prevent non-compliant actions regardless of whether they were invoked using an AWS Identity and Access Management (IAM) role you created or by an AWS service using a service-linked role.

Getting started with declarative policies
To demonstrate declarative policies, I will walk you through an example. Let’s say that as the security administrator for a large enterprise with hundreds of AWS accounts, I’m responsible for maintaining our organization’s strict security posture. At our company, we have several critical security requirements: we maintain tight control over internet access across all our networks, we only allow AMIs from specific trusted providers, and we need to ensure that no VPC resources are accidentally exposed to the public internet. With declarative policies, I can implement these requirements efficiently. Let me show you how I set this up in my environment.

I go to the AWS Organizations console and choose Policies in the navigation pane. I choose Declarative policies for EC2 under the Supported policy types.

I choose Enable declarative policies for EC2 to enable the feature.

After declarative policies is enabled, I can define and enforce desired configurations for EC2 across all of the accounts in my AWS Organizations.

Before I create declarative policies, as the organization’s administrator, I want to understand the current status of my AWS environment using the account status report, which is a feature of declarative policies. The report offers both a summary view and a detailed CSV file, covering all accounts and AWS Regions within a selected organizational scope. It helps me assess readiness before attaching a policy.

On the next page, I choose Generate status report. I choose an Amazon Simple Storage Service (Amazon S3) bucket under Report S3 URI and choose accounts and OUs to include in the report scope.

Note that the S3 bucket requires the following policy attached to it in order to store the status report:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DeclarativePoliciesReportBucket",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "report.declarative-policies-ec2.amazonaws.com"
                ]
            },
            "Action": [
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::<bucketName>/*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceArn": "arn:<partition>:declarative-policies-ec2:<region>:<accountId>:*"
                }
            }
        }
    ]
}

I choose Submit.

When complete, the report is stored in the Amazon S3 bucket I specified. On the View account status report page, I can choose between multiple reports from Reports dropdown to observe what is the current status of various attributes.

I check the Amazon S3 bucket I provided to store a CSV file, which provides the detailed readiness report. I observe what my current state is across my organization unit across different regions.

After I assess the account status, I continue to create a policy. I choose Create policy on the Declarative policies for EC2 page.

On the next page, I enter a Policy name and optionally a Policy description.

In this demo, I use Visual Editor to show how to add service attributes. These attributes include Serial Console Access, Instance Metadata Defaults, Image Block Public Access, Snapshot Block Public Access, VPC Block Public Access, and Allowed Image Settings. I can use JSON Editor to add them manually or to observe the policies I added using Visual Editor. First, I choose VPC Block Public Access to control internet access for resources in my VPC from internet gateways. I choose Block ingress under Internet gateway state. When enabled, this immediately prevents public access without mutating resources and can be rolled back.

As a second attribute, I choose Allowed Image Settings to control the allowed images criteria for AMIs. This is useful because I can ensure all instance launches use a golden AMI that an account or set of accounts generates in my organization, or one provided by a vendor like Amazon or Ubuntu. I choose Enabled under Allowed Image Settings. I choose amazon under Provider. Declarative policies provides transparency with customizable error messages to help reduce end-user frustration. You can optionally add a Custom error message to be displayed when organization members are unable to perform a restricted action. To complete the process of generating the policy, I choose Create policy.

Now, I need to attach the policy to my organization or specific OUs. I choose Attach policy under Actions.

I choose my organization or specific OUs and choose Attach policy.

When an account joins an organization or an OU, the declarative policy attached to it takes immediate effect and all subsequent non-compliant actions will fail (except for VPC Block Public Access, which will immediately curtail public access). Existing resources in the account will not be deleted.

Now available
Declarative policies streamlines governance for AWS customers by reducing policy maintenance overhead, providing consistent enforcement across accounts, and offering transparency to administrators and end users.

Declarative policies are now available in AWS commercial, China and AWS GovCloud (US) Regions.

To learn more about declarative policies and start enforcing them in your organization, visit the declarative policies documentation.

— Esra

AWS Verified Access now supports secure access to resources over non-HTTP(S) protocols (in preview)

2024-12-01 Sébastien Stormacq

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/aws-verified-access-now-supports-secure-access-to-resources-over-non-https-protocols/

AWS Verified Access provides secure access to your corporate applications and resources without a virtual private network (VPN). We launched Verified Access in preview at re:Invent 2 years ago as a way to provide secure, VPN-less access to corporate applications, enabling organizations to manage network access based on identity and device security instead of IP addresses, which increases control and security over application access.

Today, Verified Access is launching a preview of its secure, VPN-less access capabilities to non-HTTP(S) applications and resources, enabling zero trust access to corporate resources over protocols such as Secure Shell (SSH) and Remote Desktop Protocol (RDP).

Organizations increasingly require secure, remote access to internal resources such as databases, remote desktops, and Amazon Elastic Compute Cloud (Amazon EC2) instances. Traditional VPN solutions, although effective for network access, often grant broad privileges and don’t support granular access controls, which can expose infrastructure with sensitive data. Although some organizations use bastion hosts to mediate access, this approach can create complexity and policy inconsistencies across HTTP(S) and non-HTTP(S) applications. With the rise of zero trust architectures, these gaps highlight the need for a secure access solution that extends consistent access policies across all applications and resources.

Verified Access addresses these needs by providing zero trust access controls for your corporate applications and resources. By supporting protocols such as SSH, RDP, or Java Database Connectivity (JDBC) or Open Database Connectivity (ODBC), Verified Access simplifies your security operations. Now, you can establish uniform, context-aware access policies across your corporate applications and resources. Verified Access evaluates each access request in real time, making sure access is granted only to users who meet specific identity and device security requirements. Additionally, it eliminates the need for separate VPNs or bastion hosts, streamlining operations and reducing the risk of over-privileged access.

One of my favorite capabilities is onboarding a group of resources by specifying their IP Classless Inter-Domain Routing (CIDR) and ports, rather than onboarding one resource at a time. Verified Access automatically creates DNS records for each active resource within the specified CIDR range. This eliminates the need for manual DNS configuration and users can therefore connect to new resources instantly.

Using Verified Access for non-HTTPS access
Configuring Verified Access for non-HTTPS access isn’t very different from what exists today. You can read the blog post I wrote for the launch of the preview 2 years ago or the Get started with Verified Access tutorial to learn how to get started.

Verified Access proposes two new types of endpoint targets: a target for one single resource and a target for multiple resources.

With the network interface, load balancer, or RDS endpoint target you can provide access to an individual resource such as an Amazon Relational Database Service (Amazon RDS) instance or an arbitrary TCP application fronted by a Network Load Balancer or an elastic network interface. This type of target endpoint is defined by a combination of a target type (such as a load balancer or a network interface) and a range of TCP ports. Verified Access will provide a DNS name for each endpoint upon its creation. A Verified Access DNS name is assigned for each target. This is the name end users will use to securely access the resource.

With network CIDR endpoint target, the resources are defined using an IP CIDR and port range. Through this type of endpoint target, you can easily provision secure access to ephemeral resources such as EC2 instances over protocols such as SSH and RDP. This is done without having to perform any actions such as creating or deleting endpoint targets each time a resource is added or removed. As long as these resources are assigned an IP address from the defined CIDR, Verified Access provides a unique public DNS record for each active IP detected in the defined CIDR.

Here is a diagram of the setup for this demo.

Part 1: As a Verified Access administrator

As a Verified Access administrator, I create the Verified Access instance, trust provider, access group, endpoint, and access policies, allowing access by the end user to the SSH server.

For this demo, I configure a Verified Access network CIDR endpoint target. I select TCP as Protocol and Network CIDR as Endpoint type. I make sure the CIDR range is within the one of the VPC where my target resources are. I select the TCP Port ranges and the Subnets within the VPC.

This is a good moment to stretch your legs and refill your cup of coffee, it takes a few minutes to create the endpoint.

Once, the status is Active, I launch an EC2 instance in a private Amazon Virtual Private Cloud (Amazon VPC). I enable SSH and configure the instance’s security group to only access requests coming from the VPC. A few minutes later, I can see the instance IP has been detected and assigned a DNS name to connect to from the Verified Access client application.

I also have the option during the configuration to delegate my own DNS subdomain, such as secure.mycompany.com, and Verified Access will assign DNS names for the resources within that subdomain.

Create an access policy

At this stage, there is no policy defined on the Verified Access endpoint. It will deny every request by default.

On the Verified Access groups page, I select the Policy tab. Then I select the Modify Verified Access endpoint policy button to create an access policy.

I enter a policy allowing anybody who is authenticated and has an email address ending with @amazon.com. This is the email address I used for the user defined in AWS IAM Identity Center. Note that the name after context is the name I entered as Policy reference name when I created the Verified Access trust provider. The documentation page has the details of the policy syntax, the attributes, and the operators I can use.

permit(principal, action, resource)
when {
    context.awsnewsblog.user.email.address like "*@amazon.com"
};

After a few minutes, Verified Access updates the policy and becomes Active again.

Distribute the configuration to clients

The last task as a Verified Access administrator is to extract the JSON configuration file of the client applications.

I retrieve the client application configuration file with the AWS Command Line Interface (AWS CLI). As a system administrator, I’ll distribute this configuration to each client machine.

aws ec2 export-verified-access-instance-client-configuration \
     --verified-access-instance-id "vai-0dbf2c4c011083069"

{
    "Version": "1.0",
    "VerifiedAccessInstanceId": "vai-0dbf2c4c011083069",
    "Region": "us-east-1",
    "DeviceTrustProviders": [],
    "UserTrustProvider": {
        "Type": "iam-identity-center",
        "Scopes": "verified_access_test:application:connect",
        "Issuer": "https://identitycenter.amazonaws.com/ssoins-xxxx",
        "PkceEnabled": true
    },
    "OpenVpnConfigurations": [
        {
            "Config": "Y2...bWU=",
            "Routes": [
                {
                    "Cidr": "2600:1f10:4a02:8700::/57"
                }
            ]
        }
    ]
}

Now that I have a resource to connect to and the Verified Access infrastructure in place, let me show you the end user experience to access a network endpoint.

Part 2: As an end user

As the end user, I receive a link to download and install the Verified Access Connectivity Client application. We support Windows and macOS clients at the time of this writing.

I install the configuration file I received from my administrator. I use ClientConfig1.json as the file name and I copy the file to C:\ProgramData\AWSPylon on Windows or /Library/Application Support/com.aws.pylon.client on macOS.

This is the same configuration file for all users, and the system administrator might push the file to all client machines using an endpoint management tool.

I start the Connectivity Client application. I choose Sign in to start the authentication sequence.

The authentication opens my web browser on the authentication page of my identity provider. The exact screen and login sequence varies from one provider to the other. After I’m authenticated, the Connectivity Client creates the secure tunnel to access my resource, an EC2 instance for this demo.

Once the status is Connected, I can securely connect to the resource, using the DNS name provided by Verified Access. In a terminal application, I type the ssh command to start the connection.

For this demo, I configured a delegated DNS domain secure.mycompany.com for Verified Access. The DNS address I received for the EC2 instance is 10-0-1-199.awsnews.secure.mycompany.com.

$ ssh -i mykey.pem [email protected]

   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
Last login: Sat Nov 17 20:17:46 2024 from 1.2.3.4

$

Availability and pricing
Verified Access is available as a public preview in 18 AWS Regions: US East (Ohio, N. Virginia), US West (N. California, Oregon), Asia Pacific (Jakarta, Mumbai, Seoul, Singapore, Sydney, Tokyo), Canada (Central), Europe (Frankfurt, Ireland, London, Milan, Stockholm), Israel (Tel Aviv), and South America (São Paulo).

You’re charged for each hour that your non-HTTP(S) Verified Access endpoint remains active and per connection. The first 100 connections per month on each Verified Access endpoint are free. For more information, refer to AWS Verified Access Pricing.

With Verified Access for HTTP(S) and non-HTTP(S) applications you can unify the access controls to your private applications and systems and apply zero trust policies uniformly to all applications, and SSH, RDP, and HTTP(S) resources. It reduces the complexity of your network infrastructure and helps you to implement zero-trust access to your applications and resources. Finally, it adapts to your growing infrastructure, automating DNS setup and supporting large-scale deployments without resource-specific registration.

Go, try Verified Access today, and share your feedback with the team!

— seb

Announcing AWS Transfer Family web apps for fully managed Amazon S3 file transfers

2024-12-01 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/announcing-aws-transfer-family-web-apps-for-fully-managed-amazon-s3-file-transfers/

Today I would like to introduce you to AWS Transfer Family web apps, the newest AWS Transfer Family resource. You can create a fully managed, no-code web app that allows authenticated users to list, upload, download, copy, and delete files in specific Amazon Simple Storage Service (Amazon S3) buckets. Non-developer, line-of-business users inside and outside of your organization can easily exchange file data without the need for desktop clients, scripts, faded instructions on sticky notes, or local IT help.

As the web apps administrator, you get full control over authentication, access, and permissions, and can customize the web app with a page title and a favicon. Here is the web app that I created while writing this blog post:

I can click files to download them, click folders to open them, and click columns to sort. The vertical ellipses menu provides additional options:

Each web app supports uploading and downloading of files up to 160 GiB in size, and uses multipart uploads for large files. Files are transferred across HTTPS connections protected by TLS, with automatic retries and a CRC32 end-to-end integrity check.

All about Transfer Family web apps
I will show you how to create your own web app in just a minute. But first, let’s take a look at some of the essential features and benefits…

Security – Transfer Family web apps use AWS IAM Identity Center, allowing you to use your existing SAML or OIDC identity provider or the built-in Identity Store. Either way, you can use S3 Access Grants to exercise full, fine-grained control over the users and groups that are allowed to see, download, delete, and upload files and to create directories. Your organization can also benefit from AWS Transfer Family’s compliance with SOC, PCI DSS, FedRAMP, HIPAA, and other programs.

Customization – You can customize each Transfer Family web app with a page title and a favicon. You can also put a Amazon CloudFront distribution in front of the web app and host it at a custom domain name, with HTTPS access and a public certificate.

AWS Ecosystem – Transfer Family web apps are hosted on AWS and as such are scalable and highly available. All files are stored in designated S3 buckets, with eleven nines (99.999999999%) of durability. You can take advantage of S3 features including S3 Versioning, S3 server access logging, S3 Event Notifications, and more. You can also use Amazon EventBridge to orchestrate complex post-upload workflows.

Creating a Transfer Family web app
Let’s go through the steps to create a Transfer Family web app. Each web app exists in a specific AWS Region, so I open the AWS Transfer Family console, choose the desired Region (us-east-2 for this post), and select Web apps on the left:

Then I click Create web app to proceed:

I connect to my IAM Identity Center if necessary, then create or choose an IAM service role (details) that allows the Transfer Family web app to access S3 and S3 Access Grants:

I add a Name tag and set the maximum number of concurrent web app users, then click Next:

Now I design my web app, setting the page title and the logo (both optional) before clicking Next:

On the next page I review my settings and click Create to move ahead:

And my web app is created and almost ready to use (I still need to set up permissions and users):

I will use the Access endpoint in the CORS policy that I will soon create for the bucket associated with the web app, so I copy and save it.

Setting Permissions and Users
I create an IAM custom trust policy that provides the necessary read and write permissions to the S3 bucket(s) that will be accessible through my web app (details). This policy will be referenced in an S3 Access Grant that I will create in a minute:

Moving right along, I create the initial set of users and groups in IAM Identity Center (I can add more later):

Next, I create an S3 bucket in the same region as the web app and create an S3 Access Grant. Each S3 Access Grant allows a particular IAM Identity Center identity (a user or a group) to access a specific scope (a bucket or a prefixed part of a bucket) for reading and/or writing:

I also need to attach a CORS policy (details) to the bucket so that the web app is allowed to access it from the browser:

The final step is to associate the users with the new web app. I return to the AWS Transfer Family Web apps page, find my app, and click Assign users and groups:

I can add new users to my directory or pick existing ones:

I’ll add myself to start:

Once assigned, I can share the Access endpoint (as seen above) with the user and they (me, in this case) can log in to the web app:

The Web app endpoint and the Access endpoint are the same by default. If you set up a CloudFront distribution for your web app, the Access endpoint will reflect the URL of the endpoint.

I have shown you the express path through the setup process. As you probably noticed, there are lots of options to control read and write access at the individual and group level. Be sure to explore and fully understand all of these options before you set up your production web app!

Things to Know
Here are a couple of things to know about S3 Transfer Family web apps:

Regions – Web apps can be created in nine AWS Regions; check out the web app documentation for a current list.

Pricing – Pricing is per web app/hour.

API and CLI – You can create and manage web apps programmatically by using create-web-app, describe-web-app, and other AWS Transfer Family actions.

Storage Browser for S3 – Transfer Family web apps are built using Storage Browser for Amazon S3 and offer the same end-user functionality in a fully managed offering.

Getting Started – You can get started with Transfer Family web apps in the Transfer Family console.

— Jeff;

Introducing Amazon OpenSearch Service and Amazon Security Lake integration to simplify security analytics

2024-12-01 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-amazon-opensearch-service-zero-etl-integration-for-amazon-security-lake/

Today, we’re announcing the general availability of Amazon OpenSearch Service zero-ETL integration with Amazon Security Lake. This integration enables organizations to efficiently search, analyze, and gain actionable insights from their security data, streamlining complex data engineering requirements and unlocking the full potential of security data. It’s a new way to in-place query and analyze logs in Security Lake that minimizes the need to duplicate data and reduces the operational overhead of managing custom data pipelines. You can directly query your Security Lake data, saving the costs of moving data.

With OpenSearch Service zero-ETL integration with Security Lake, you can use the rich analytics capabilities of OpenSearch Dashboards to query and visualize your data in Security Lake. You can also analyze multiple data sources within a single tool and a single schema, the Open Cybersecurity Schema Framework (OCSF) schema to help with threat-hunting and investigation scenarios.

For time-sensitive investigations and monitoring, you can optionally boost query performance by enabling additional accelerations such as indexed views and dashboards in Amazon OpenSearch Service when you need fast and frequent access to a subset of your data. These capabilities provide complete visibility into all your data stored in Security Lake, regardless of the log volume, to support security investigations, better understanding of your security posture, and gain security-relevant insights.

Getting started with direct queries with Amazon Security Lake
You can get started in a few steps. First, you need to enable Security Lake by creating a Security Lake subscriber. Then, you enable a data connection in Amazon OpenSearch Service. This will automatically create an OpenSearch Serverless collection to store your direct query results and indices.

1. Enable Security Lake and setup permissions for a data lake

To enable Security Lake in the AWS Management Console, specify the data sources that you want to collect such as Amazon Route 53 DNS queries, AWS CloudTrail logs, Amazon VPC Flow logs, and AWS Security Hub findings and your AWS Regions. I chose several Regions and set the Amazon Simple Storage Service (Amazon S3) storage class and roll-up Regions to consolidate data.

Security Lake offers a 15-day trial at no cost so you can deploy it across your organization with the desired data sources and estimate the costs specific to your organization.

Once the enablement is complete, all collected data is ingested into an Amazon Simple Storage Service (Amazon S3) bucket in your account.

To access Security Lake data from an account other than the Security Lake delegated admin account, you should create an AWS Lake Formation subscriber to access and query data from AWS Glue tables associated with Security Lake. Enter the AWS account and external ID that’s authorized to access Security Lake and select the data sources to be accessed. Lake Formation provides cross-account permissions for security analysts to access data in the lake.

After you create the query subscriber, you can go to the account where you plan to deploy your OpenSearch resources and accept the AWS Resource Access Manager (AWS RAM) share that is shared by the Security Lake delegated admin account. The subscriber account will show the share status as pending until it’s manually accepted.

To learn more, visit Enabling Security Lake using the console and Create query subscriber procedures in the Amazon Security Lake User Guide.

2. Create a data connection with OpenSearch Service

You can create a zero-ETL integration in a few steps. In the OpenSearch Service console of the subscriber’s account, choose Connected data source in the Data connections section of the left navigation pane. You can then choose Security Lake as a data source type.

In the next step, you can set up the IAM permissions for accessing the Security Lake data source using the zero-ETL integration. It will also automatically create an OpenSearch Serverless collection and an OpenSearch application.

After the connection is created, you can select one of the pre-built OpenSearch dashboards that periodically query your data in Security Lake to create visualizations. You can create a dashboard using templates for VPC Flow Logs, WAF logs, and CloudTrail data sources in Security Lake.

The following is an example of a pre-built dashboard for VPC Flow logs.

To learn more about data connection, visit Data connections and permissions in the Amazon OpenSearch Service Developer Guide.

3. Query Security Lake data in the OpenSearch Dashboard

To directly query your Security Lake data in OpenSearch Dashboards, go to the Discover page.

In the Discover page, you can use the data picker workflow to locate on a specific Security Lake table to query. There is one table for each Security Lake log source.

After making a selection, you can choose the query language that you want to use, either PPL (Piped Processing Language) or SQL (Structured Query Language), and then write and run your query. The following is a PPL sample result:

You can also choose to search and run a pre-built query template to start your query. There are more than 200 SQL and PPL queries that cover all AWS log sources that are available in Security Lake. You can use the search box to find queries that you’re interested in. For example, search for “VPC Flow” to see all queries related to VPC Flow logs. There’s a description explaining each query and when you might want to use it.

If you want to perform multiple queries on the same data set, for example to support security investigations, you can create an on-demand indexed view for the results of your direct query. After the results are ingested into an OpenSearch index, you can perform low-latency subsequent queries and analysis using analytics features in OpenSearch.

To create an indexed view, choose Create indexed view and select a specified query, an index name, and a time range. After the view is created, the query results will be ingested and available to query as part of the newly created index under available indexed views.

To learn more, visit Searching data in the Amazon OpenSearch Service Developer Guide.

Now available
Amazon OpenSearch Service zero-ETL integration with Amazon Security Lake is now available in the US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Mumbai), Asia Paciﬁc (Singapore), Asia Paciﬁc (Sydney), Asia Paciﬁc (Tokyo), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), South America (São Paulo), and Canada (Central) AWS Regions.

OpenSearch Service separately charges for only the compute needed (as OpenSearch Compute Units) to query your external data in addition to maintaining indexes in OpenSearch Service. For more information, see Amazon OpenSearch Service Pricing.

Give it a try and send feedback to the AWS re:Post for Amazon OpenSearch Service or through your usual AWS Support contacts.

— Channy

Use your on-premises infrastructure in Amazon EKS clusters with Amazon EKS Hybrid Nodes

2024-12-01 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/use-your-on-premises-infrastructure-in-amazon-eks-clusters-with-amazon-eks-hybrid-nodes/

Today, we’re announcing the general availability of Amazon Elastic Kubernetes Service (Amazon EKS) Hybrid Nodes, a new feature that you can use to attach your on-premises and edge infrastructure as nodes to EKS clusters in the cloud.

With Amazon EKS Hybrid Nodes, you can unify Kubernetes management across cloud and on-premises environments and take advantage of the scale and availability of Amazon EKS in all the places your applications need to run. You can use your existing on-premises hardware, while oﬄoading the responsibility for managing Kubernetes control planes to EKS and conserving on-premises capacity for your workloads. Using Amazon EKS Hybrid Nodes, you can adopt consistent operational practices and tooling across your cloud and on-premises environments.

Amazon EKS Hybrid Nodes expands our support for hybrid Kubernetes deployments, adding to Amazon EKS on AWS Outposts and Amazon EKS Anywhere, which we introduced previously. You can compare how Kubernetes and hardware components are managed with each of the EKS hybrid deployment options.

Component	EKS on Outposts	EKS Hybrid Nodes	EKS Anywhere
Hardware	Managed by AWS	Managed by customer
Kubernetes control plane	Hosted and managed by AWS		Hosted and managed by customer
Kubernetes nodes	Amazon EC2	Customer-managed physical or virtual machines

When you use Amazon EKS Hybrid Nodes to attach your on-premises and edge infrastructure to EKS clusters, you can use other Amazon EKS features and integrations, including Amazon EKS add-ons, Pod Identities, cluster access entries, cluster insights, and extended Kubernetes version support. Amazon EKS Hybrid Nodes inherently integrates with AWS services including AWS Systems Manager, AWS IAM Roles Anywhere, Amazon Managed Service for Prometheus, Amazon CloudWatch, and Amazon GuardDuty for centralized monitoring, logging, and identity management.

Get started with Amazon EKS Hybrid Nodes
Here are steps to use Amazon EKS Hybrid Nodes. First, create an EKS cluster and specify your on-premises node and pod subnets. After setting up network connectivity and AWS Identity and Access Management (AWS IAM) permissions for your on-premises environment, run the Amazon EKS Hybrid Nodes CLI (nodeadm) on each host that will join the cluster. When hybrid nodes join your cluster, required networking components, such as kube-proxy and CoreDNS, are automatically installed. Before your hybrid nodes become ready to serve applications, you must install a compatible Container Network Interface (CNI) driver. The Cilium and Calico CNI drivers are supported for use with Amazon EKS Hybrid Nodes.

1. Prerequisites

You must have certain prerequisites in place before your on-premises infrastructure can join your EKS cluster as hybrid nodes, including the following:

Hybrid network connectivity from your on-premises environment to and from AWS using with AWS Site-to-Site VPN, AWS Direct Connect, or another virtual private network (VPN) solution
A virtual private cloud (VPC) with routes in its routing table for your on-premises node and, optionally, pod networks, with your virtual private gateway (VGW) or transit gateway (TGW) as the target
Infrastructure in the form of physical or virtual machines
Operating system that is compatible with hybrid nodes
Either AWS IAM Roles Anywhere or AWS Systems Manager set up to authenticate your hybrid nodes with the control plane
An EKS cluster IAM role and an EKS Hybrid Nodes IAM role

You can use Amazon Linux 2023, Ubuntu 20.04, Ubuntu 22.04, Ubuntu 24.04, or Red Hat Enterprise Linux (RHEL) 8 and 9 as the node operating system for your hybrid nodes. AWS supports the hybrid nodes integration with these operating systems but doesn’t provide support for the operating systems themselves. You’re responsible for operating system provisioning and management.

To learn more, visit Prerequisites for EKS Hybrid Nodes in the Amazon EKS User Guide.

2. Create EKS cluster and enable hybrid nodes

Go to the Amazon EKS console and start to create your EKS cluster. In the Step 2 Specify networking screen, turn on Specify the CIDR blocks for your on-premises environments that you will use for hybrid nodes in the Configure remote networks to enable hybrid nodes option.

The Classless Inter-Domain Routing (CIDRs) of remote nodes and pods need to be RFC-1918 IPv4 IPv4 addresses, and they can’t overlap with the VPC CIDR or the EKS cluster Kubernetes service CIDR. Additionally, the remote node CIDR and the remote pod CIDR can’t overlap. Specifying a pod CIDR block is required if you will run webhooks on your nodes or if your CNI doesn’t use NAT for pod addresses as pod traffic leaves your nodes.

You can also create an EKS cluster using AWS Comand Line Interface (AWS CLI), eksctl, and AWS CloudFormation. To enable your cluster for Amazon EKS Hybrid Nodes, use the remote-network-config flag to specify your remote node and, optionally, your remote pod CIDR blocks.

$ aws eks create-cluster --name channy-hybrid-cluster --region=us-east-1 \
    --role-arn arn:aws:iam::012345678910:role/eks-cluster-role \
    --resources-vpc-config subnetIds=subnet-1234a11a,subnet-5678b11b \
    --remote-network-config \
{"remoteNodeNetworks":[{"cidrs":["10.80.0.0/16"]}],"remotePodNetworks":[{"cidrs":["10.85.0.0/16"]}]}}

Your cluster must be configured with API or API_AND_CONFIG_MAP cluster access authentication modes. Create an Amazon EKS access entry for your EKS Hybrid Nodes IAM role to enable nodes to join the cluster.

$ aws eks create-access-entry \
  --cluster-name my-hybrid-cluster \
  --principal-arn arn:aws:iam::012345678910:role/eksHybridNodesRole \ 
  --type HYBRID_LINUX

Amazon EKS Hybrid Nodes use temporary IAM credentials provisioned by AWS Systems Manager hybrid activations or AWS IAM Roles Anywhere to authenticate with the EKS cluster. Before connecting your on-premises nodes, you must either create an AWS Systems Manager hybrid activation or add certificates and keys to your nodes for use with AWS IAM Roles Anywhere. To learn more, visit Prepare credentials for EKS Hybrid Nodes in the Amazon EKS User Guide.

3. Connect your hybrid nodes to the EKS cluster

You’re now ready to connect Amazon EKS Hybrid Nodes to your EKS cluster. You can use the Amazon EKS Hybrid Nodes CLI (nodeadm) to simplify the installation, configuration, and registration of your hosts as hybrid nodes. nodeadm automatically installs the required AWS Systems Manager or IAM Roles Anywhere components when you run the nodeadm install command.

You can run the nodeadm install process on each running host, or you can run nodeadm install as part of your operating system build pipelines to produce an image with the components needed to join your host to an EKS cluster.

$ nodeadm install 1.31 --credential-provider <ssm, iam-ra>
{"level":"info","ts":...,"caller":"...","msg":"Loading configuration","configSource":"file://nodeConfig.yaml"}
{"level":"info","ts":...,"caller":"...","msg":"Validating configuration"}
{"level":"info","ts":...,"caller":"...","msg":"Validating Kubernetes version","kubernetes version":"1.30"}
{"level":"info","ts":...,"caller":"...","msg":"Using Kubernetes version","kubernetes version":"1.30.0"}
{"level":"info","ts":...,"caller":"...","msg":"Installing SSM agent installer..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing kubelet..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing kubectl..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing cni-plugins..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing image credential provider..."}
{"level":"info","ts":...,"caller":"...","msg":"Installing IAM authenticator..."}
{"level":"info","ts":...,"caller":"...","msg":"Finishing up install..."}

Create a nodeConfig.yaml file on each host that contains the information required to connect to your EKS cluster. Here is an example nodeConfig.yaml that uses AWS Systems Manager hybrid activations.

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
metadata:
  name: hybrid-node
spec:
  cluster:
    name: my-cluster
    region: us-east-1
  hybrid:
    roleArn: arn:aws:iam:012345678910:role/eksHybridNodesRole
    ssm:
      activationCode: <activation-code>
      activationId: <activation-id>

Now, run nodeadm on each host.

$ nodeadm init -c file:/// nodeConfig.yaml

If the preceding command is completed successfully, your hybrid node has joined your EKS cluster. You can verify this in the Amazon EKS console or with the kubectl get nodes command. Before your hybrid nodes have status as Ready, you must install a compatible CNI. To learn more, visit Install CNI for EKS Hybrid Nodes in the Amazon EKS User Guide.

4. View and manage connected your hybrid nodes in EKS console

Now that the nodes are ready, you can view your hybrid nodes and the resources running on them in the EKS console.

You’re responsible for managing your hybrid nodes and updating the software they run. You can update to the latest version of the Amazon EKS Hybrid Nodes CLI to pull in the latest fixes and updates and upgrade Kubernetes versions. To learn more, visit Upgrade EKS Hybrid Nodes in the Amazon EKS User Guide.

Now available
Amazon EKS Hybrid Nodes is now available in all AWS Regions except the AWS GovCloud (US) Regions and the China Regions.

There are no upfront commitments or minimum fees, and you pay for the hourly usage of your EKS cluster and EKS Hybrid Nodes as you use them. EKS clusters with your hybrid nodes have the same per cluster per hour cost as EKS clusters with nodes running in AWS Cloud for both standard and extended support. Additionally, EKS clusters with your hybrid nodes incur an hourly fee per hybrid node vCPU. To learn more, visit the Amazon EKS pricing page.

Give EKS Hybrid Nodes a try in the Amazon EKS console. To learn more, visit the EKS Hybrid Nodes documentation and send feedback to AWS re:Post for EKS or through your usual AWS Support contacts.

— Channy

Streamline Kubernetes cluster management with new Amazon EKS Auto Mode

2024-12-01 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/streamline-kubernetes-cluster-management-with-new-amazon-eks-auto-mode/

Today, we’re announcing the general availability of Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode, a new capability to streamline Kubernetes cluster management for compute, storage, and networking, from provisioning to on-going maintenance with a single click. You can achieve higher agility, performance, and cost-efficiency by eliminating the operational overhead of managing the cluster infrastructure required to run production-grade Kubernetes applications at scale on Amazon Web Services (AWS).

Customers choose Amazon EKS because they can use the open standards and portability of Kubernetes with the security, scalability, and availability of AWS cloud. While Kubernetes gives advanced customers deep controls over application operations, other customers find managing the components required for production-grade Kubernetes applications to be complex and labor-intensive.

With the EKS Auto Mode, you can automate cluster management without deep Kubernetes expertise, because it selects optimal compute instances, dynamically scales resources, continuously optimizes costs, manages core add-ons, patches operating systems, and integrates with AWS security services. AWS expands its operational responsibility in EKS Auto Mode compared to customer-managed infrastructure in your EKS clusters. In addition to the EKS control plane, AWS will configure, manage, and secure the AWS infrastructure in EKS clusters that your applications need to run.

You can now get started quickly, improve performance, and reduce overhead, enabling you to focus on building applications that drive innovation instead of on cluster management tasks. EKS Auto Mode also reduces the work required to acquire and run cost-efficient GPU-accelerated instances so that your generative AI workloads have the capacity they need when they need it.

Get started with Amazon EKS Auto Mode
To get started, go to the Amazon EKS console and start to create your EKS cluster. You’ll have two options, Quick configuration (with EKS Auto Mode) and Custom configuration.

After you choose quick configuration, enter your cluster name and Kubernetes version, IAM roles, VPC subnets. You can view configuration default values in EKS Auto Mode whether you can edit after the cluster is created.

EKS Auto Mode enables the following Kubernetes capabilities in your EKS cluster:

Compute auto scaling and management
Application load balancing management
Pod and service networking and network policies
Cluster DNS and GPU support
Block storage volume support

When you choose Create, your EKS cluster with Auto Mode will be deployed in minutes with a single click.

If you choose the custom configuration option, you can customize other aspects of your cluster. You can use EKS Auto Mode in this option too.

You can also create an EKS Auto Mode cluster using AWS Command Line Interface (AWS CLI), eksctl, and AWS CloudFormation. Run the following eksctl command to create a new EKS Auto Mode cluster with:

$ eksctl create cluster --name=<cluster-name> --enable-auto-mode

To learn more, visit Create cluster with EKS Auto Mode in the Amazon EKS User Guide.

If you want to enable EKS Auto Mode for an existing EKS cluster, choose Manage in the EKS Auto Mode section of the Overview tab in the EKS cluster detail page.

Select the box next to Use EKS Auto Mode to enable the EKS Auto Mode. You can unselect the EKS Auto Mode that will be configured in the cluster. The default is to create both a system and a default node pool and a node class.

You can also migrate from Karpenter, EKS Managed Node Groups, and EKS Fargate to EKS Auto Mode. To learn more, visit Enable EKS Auto Mode on existing EKS clusters in the Amazon EKS User Guide.

To meet your workload requirements, you can configure specific aspects of your EKS Auto Mode clusters. While EKS Auto Mode manages most infrastructure components automatically, you can customize node networking settings, node compute resources, storage class settings, and application load balancing behaviors while maintaining the benefits of automated infrastructure management. To learn more, visit Change EKS Auto cluster settings in the Amazon EKS User Guide.

Now, you can deploy different types of workloads to Amazon EKS clusters running in EKS Auto Mode. We provide key workload patterns including sample applications, load-balanced web applications, stateful workloads using persistent storage, and workloads with specific node placement requirements. Each example includes complete manifests and step-by-step deployment instructions that you can use as templates for your own applications. To learn more, visit Run workloads in EKS Auto Mode clusters in the Amazon EKS User Guide.

Now available
Amazon EKS Auto Mode is now available in all commercial AWS Regions except China Regions where Amazon EKS is available. You can enable EKS Auto Mode in any EKS cluster running Kubernetes 1.29 and above with no upfront fees or commitments—you pay for the management of the compute resources provisioned, in addition to your regular EC2 costs. To learn more, visit Amazon EKS pricing page.

Please register for the online webinar: Simplifying Kubernetes operations with Amazon EKS Auto Mode on December 12, 2024 to learn more about how EKS Auto Mode can accelerate your time to deploy workloads to production and reduce the operational overheads of Kubernetes. To learn more, visit Automate cluster infrastructure with EKS Auto Mode in the Amazon EKS User Guide.

Give EKS Auto Mode a try in the Amazon EKS console and send feedback to AWS re:Post for EKS or through your usual AWS Support contacts.

— Channy

Introducing storage optimized Amazon EC2 I8g instances powered by AWS Graviton4 processors and 3rd gen AWS Nitro SSDs

2024-12-01 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-storage-optimized-amazon-ec2-i8g-instances-powered-by-aws-graviton4-processors-and-3rd-gen-aws-nitro-ssds/

Today, we’re announcing the general availability of Amazon EC2 I8g instances, a new storage optimized instance type to provide the highest real-time storage performance among storage-optimized EC2 instances with the third generation of AWS Nitro SSDs and AWS Graviton4 processors.

AWS Graviton4 is the most powerful and energy efficient processor we have ever designed for a broad range of workloads running on EC2 instances using a 64-bit ARM instruction set architecture. AWS Nitro System SSDs are custom built by AWS and offer high I/O performance, low latency, minimal latency variability, and security with always-on encryption.

EC2 I8g instances are the first instance type to use third-generation AWS Nitro SSDs. These instances offer up to 22.5 TB local NVME SSD storage with up to 65 percent better real-time storage performance per TB and 60 percent lower latency variability compared to the previous generation I4g instances. Based on the AWS Graviton4 processors, I8g instances deliver up to 60 percent better compute performance and two times larger caches compared to I4g.

I8g instances offer up to 96 vCPUs, 768 GiB of memory, and 22.5 TB of storage to deliver more compute and storage choices compared with I4g instances.

Instance name	vCPUs	Memory (Gib)	Storage (GB)	Network bandwidth (Gbps)	EBS bandwidth (Gbps)
I8g.large	2	16	468	up to 10	up to 10 Gbps
I8g.xlarge	4	32	937	up to 10	up to 10 Gbps
I8g.2xlarge	8	64	1,875	up to 12	up to 10 Gbps
I8g.4xlarge	16	128	3,750	up to 25	up to 10 Gbps
I8g.8xlarge	32	256	7,500 (2 x 3,750)	up to 25	10 Gbps
I8g.12xlarge	48	384	11,520 (3 x 3,750)	up to 28.125	15 Gbps
I8g.16xlarge	64	512	15,000 (4 x 3,750)	up to 37.5	20 Gbps
I8g.24xlarge	96	768	22,500 (6 x 3,750)	up to 56.25	20 Gbps
I8g.metal-24×1	96	768	22,500 (6 x 3,750)	up to 56.25	30 Gbps

You can use I8g instances for I/O intensive workloads that require low latency access to data such as transactional databases (MySQL and PostgreSQL), real-time databases, NoSQL databases, (Aerospike, Apache Druid, MongoDB) and real-time analytics such as Apache Spark.

Additionally, I8g instances are built on the AWS Nitro System, which offloads CPU virtualization, storage, and networking functions to dedicated hardware and software to enhance the performance and security of your workloads. The Graviton4 processors offer you enhanced security by fully encrypting all high-speed physical hardware interfaces.

Things to know
Here are some things that you should know about EC2 I8g instances:

Operating system – EC2 I8g instances support Amazon Linux 2023, Amazon Linux 2, CentOS Stream 8 or newer, Ubuntu 18.04 or newer, SUSE 15 SP2 or newer, Debian 11 or newer, Red Hat Enterprise 8.2 or newer, CentOS 8.2 or newer, FreeBSD 13 or newer, Rocky Linux 8.4 or newer, Alma Linux 8.4 or newer, and Alpine Linux 3.12.7 or newer.
Networking – You can use I8g instances in storage intensive workloads that typically have burst network usage patterns. All I8g instance sizes have burst network bandwidth and can burst more than 60 minutes, depending on the instance sizes, to support the majority of the workloads requiring instance storage data hydration, backup, and snapshot over the network.
Migration – If you’re using I4g instances now, you will have straightforward experience migrating storage intensive workloads to I8g instances because these instances offer similar memory and storage ratios to existing I4g instances.

Now available
Amazon EC2 I8g instances are now available in the US East (N. Virginia) and US West (Oregon) AWS Regions through On-Demand instances, Savings Plans, Spot Instances, Dedicated Instances, or Dedicated Hosts.

Give EC2 I8g instances a try in the Amazon EC2 console. To learn more, visit the EC2 I8g instances page and send feedback to AWS re:Post for EC2 or through your usual AWS Support contacts.

— Channy

Now available: Storage optimized Amazon EC2 I7ie instances

2024-12-01 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/now-available-storage-optimized-amazon-ec2-i7ie-instances/

The new storage optimized Amazon Elastic Compute Cloud (Amazon EC2) I7ie instances feature up to 120 TB of low latency NVMe storage and 5th generation Intel Xeon Scalable Processors with an all-core turbo frequency of 3.2 GHz. Fueled by 3rd Generation AWS Nitro SSDs, these instances deliver the highest storage density available in the cloud today. When compared to the previous generation of storage optimized instances, they provide:

Up to 65% better real-time storage performance per TB
Up to 50% lower I/O latency with up to 65% lower latency variability
Up to 40% better compute performance
Up to twice as many vCPUs and twice as much memory
20% better price-performance

The instances are designed to support I/O intensive workloads that need a high degree of random IOPS: NoSQL databases, distributed file systems, search engines, data warehouses, and analytics.

I7ie instances are available in nine sizes with up to 192 vCPUs and 1.5 TiB of memory:

Instance Name	vCPUs	Memory	NVMe Storage (Nitro SSD)	EBS Bandwidth	Network Bandwidth
I7ie.large	2	16 GiB	1.25 TB (1 x 1.25 TB)	Up to 10 Gbps	Up to 25 Gbps
I7ie.xlarge	4	32 GiB	2.5 TB (1 x 2.5 TB)	Up to 10 Gbps	Up to 25 Gbps
I7ie.2xlarge	8	64 GiB	5 TB (2 x 2.5 TB)	Up to 10 Gbps	Up to 25 Gbps
I7ie.3xlarge	12	96 GiB	7.5 TB (3 x 2.5 TB)	Up to 10 Gbps	Up to 25 Gbps
I7ie.6xlarge	24	192 GiB	15 TB (2 x 7.5 TB)	Up to 10 Gbps	Up to 25 Gbps
I7ie.12xlarge	48	384 GiB	30 TB (4 x 7.5 TB)	15 Gbps	Up to 25 Gbps
I7ie.18xlarge	72	576 GiB	45 TB (6 x 7.5 TB)	22.5 Gbps	Up to 75 Gbps
I7ie.24xlarge	96	768 GiB	60 TB (8 x 7.5 TB)	30 Gbps	Up to 100 Gbps
I7ie.48xlarge	192	1,536 GiB	120 TB (16 x 7.5 TB)	60 Gbps	100 Gbps

A larger L3 cache, increased memory bandwidth, and other improvements deliver increased processing power. The VP2INTERSECT instruction (part of AVX-512) accelerates Machine Learning and graph processing workloads; the Advanced Matrix Extensions (AMX) increase deep learning training and inferencing performance.

On the network side, the instances feature over 3x the EBS bandwidth of the previous generation of storage optimized instances. This accelerates just about every I/O-intensive use case, and is especially helpful when hydrating an in-memory database or caching server. All instances sizes support the Elastic Network Adapter (ENA) and can be launched in cluster placement groups; the 48xlarge instance size also supports the Elastic Fabric Adapter (EFA).

Things to Know
Here are a couple of things that you should know about these new instances:

Regions – We are launching in the US East (Ohio, N. Virginia), US West (Oregon), Asia Pacific (Tokyo), and Europe (Frankfurt, London) AWS Regions today, with plans to expand to others in the future.

Purchase Options – I7ie instances are available in On-Demand, Spot, Savings Plan, Dedicated Instance, and Dedicated Host form.

— Jeff;

New Amazon CloudWatch Database Insights: Comprehensive database observability from fleets to instances

2024-12-01 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-database-insights-comprehensive-database-observability-from-fleets-to-instances/

Observing your Amazon Aurora databases is now a whole lot easier. Instead of spending time setting up telemetry, building dashboards, and configuring alarms, you just open Amazon CloudWatch Database Insights and take a look. With no further setup, you can monitor the health of all of your Amazon Aurora MySQL and PostgreSQL instances in the selected Region:

Each of the sections contains a wealth of detail and I’ll get to that in a moment (this may be the ultimate “but wait, there’s more” post). From this view, I can open the filter control on the left and filter the set of instances in a couple of different ways. For example, I can filter for all of the instances running Amazon Aurora MySQL, and see that I have 66 such instances, with 3 raising alarms:

I can save the filter as a Fleet (note that Fleets are defined by specific properties and tags of the database instances and as such are inherently dynamic):

And then I can see the overall health of the fleet with a click. The entire page updates to reflect the fleet; I focus on the summary:

Behind the scenes, Database Insights looks for CloudWatch alarms that include a DBInstanceIdentifier dimension, and uses these alarms to establish a correlation between database instances and alarms. This, along with other built-in heuristics and correlation steps, allows Database Insights to deliver helpful, well-organized information that will help you to better understand the overall health of your fleet and to dive deep in order to find bottlenecks and other issues.

Clicking on an instance (represented by a hexagon) reveals details; I click on the instance name (demo-mysql-reader0) to learn more:

In the per-instance view I can also see a myriad of details:

Each of the tabs at the bottom provides additional insights into what’s happening inside the database instance. For example, selecting DB Load Analysis / Top SQL / SQL Metrics shows me which SQL statements are imposing the heaviest load, along with 29 additional metrics (not shown):

From past experience, I know that finding and understanding slow queries is a tedious yet important task. with Database Insights I can see patterns that are common to the slow queries, as well as the actual queries:

With help from AWS X-Ray, Application Signals, and the AWS Distro for OpenTelemetry SDK, I can see the services and operations that originate the queries to the database instance:

The red X indicates that this operation is failing the associated Service Level Objective (SLO), an application performance monitoring aspect of Application Signals. An SLO defines the reliability of a service against customer expectations, and can be set up by selecting the service and clicking Create SLO. There are a couple of steps and some very helpful options, but at the core a SLO is measured as a percentage of successful requests over an extended period of time:

If the database instance is configured to send logs to CloudWatch Logs, I can see and search the logs, filtered by the selected time period, and within a particular log group:

There’s still a lot more to explore at the fleet level. For example, I can see the ten calling services which drive the highest DB load (again, this is powered by AWS X-Ray, Application Signals, and the AWS Distro for OpenTelemetry SDK):

And I can see the top 10 instances with respect to any of eight different metrics:

I could go on all day, but I will leave the rest for you to explore. As I never tire of saying, this feature is available now and you can start using it today.

Things to Know
Here are a couple of things to know about Database Insights:

Supported Databases – You can use Database Insights with Amazon Aurora MySQL and Amazon Aurora PostgreSQL database instances.

Pricing – There is a per-hour, per-database instance charge based on the average number of vCPUs used (for provisioned instances) or Aurora Capacity Units (for Serverless v2 databases) monitored, with separate charges for ingestion and storage of database logs. See the CloudWatch Pricing page for more information.

Regions – This feature is available in all commercial AWS Regions.

— Jeff;

New Amazon CloudWatch and Amazon OpenSearch Service launch an integrated analytics experience

2024-12-01 Elizabeth Fuentes

Post Syndicated from Elizabeth Fuentes original https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-and-amazon-opensearch-service-launch-an-integrated-analytics-experience/

Today, Amazon Web Services (AWS) announces a new integrated analytics experience and zero-ETL integration between Amazon CloudWatch and Amazon OpenSearch Service. This integration simplifies log data analysis and visualization without data duplication, streamlining log management while reducing technical overhead and operational costs. CloudWatch Logs customers now have access to two additional query languages beyond CloudWatch Logs Insights QL, while OpenSearch customers can query CloudWatch logs in place without creating separate extract, transform, and load (ETL) pipelines.

Organizations often need different analytics capabilities for their log data. Some teams prefer CloudWatch Logs for its scalability and simplicity in centralizing logs from all their systems, applications, and AWS services. Others require OpenSearch Service for advanced analytics and visualizations. Previously, integration between these services required maintaining separate ingestion pipelines or creating ETL processes. This new integration helps customers get the best of both services by eliminating this complexity by bringing the power of OpenSearch analytics directly to CloudWatch Logs, without any data copy.

Amazon CloudWatch Logs now supports OpenSearch Piped Processing Language (PPL) and OpenSearch SQL directly within the CloudWatch Logs Insights console. You can use SQL to analyze data and correlate logs using JOIN. You can use SQL functions (such as JSON, mathematical, datetime, and string functions) for intuitive log analytics. You can also use the OpenSearch PPL to filter, aggregate, and analyze data. With a few clicks, you can access pre-built, out-of-the-box dashboards for vended logs, such as Amazon Virtual Private Cloud (VPC), AWS CloudTrail, and AWS WAF. These dashboards enable faster monitoring and troubleshooting through visualizations, such as analyzing flows over time, top talkers, megabytes, and packets transferred over time, without having to configure individual widgets or build specific queries. You can analyze VPC flows over time, identify top talkers, track network traffic metrics, monitor web request trends in AWS WAF, or analyze API activity patterns in AWS CloudTrail.

Additionally, OpenSearch Service users can now analyze CloudWatch logs using OpenSearch Discover and run SQL and PPL, similar to how they analyze data in Amazon Simple Storage (Amazon S3), and build indexes and create dashboards directly without any ETL operations or separate ingestion pipelines.

Let’s explore how this integration works
To demonstrate the new OpenSearch SQL and PPL query capabilities in CloudWatch, I start in the CloudWatch console. In the navigation pane, I choose Logs then Logs Insights. After selecting log groups for the query, I can now use OpenSearch PPL or OpenSearch SQL query languages directly within CloudWatch Logs Insights, with no additional setup or integration required. Using this new capability, I can write complex queries using familiar SQL syntax or OpenSearch PPL, making log analysis more intuitive and efficient. In the Query commands menu, you can find sample queries to help you get started.

This example demonstrates how to use SQL JOIN to combine data from two log groups: pet adoptions and pet availability. By filtering for specific customer IDs, you can analyze related log records and trace IDs for troubleshooting purposes.

One of the powerful features of this integration for CloudWatch Logs customers is the ability to create pre-built dashboards for Amazon VPC Flows, AWS CloudTrail and AWS WAF logs. Let’s explore this by creating a dashboard for AWS WAF logs. In the Analyze with OpenSearch tab, I choose Settings and follow the steps.

After a few minutes, my integration is ready and I go to Create an OpenSearch dashboard. In the options Select automatic dashboard type, I choose AWS WAF logs.

In the Dashboard data configuration tab, I can select Data synchronization frequency to occur every 15 minutes. I Select the log groups and View log samples of the selected log groups. I finish by choosing Create a dashboard.

After creating my dashboard, I can explore my logs. The AWS WAF logs dashboard provides comprehensive visibility into web application firewall metrics and events, with automatically configured visualizations that help you monitor and analyze security patterns.

Similarly, the CloudTrail dashboard offers deep insights into API activity across your AWS environment. It’s useful for monitoring API activity, auditing actions, and identifying potential security or compliance issues.

The VPC Flow Logs dashboard provides detailed visualization of key metrics from your logs for network traffic analysis. You can analyze network traffic, detect unusual patterns, and monitor resource usage. The dashboard currently supports only VPC v2 fields (default format). Custom formatted fields are not supported.

With zero-ETL to access CloudWatch data from OpenSearch Services, I also can build an OpenSearch dashboard from the OpenSearch Service console without having to build and maintain an ETL process. For this, I go to Central management, then I select the new Connected data sources menu, click choose Connect to create a new connected data source, and choose CloudWatch Logs.

In the next step, I name my data source and choose to Create a new role, which must have the necessary permissions to execute actions on OpenSearch Service. You can see them in the Sample custom policy.

In the Set up OpenSearch step, configure a OpenSearch data connection for CloudWatch Logs by selecting Create a new collection. As part of setting up the CloudWatch Logs source, a new OpenSearch Service serverless collection and OpenSearch UI application is created to store the indexed views and provide a user interface to analyze your CloudWatch Logs data. I create a new collection, name it, and conﬁgure the OpenSearch application and workspace within the application. After setting the Data retention days, I choose Next and ﬁnish with Review and connect.

When the integration with CloudWatch is ready, I can choose between Explore logs without indexing data which will take me to a querying interface in Discover or Explore vended logs by creating a dashboard for Amazon VPC Flows, CloudTrail and AWS WAF logs.

After I select Explore logs, OpenSearch UI takes me to Discover in the application workspace I created during the data source setup. In Discover, I select the data picker and choose View all available data to access my CloudWatch Logs data source and log groups.

After I select the log groups, I can analyze my CloudWatch logs using OpenSearch SQL and PPL directly in Discover, without having to switch between applications.

To create a dashboard, I return to the Connected data sources overview page on the console. From there, I select Create dashboard, which allows me to visually analyze my CloudWatch data without having to define queries or build visualizations, as I previously did in the CloudWatch console

After the dashboard is created, I navigate to OpenSearch resources where I can see the newly created indexes being populated with data in my Collection. After I have the data, I can go to the dashboard with the data from the CloudWatch logs that I selected in the configuration, and as more data comes in, it will be displayed in near real-time on the OpenSearch dashboard.

With this zero-ETL integration you can ingest data directly into OpenSearch, using its powerful query capabilities and visualization features while maintaining data consistency and reducing operational overhead.

Integration Highlights
For CloudWatch customers:

Query capabilities – Streamline log investigation by using OpenSearch SQL and PPL queries directly within the CloudWatch Logs Insights console.
Analytics features – With a few clicks, access pre-built, out-of-the-box dashboards for vended logs, such as VPC, AWS WAF, and CloudTrail logs. These dashboards enable faster monitoring and troubleshooting through visualizations for analyzing flows over time, top talkers, megabytes, and packets transferred over time, without having to configure individual widgets or build specific queries.
Getting started for CloudWatch users – Configure integration from CloudWatch Logs to OpenSearch Service. For more information refer to the Amazon CloudWatch Logs query capabilities and Amazon CloudWatch Logs vended dashboard documentation.

For OpenSearch Service customers:

Zero-ETL integration – Access and analyze CloudWatch data directly from OpenSearch Service without building or maintaining ETL processes. This integration eliminates separate ingestion pipelines while reducing storage costs and operational overhead through simplified data management and zero data duplication.
Getting started for OpenSearch users – Create a data connection selecting CloudWatch as a data source from OpenSearch Service. For more information, refer to the Amazon OpenSearch Service Developer Guide.

Regional availability and pricing
This integration is now available in AWS Regions where Amazon OpenSearch Service direct query is available. For pricing details and free trial information, you can visit the Amazon CloudWatch Pricing and Amazon OpenSearch Service Pricing pages.

PS: Writing a blog post at AWS is always a team effort, even when you see only one name under the post title. In this case, I want to thank Joshua Bright, Ashok Swaminathan, Abeetha Bala, Calvin Weng, and Ronil Prasad for their generous help with screenshots, technical guidance, and sharing their expertise in both services, which made this integration overview possible and comprehensive.

— Eli

Amazon FSx for Lustre increases throughput to GPU instances by up to 12x

2024-11-28 Danilo Poccia

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/amazon-fsx-for-lustre-unlocks-full-network-bandwidth-and-gpu-performance/

Today, we are announcing support for Elastic Fabric Adapter (EFA) and NVIDIA GPUDirect Storage (GDS) on Amazon FSx for Lustre. EFA is a network interface for Amazon EC2 instances that makes it possible to run applications requiring high levels of inter-node communications at scale. GDS is a technology that creates a direct data path between local or remote storage and GPU memory. With these enhancements, Amazon FSx for Lustre with EFA/GDS support provides up to 12 times higher (up to 1200 Gbps) per-client throughput compared to the previous FSx for Lustre version.

You can use FSx for Lustre to build and run the most performance demanding applications, such as deep learning training, drug discovery, financial modeling, and autonomous vehicle development. As datasets grow and new technologies emerge, you can adopt increasingly powerful GPU and HPC instances such as Amazon EC2 P5, Trn1, and Hpc7a. Until now, when accessing FSx for Lustre file systems, the use of traditional TCP networking limited throughput to 100 Gbps for individual client instances. This adoption is driving the need for FSx for Lustre file systems to provide the performance necessary to optimally utilize the increasing network bandwidth of these cutting-edge EC2 instances when accessing large datasets.

With EFA and GDS support in FSx for Lustre, you can now achieve up to 1,200 Gbps throughput per client instance (twelve times more throughput than previously) when using P5 GPU instances and NVIDIA CUDA in your applications.

With this new capability, you can fully utilize the network bandwidth of the most powerful compute instances and accelerate your machine learning (ML) and HPC workloads. EFA enhances performance by bypassing the operating system and using the AWS Scalable Reliable Datagram (SRD) protocol to optimize data transfer. GDS further improves performance by enabling direct data transfer between the file system and GPU memory, bypassing the CPU and eliminating redundant memory copies.

Let’s see how this works in practice.

Creating an Amazon FSx for Lustre file system with EFA enabled
To get started, in the Amazon FSx console, I choose Create file system and then Amazon FSx for Lustre.

I enter a name for the file system. In the Deployment and storage type section, I select Persistent, SSD and the new with EFA enabled option. I select 1000 MB/s/TiB in the Throughput per unit of storage section. With these settings, I enter 4.8 TiB for Storage capacity, which is the minimum supported with these settings.

For networking, I use the default virtual private cloud (VPC) and an EFA-enabled security group. I leave all other options to their default values.

I review all the options and proceed to create the file system. After a few minutes, the file system is ready to be used.

Mounting an Amazon FSx for Lustre file system with EFA enabled from an Amazon EC2 instance
In the Amazon EC2 console, I choose Launch instance, enter a name for the instance, and select the Ubuntu Amazon Machine Image (AMI). For Instance type, I select trn1.32xlarge.

In Network settings, I edit the default settings and select the same subnet used by the FSx Lustre file system. In Firewall (security groups), I select three existing security groups: the EFA-enabled security group used by the FSx for Lustre file system, the default security group, and a security group that provides Secure Shell (SSH) access.

In Advanced network configuration, I select ENA and EFA as Interface type. Without this setting, the instance would use traditional TCP networking and the connection with the FSx for Lustre file system would still be limited to 100 Gbps in throughput.

To have more throughput, I can add more EFA network interfaces, depending on the instance type.

I launch the instance and, when the instance is ready, I connect using EC2 Instance Connect and follow the instructions for installing the Lustre client in the FSx for Lustre User Guide and configuring EFA clients.

Then, I follow the instructions for mounting an FSx for Lustre file system from an EC2 instance.

I create a folder to use as mount point:

sudo mkdir -p /fsx

I select the file system in the FSx console and lookup the DNS name and Mount name. Using these values, I mount the file system:

sudo mount -t lustre -o relatime,flock file_system_dns_name@tcp:/mountname /fsx

EFA is automatically used when you access an EFA-enabled file system from client instances that support EFA and are using Lustre version 2.15 or higher.

Things to know
EFA and GDS support is available today with no additional cost on new Amazon FSx for Lustre file systems in all AWS Regions where persistent 2 is offered. FSx for Lustre automatically uses EFA when customers access an EFA-enabled file system from client instances that support EFA, without requiring any additional configuration. For a list of EC2 client instances that support EFA, see supported instance types in the Amazon EC2 User Guide. This network specifications table describes network bandwidths and EFA support for instance types in the accelerated computing category.

To use EFA-enabled instances with FSx for Lustre file systems, you must use Lustre 2.15 clients on Ubuntu 22.04 with kernel 6.8 or higher.

Note that your client instances and your file systems must be located in the same subnet within your Amazon Virtual Private Cloud (Amazon VPC) connection.

GDS is automatically supported on EFA-enabled file systems. To use GDS with your FSx for Lustre file systems, you need the NVIDIA Compute Unified Device Architecture (CUDA) package, the open source NVIDIA driver, and the NVIDIA GPUDirect Storage Driver installed on your client instance. These packages come preinstalled on the AWS Deep Learning AMI. You can then use your CUDA-enabled application to use GPUDirect storage for data transfer between your file system and GPUs.

When planning your deployment, note that EFA-enabled file systems have larger minimum storage capacity increments than file systems that are not EFA-enabled. For instance, if you choose the 1,000 MB/s/TiB throughput tier, the minimum storage capacity for EFA-enabled file systems starts at 4.8 TiB as compared to 1.2TB for FSx for Lustre file systems not enabling EFA. If you’re looking to migrate your existing workloads, you can use AWS DataSync to move your data from an existing file system to a new one that supports EFA and GDS.

For maximum flexibility, FSx for Lustre maintains compatibility with both EFA and non-EFA workloads. When accessing an EFA-enabled file system, traffic from non-EFA client instances automatically flows over traditional TCP/IP networking using Elastic Network Adapter (ENA), allowing seamless access for all workloads without any additional configuration.

To learn more about EFA and GDS support on FSx for Lustre, including detailed setup instructions and best practices, visit the Amazon FSx for Lustre documentation. Get started today and experience the fastest storage performance available for your GPU instances in the cloud.

— Danilo

Update 11/27: post updated to reflect 12x throughput

Time-based snapshot copy for Amazon EBS

2024-11-27 Jeff Barr

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/time-based-snapshot-copy-for-amazon-ebs/

You can now specify a desired completion duration (15 minutes to 48 hours) when you copy an Amazon Elastic Block Store (Amazon EBS) snapshot within or between AWS Regions and/or accounts. This will help you to meet time-based compliance and business requirements for critical workloads. For example:

Testing – Distribute fresh data on a timely basis as part of your Test Data Management (TDM) plan.

Development – Provide your developers with updated snapshot data on a regular and frequent basis.

Disaster Recovery – Ensure that critical snapshots are copied in order to meet a Recovery Point Objective (RPO).

Regardless of your use case, this new feature gives you consistent and predictable copies. This does not affect the performance or reliability of standard copies—you can choose the option and timing that works best for each situation.

Creating a Time-Based Snapshot Copy
I can create time-based snapshot copies from the AWS Management Console, CLI (copy-snapshot), or API (CopySnapshot). While working on this post I created two EBS volumes (100 GiB and 1 TiB), filled each one with files, and created snapshots:

To create a time-based snapshot, I select the source as usual and choose Copy snapshot from the Action menu. I enter a description for the copy, choose the us-east-1 AWS Region as the destination, select Enable time-based copy, and (because this is a time-critical snapshot), enter a 15 minute Completion duration:

When I click Copy snapshot, the request will be accepted (and the copy will become Pending) only if my account’s throughput quotas are not already exceeded due to the throughput consumed by other active copies that I am making to the destination region. If the account level throughput quota is already exceeded, the console will display an error.

I can click Launch copy duration calculator to get a better idea of the minimum achievable copy duration for the snapshot. I open the calculator, enter my account’s throughput limit, and choose an evaluation period:

The calculator then uses historical data collected over the course of previous snapshot copies to tell me the minimum achievable completion duration. In this example I copied 1,800,000 MiB in the last 24 hours; with time-based copy and my current account throughput quota of 2000 MiB/second I can copy this much data in 15 minutes.

While the copy is in progress, I can monitor progress using the console or by calling DescribeSnapshots and examining the progress field of the result. I can also use the following Amazon EventBridge events to take actions (if the copy operation crosses regions, the event is sent in the destination region):

copySnapshot – Sent after the copy operation completes.

copyMissedCompletionDuration – Sent if the copy is still pending when the deadline has passed.

Things to Know
And that’s just about all there is to it! Here’s what you need to know about time-based snapshot copies:

CloudWatch Metrics – The SnapshotCopyBytesTransferred metric is emitted in the destination region, and reflect the amount of data transferred between the source and destination region in bytes.

Duration – The duration can range from 15 minutes to 48 hours in 15 minute increments, and is specified on a per-copy basis.

Concurrency – If a snapshot is being copied and I initiate a second copy of the same snapshot to the same destination, the duration for the second one starts when the first one is completed.

Throughput – There is a default per-account limit of 2000 MiB/second between each source and destination pair. If you need additional throughput in order to meet your RPO you can request an increase via the AWS Support Center. Maximum per-snapshot throughput is 500 MiB/second and cannot be increased.

Pricing – Refer to the Amazon EBS Pricing page for complete pricing information.

Regions – Time-based snapshot copies are available in all AWS Regions.

— Jeff;

Announcing future-dated Amazon EC2 On-Demand Capacity Reservations

2024-11-25 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/announcing-future-dated-amazon-ec2-on-demand-capacity-reservations/

Customers use Amazon Elastic Compute Cloud (Amazon EC2) to run every type of workload imaginable, including web hosting, big data processing, high-performance computing (HPC), virtual desktops, live event streaming, and databases. Some of these workloads are so critical that customers asked for the ability to reserve capacity for them.

To help customers flexibly reserve capacity, we launched EC2 On-Demand Capacity Reservations (ODCRs) in 2018. Since then, customers have used capacity reservations (CRs) to run critical applications like hosting consumer websites, streaming lives sporting events and processing financial transactions.

Today, we’re announcing the ability to get capacity for future workloads using CRs. Many customers have future events such as product launches, large migrations, or end-of-year sales events like Cyber Monday or Diwali. These events are critical, and customers want to ensure they have the capacity when and where they need it.

While CRs helped customers reserve capacity for these events, they were only available just-in-time. So customers either needed to provision the capacity ahead of time and pay for it or plan with precision to provision CRs just-in-time at the start of the event.

Now you can plan and schedule your CRs up to 120 days in advance. To get started you specify the capacity you need, the start date, delivery preference, and the minimum duration you commit to use the capacity reservation. There are no upfront charges to schedule a capacity reservation. After Amazon EC2 evaluates and approves the request, it will activate the reservation on the start date, and customers can use it to immediately launch instances.

Getting started with future-dated capacity reservations
To reserve your future-dated capacity, choose Capacity Reservations on the Amazon EC2 console and select Create On-Demand Capacity Reservation, and choose Get started.

To create a capacity reservation, specify the instance type, platform, Availability Zone, platform, tenancy, and number of instances you are requesting.

In the Capacity Reservation details section, choose At a future date in the Capacity Reservation starts option and choose your start date and commitment duration.

You can also choose to end the capacity reservation at a specific time or manually. If you select Manually, the reservation has no end date. It will remain active in your account and continue to be billed until you manually cancel it. To reserve this capacity, choose Create.

After you create your capacity request, it appears in the dashboard with an Assessing status. During this state, AWS systems will work to determine if your request is supportable which is usually done within 5 days. Once the systems determine the request is supportable, the status will be changed to Scheduled. In rare cases, your request may be unsupported.

On your scheduled date, the capacity reservation will change to an Active state, the total instance count will be increased to the amount requested, and you can immediately launch instances.

After activation, you must hold the reservation for at least the commitment duration. After the commitment duration elapses, you can continue to hold and use the reservation if you’d like or cancel it if no longer needed.

Things to know
Here are some things that you should know about the future-dated CRs:

Evaluation – Amazon EC2 considers multiple factors when evaluating your request. Along with forecasted supply, Amazon EC2 considers how long you plan to hold the capacity, how early you create the Capacity Reservation relative to your start date, and the size of your request. To improve the ability of Amazon EC2 to support your request, create your reservation at least 56 days (8 weeks) before the start date. You need to submit a request for at least 100 vCPUs for only C, M, R, T, I instance types. The recommended minimum commitment for most requests will be 14 days.
Notification – We recommend monitoring the status of your request through the console or Amazon EventBridge You can use these notifications to trigger automation or send an email or text update. To learn more, visit Send an email when events happen using Amazon EventBridge in the Amazon EventBridge User Guide.
Pricing – Future dated capacity reservations are billed just like regular CRs. It is charged at the equivalent On-Demand rate whether you run instances in reserved capacity or not. For example, if you create a future dated CR for 20 instances and run 15 instances, you will be charged for 15 active instances and for 5 unused instances in the reservation including the minimum duration. Savings Plans apply to both unused reservations and instances running on the reservation. To learn more, visit Capacity Reservation pricing and billing in the Amazon EC2 User Guide.

Now available
Future dated EC2 Capacity Reservations are now available today in all AWS Regions where Amazon EC2 Capacity Reservations are available.

Give Amazon EC2 Capacity Reservations a try in the Amazon EC2 console. To learn more, visit On-Demand Capacity Reservations in the Amazon EC2 User Guide and send feedback to AWS re:Post for Amazon EC2 or through your usual AWS Support contacts.

— Channy

AWS Weekly Roundup: 197 new launches, AI training partnership with Anthropic, and join AWS re:Invent virtually (Nov 25, 2024)

2024-11-25 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-197-new-launches-ai-training-partnership-with-anthropic-and-join-aws-reinvent-virtually-nov-25-2024/

Last week, I saw an astonishing 197 new service launches from AWS. This means we are getting closer to AWS re:Invent 2024! Our News Blog team is also finalizing blog posts for re:Invent to introduce some awesome launches from service teams for your reading pleasure.

The most interesting news is that we’re expanding our strategic collaboration with Anthropic as our primary training partner for development of our AWS Trainium chips. This is in addition to being their primary cloud provider for deploying Anthropic’s Claude models in Amazon Bedrock. We’ll keep pushing the boundaries of what customers can achieve with generarive AI technologies with these kinds of collaborations.

Last week’s launches
Here are some AWS bundled feature launches:

Amazon Aurora – Amazon Aurora Serverless v2 now supports scaling to 0 Aurora Capacity Units (ACUs). With 0 ACUs, you can now save cost during periods of database inactivity. Instead of scaling down to 0.5 ACUs, the database can now scale down to 0 ACUs. Amazon Aurora is now compatible with MySQL 8.0.39 and PostgreSQL 17.0 in the Amazon RDS Database preview environment.

Amazon Bedrock – You can quickly build and execute complex generative AI workflows without writing code with the general availability of Amazon Bedrock Flows (previously known as Prompt Flows). Amazon Bedrock Knowledge Bases now supports binary vector embeddings for building Retrieval Augmented Generation (RAG) applications. Amazon Bedrock also introduce a preview launch of Prompt Optimization to rewrite prompts for higher quality responses from foundational models (FMs). You can use AWS Amplify AI kit to easily leverage your data to get customized responses from Bedrock AI models to build web apps with AI capabilities such as chat, conversational search, and summarization.

Amazon CloudFront – You can use gRPC applications in Amazon CloudFront that allows bidirectional communication between a client and a server over HTTP/2 connections. Amazon CloudFront introduces Virtual Private Cloud (VPC) origins to deliver content from applications hosted in VPC private subnets, and Anycast Static IPs to provide you with a dedicated list of IP addresses for connecting to all CloudFront edge locations worldwide. You can also conditionally change or update origin servers on each request with origin modification within CloudFront Functions, and use new log configuration and delivery options.

Amazon CloudWatch – You can use field indexes and log transformation to improve log analytics at scale in the CloudWatch Logs. You can also use enhanced search and analytics experience and runtime metrics support with CloudWatch Application Signals, and percentile aggregation and simplified events-based troubleshooting directly from the web vitals anomaly in CloudWatch Real User Monitoring (RUM).

Amazon Cognito – You can secure user access to your applications with passwordless authentication, including sign-in with passkeys, email, and text message. Amazon Cognito introduces Managed Login, hosted sign-in and sign-up experience that customers can personalize to align with their company or application branding. Cognito launches new user pool feature tiers: Essentials and Plus as well as a new developer-focused console experience. To learn more, visit Donnie’s blog post.

Amazon Connect – You can use new customer profiles and outbound campaigns to help you proactively address customer needs before they become potential issues. Amazon Connect Contact Lens now supports creating custom dashboards, as well as adding or removing widgets from existing dashboards. With new Amazon Connect Email, you can receive and respond to emails sent by customers to business addresses or submitted via web forms on your website or mobile app.

Amazon EC2 – You can shift the launches of EC2 instances in an Auto Scaling Group (ASG) away from an impaired Availability Zone (AZ) to quickly recover your unhealthy application in another AZ with Amazon Application Recovery Controller (ARC) zonal shift and zonal autoshift. Application Load Balancer (ALB) now supports HTTP request and response header modification giving you greater controls to manage your application’s traffic and security posture without having to alter your application code.

AWS End User Messaging (aka Amazon Pinpoint) – You can now track feedback for messages sent through the SMS and MMS channel, explicitly block or allow messages to individual phone numbers overriding your country rule settings, and cost allocation tags for SMS resources to track spend for each tag associated with a resource. AWS End User Messaging also now support integration with Amazon EventBridge.

AWS Lambda – You can use Lambda SnapStart for Python and .NET functions to deliver as low as sub-second startup performance. AWS Lambda now supports Amazon S3 as a failed-event destination for asynchronous invocations and Amazon CloudWatch Application Signals to easily monitor the health and performance of serverless applications built using Lambda. You can also use a new Node.js 22 runtime and Provisioned Mode for event source mappings (ESMs) that subscribe to Apache Kafka event sources.

Amazon OpenSearch Service – You can scale a single cluster to 1000 data nodes (1000 hot nodes and/or 750 warm nodes) to manage 25 petabytes of data. Amazon OpenSearch Service introduces Custom Plugins, a new plugin management option to extend the search and analysis functions in OpenSearch.

OpenSearch Serverless – You can use OpenSearch SQL and OpenSearch Piped Processing Language (PPL) query to leverage your existing SQL skills and tools, Binary Vector and FP16 compression to help reduce costs by lowering the memory requirements, and Point in Time (PIT) search to run multiple queries against a dataset fixed at a specific moment in OpenSearch Serverless.
OpenSearch Ingestion – You can now use AWS Lambda to define custom Lambda functions in your OpenSearch Ingestion pipelines and it now supports writing security data to Amazon Security Lake to ingest and transform security data from popular 3rd party sources.

Amazon Q Business – You can use tabular search to extract answers from tables embedded in documents ingested in Q Business. You can drag and drop files to upload and reuse any recently uploaded files in new conversations without uploading the files again. Amazon Q Business now supports integrations to Smartsheet in general, and Asana, Google Calendar in preview to automatically sync your index with your selected data sources. You can also use Q Business browser extensions for Google Chrome, Mozilla Firefox, and Microsoft Edge.

Amazon Q Developer – You can ask questions directly related to the AWS Management Console page you’re viewing, eliminating the need to specify the service or resource in your query. You can also use customizable chat responses generated by Q Developer in the IDE to securely connect Q Developer to your private codebases to receive more precise chat responses. Finally, you can use voice input and output capabilities in the AWS Console Mobile App along conversational prompts to list resources in your AWS account.

Amazon QuickSight – You can use Layer Map to visualize custom geographic boundaries, such as sales territories, or user-defined regions, and Image Component to upload your images directly for a variety of use cases, such as adding company logos. Amazon QuickSight also provides the ability to import visuals from an existing dashboard or analysis into your current analysis and Highcharts visuals to create custom visualizations using the Highcharts Core library in preview.

Amazon Redshift – You can ingest data from a wider range of streaming sources from Confluent Managed Cloud and self-managed Apache Kafka clusters on Amazon EC2 instances. You can also use enhanced security defaults which helps you adhere to best practices in data security and reduce the risk of potential misconfigurations.

AWS System Manager – You can use a new and improved version of AWS Systems Manager that brings a highly requested cross-account, and cross-Region experience for managing nodes at scale. AWS Systems Manager now supports instances running Windows Server 2025, Ubuntu Server 24.04, and Ubuntu Server 24.10.

Amazon S3 – You can configure S3 Lifecycle rules for S3 Express One Zone to expire objects on your behalf and append data to objects in S3 Express One Zone. You can also use Amazon S3 Express One Zone as a high performance read cache with Mountpoint for Amazon S3. Amazon S3 Connector for PyTorch now supports Distributed Checkpoint (DCP), improving the time to write checkpoints to Amazon S3.

Amazon VPC – You can use Block Public Access (BPA) for VPC, a new centralized declarative control that enables network and security administrators to authoritatively block Internet traffic for their VPCs. Amazon VPC Lattice now provides native integration with Amazon ECS, easily to deploy, manage, and scale containerized applications.

There’s a lot more launch news that I haven’t covered here. See AWS What’s New for more details.

See you virtually in AWS re:Invent
Next week we’ll hear the latest news from AWS, learn from experts, and connect with the global cloud community in Las Vegas. If you come, check out the agenda, session catalog, and attendee guides before your departure.

If you’re not able to attend re:Invent in person, we’re offering the option to livestream our Keynotes and Innovation Talks. With the registration for online pass, you will have access to on-demand keynote, Innovation Talks, and selected breakout sessions after the event. You can also register with AWS Builder ID, a personal account that enables one-click event registration and provides access to many AWS tools and services.

Please stay tuned in the next week!

– Channy

Introducing a new experience for AWS Systems Manager

2024-11-23 Matheus Guimaraes

Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/introducing-a-new-experience-for-aws-system-manager/

Today, I’m excited to introduce a new and improved version of AWS Systems Manager that brings a highly requested cross-account, and cross-Region experience for managing nodes at scale.

The new System Manager experience provides centralized visibility of all your managed nodes which include various infrastructure types, such as Amazon Elastic Compute Cloud (EC2) instances, containers, virtual machines on other cloud providers, on-premise servers, and edge Internet of Things (IoT) devices. They are referred to as “managed nodes” when they have the Systems Manager Agent (SSM Agent) installed and are connected to Systems Manager.

If an SSM Agent stops working on a node for whatever reason, then Systems Manager loses connection to it and that node is then referred to as an “unmanaged node.” With the new update, Systems Manager can also help you to easily discover and troubleshoot unmanaged nodes. You can run and even schedule an automated diagnosis that provides you with recommended runbooks that you can execute to fix any issues and reestablish connection so they become managed nodes again.

Systems Manager is also now integrated with Amazon Q Developer, the most capable generative AI–powered assistant for software development. You can ask questions about your managed nodes to Amazon Q Developer using natural language and it will provide you with rapid insights plus links straight to Systems Manager where you can perform actions or continue to explore further.

With this release, you can also use AWS Organizations, to allow a delegated administrator to centrally manage nodes across the organization thanks to the new integration with Systems Manager.

Let’s examine a quick example that helps to demonstrate some of these new capabilities.

Imagine a scenario where you are a cloud platform engineer leading a migration plan aiming to replace all nodes running Windows Server 2016 Datacenter in the organization. Let’s use the new Systems Manager experience to quickly gather information about all the nodes that needs to be included in our plan.

Step 1 – Asking Amazon Q Developer
The easiest starting point is using Amazon Q Developer to ask what you want to find using natural language. Using the AWS Console, I open the Amazon Q chatbot and type Find all of my managed nodes running Microsoft Windows Server 2016 Datacenter in my organization.

Amazon Q quickly comes back with an answer: it tells us that there are ten nodes that fit the criteria and provides a list with an overview of each one.

There is also a link that redirects to the new Explore nodes page in System Manager where we can learn more information. Let’s follow it.

Step 2 – Reviewing our infrastructure
The Explore nodes page provides a comprehensive overview of all managed nodes across your organization, with options to group and filter results for quick access. In this case, we can see that the results are already filtered by Operating system name providing us with a list of all the nodes that are running Microsoft Windows Server 2016 Datacenter.

This is a great start! We could just finish here by downloading the report and add those nodes to our migration plan, however, this page only shows you information about your managed nodes. Could it be that there are unmanaged nodes that need to included in our plan? Let’s find out.

Step 3 – Handling unmanaged nodes
Open the menu, and navigate to the Review node insights page. Here you can see a dashboard with widgets that provide insightful interactive charts that you can use to drill down and discover more information about your nodes or even take actions. For example, the Managed node types pie chart shows the types of managed nodes we have whereas the SSM Agent versions graph provides us with an overview of all the different versions of SSM Agent running on them. You can also customize this view by adding and replacing widgets.

We want to investigate any unmanaged nodes to make sure we don’t miss any that may need to be added to our migration plan. The Node summary widget clearly shows that there are two unmanaged nodes. This could mean that these nodes don’t have the SSM Agent installed in which case we will need to investigate them manually. However, it could also just mean there are issues with the SSM agent permissions or network connectivity preventing Systems Manager from managing these nodes and treating them like any other managed node. The new Systems Manager experience allows you easily troubleshoot and remediate SSM Agents issues so let’s attempt to do this now.

Start by selecting the piece of the chart displaying our unmanaged nodes. This pops up an option to initiate a comprehensive diagnosis of all our unmanaged nodes with only one click. Let’s run this.

The diagnosis reviews key configurations such as missing virtual private cloud (VPC) endpoints, misconfigured VPC DNS settings, and misconfigured instance security groups that may be preventing the SSM Agent from connecting to Systems Manager. After the scanning is complete, we can see that it displays two Misconfigured VPC endpoint findings. It also gives you a link that you can use to open a side panel containing a recommended runbook that you can execute to solve the issues as well as links to relevant documentation.

Choosing to execute the recommended runbook presents you with a detailed preview of the changes which include a thorough overview of the actions it’s going to take in addition to the input parameters used, a link to view a breakdown of the steps involved, and the target nodes for this execution.

Let’s choose to go ahead and select Execute. Keep in mind that this may incur costs, so make sure to review them before executing. You can keep an eye on progress on this page as it goes through the steps to attempt to fix the issues on each node.

Aha! After the remediation is complete, we can see that Systems Manager has found and corrected issues with the SSM Agent with two nodes. This means that Systems Manager is able to connect with the SSM Agent running in those nodes successfully making them “managed nodes.” We can verify this by returning to the Explore nodes page and noticing that the count of “unmanaged nodes” has been reduced to zero now.

Now that all of our nodes are managed, we’re ready to get a full list of all of those that need to be added to our migration plan.

Step 4 – Downloading a report
Back on the Explore nodes page we can see that the count for nodes running Microsoft Windows Server 2016 Datacenter has gone up from ten to twelve! That means that those previously unmanaged nodes that we fixed through the automated diagnosis are indeed running our target operating system.

This is exactly what we need so we choose to download a Report. You give it a file name, and then choose from a few options such as which columns to include. In this case, we choose to download a CSV file with a row containing the column names.

That’s it! We have our CSV with detailed information about the nodes that need upgrading across our entire infrastructure. And the best part? You can also use Systems Manager to automate the upgrade once you’re ready to go ahead with the migration.

Conclusion
Systems Manager is a critical tool for gaining visibility and control over your compute infrastructure and performing operational actions at scale. The new experience offers a centralized cross-account, cross-Region view of all your nodes in your AWS accounts, on-premises, and multicloud environments through a centralized dashboard, offering integration with Amazon Q Developer for natural language queries, and one-click SSM Agent troubleshooting. You can enable the new experience at no extra cost by navigating to the Systems Manager console and following the straightforward instructions.

To learn more, see the documentation for more detail about the new Systems Manager experience.

Check out this interactive demo for a full visual tour of this experience.

AWS named as a leader again in the Gartner Magic Quadrant for Distributed Hybrid Infrastructure

2024-11-22 Channy Yun (윤석찬)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/aws-named-as-a-leader-again-in-the-gartner-magic-quadrant-for-distributed-hybrid-infrastructure/

Gartner published the second Magic Quadrant for Distributed Hybrid Infrastructure (DHI), which includes Amazon Web Services (AWS) as a leader again. AWS has three products in this DHI portfolio: AWS Outposts, AWS Snowball, and AWS Local Zones. In the accompanying Gartner’s Critical Capabilities for DHI, AWS is ranked number one in four out of six use cases evaluated by Gartner—including hybrid infrastructure management, edge computing, assured workloads, and artificial intelligence & machine learning (AI/ML)—and among the top two in the use case of container management.

Gartner evaluates 10 DHI providers based on their Ability to Execute, which measures a vendor’s capacity to deliver its products or services effectively, and Completeness of Vision, which assesses a vendor’s understanding of the market and its strategy for future growth.

Here is the graphical representation of the 2024 Gartner Magic Quadrant for DHI.

Gartner recognized AWS strengths as:

Leading public cloud provider – AWS DHI solutions appeal to AWS public cloud customers that want to extend their infrastructure to their data center and edge locations, while also migrating from their remaining private cloud infrastructure.
As-a-service delivery – The fully managed infrastructure delivery of AWS Outposts simplifies operations and enables a hands-off, single-vendor approach to infrastructure management, including integration with some on-premises technologies.
AWS support – Gartner clients report high satisfaction with the AWS worldwide support and services team.

We believe this leader placement reflects our innovation at the edge of the cloud for workloads that require low latency, local data processing, data residency, or migration with on-premises interdependencies. At AWS, we extend the same AWS infrastructure, AWS services, APIs, and tools wherever you need them for a truly consistent cloud experience.

Whether your workloads are running in the AWS Regions, in metro areas with AWS Local Zones, on premises with AWS Outposts, in the telco networks with AWS Wavelength, or at the far edge with AWS Snow Family, you can standardize on the same cloud operating model for all your applications. You can streamline developer workflow by standardizing on a common set of continuous integration and continuous deployment (CI/CD) pipelines. It also reduces the time, resources, operational risk, and maintenance downtime required to manage IT infrastructure.

As examples of accelerated innovation, we have added the latest generation of GPU-backed instances to Local Zones to better support ML workloads and expanded the number of locations. We have made Outposts available in more countries and added AWS services supported on Outposts to facilitate migration and disaster recovery, such as AWS Elastic Disaster Recovery and Amazon Route 53 Resolver to improve application availability and performance.

In addition, we have improved the disconnection tolerance for container-based workloads on Outposts by making it possible for customers to run both the Kubernetes control plane and nodes locally, and we enhanced its capabilities for multi-rack Outposts deployments.

Access the complete 2024 Gartner Magic Quadrant for DHI report to learn more.

— Channy

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

GARTNER is a registered trademark and service mark of Gartner and Magic Quadrant is a registered trademark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and are used herein with permission. All rights reserved.

Improve your app authentication workflow with new Amazon Cognito features

2024-11-22 Donnie Prakoso

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/

Introduced 10 years ago, Amazon Cognito is a service that helps you implement customer identity and access management (CIAM) in your web and mobile applications. You can use Amazon Cognito for various use cases, from providing your customers to quickly add sign-in and sign-up experiences to your applications and authorization to securing machine-to-machine authentication and enabling role-based access to AWS resources.

Today, I’m excited to share a series of significant updates to Amazon Cognito. These enhancements aim to provide you with more flexibility, improved security, and a better user experience for your applications.

Here’s a quick summary:

A new developer-focused console experience
Amazon Cognito now offers a streamlined getting-started experience featuring a quick wizard and use case-specific recommendations. This new approach helps you set up configurations and reach your end users faster and more efficiently than ever before.

This is the new Amazon Cognito flow to help you quickly set up your application. You can get started in three steps:

Choose the type of application you need to build
Configure the sign-in options according to the type of your application
Follow the instructions to integrate the sign-in and sign-up pages with your application

Then, select Create.

Amazon Cognito then automatically creates your application and a new user pool, which is a user directory for authentication and authorization. From here, you can review your sign-in page by selecting View login page or get started with the example code for your application. Furthermore, Amazon Cognito supports major application frameworks and offers detailed instructions for integrating them using standard OpenID Connect (OIDC) and OAuth open source libraries.

This is the new overview dashboard for your application. The user pool dashboard now provides important information in the Details section, as well as a set of Recommendations to help you continue your development journey.

On this page, you can customize your users’ sign-in and sign-up experience with the Managed Login feature. This is a good segue for me to provide you with a quick overview of the next new feature.

Introducing Managed Login
The introduction of Managed Login brings a new level of customization to Amazon Cognito. Managed Login handles the heavy lifting of availability, scaling, and security for your company. Once integrated, you automatically get all the new security patches and future features without further code changes.

This feature allows you to create personalized sign-up and sign-in experiences that are a seamless part of your company’s application for your end users.

Before you can use Managed Login, you need to assign a domain. There are two ways to do this: use a prefix domain, a randomly generated sub-domain of Amazon Cognito domain, or use your own custom domain to provide your users with a familiar domain name.

Then, you can choose your Branding version, selecting either Managed login or classic Hosted UI.

If you’re an existing Amazon Cognito user, you might be familiar with the classic Hosted UI feature. Managed Login is the improved version of Hosted UI, offering a new collection of web interfaces for sign-up and sign-in, built-in responsiveness for different screen sizes, multi-factor authentication, and password-reset activities in your user pool.

With Managed Login, you can use the new branding designer, a no-code visual editor for managed login assets and style, and a set of API operations for programmatic configuration or deployment via infrastructure-as-code with AWS CloudFormation.

With the branding designer, you have the flexibility to customize the look and feel of the entire user journey, from sign up and sign in to password recovery and multi-factor authentication. This feature provides a real time preview and convenient shortcuts to preview screens in different screen sizes and display modes before you launch it.

You can learn more about Managed Login by visiting the Managed Login documentation page.

Passwordless login support
The Managed Login feature also offers pre-built integrations for passwordless authentication methods, including signing in with passkeys, email OTP (one-time-password) and SMS OTP. Passkey support allows users to authenticate using cryptographic keys stored securely on their devices, offering better security compared to traditional passwords. This capability helps you implement low-friction and secure authentication methods without the need to understand and implement WebAuthn related protocols.

By reducing the friction associated with traditional password-based sign-ins, this feature simplifies application access for your users while maintaining high security standards.

Visit the user pools authentication flow documentation page to learn more about the passwordless login support.

More options on pricing tiers: Lite, Essentials and Plus
Amazon Cognito has introduced new user pool feature tiers: Lite, Essentials, and Plus. These tiers are designed to cater to different customer needs and use cases with the Essentials tier being the default tier for new users pools created by customers. This new tier structure also allows you to choose the most appropriate option based on your application requirements, with the flexibility to switch between tiers as needed.

To check your current tier, you can go to your application dashboard and select Feature plan. You can also select Settings from the navigation menu.

On this page, you’ll get detailed information for each tier and the option to downgrade or upgrade your plan.

Here’s a quick overview of each tier:

Lite tier: Existing features such as user registration, password-based authentication, and social identity provider integration are now packaged in this tier. If you’re an existing Amazon Cognito user, you can continue using these features without making changes to your user pools.
Essentials tier: Offers comprehensive authentication and access control features, allowing you to implement secure, scalable, and customized sign-up and sign-in experiences for your application within minutes. It includes all capabilities in Lite along with supporting Managed Login and passwordless login options using passkeys, email, or SMS. Essentials also supports customizing access tokens and disallowing password reuse.
Plus tier: Builds upon the Essentials tier, focusing on elevated security needs. It includes all Essentials features plus threat protection capabilities against suspicious login activity, detection of compromised credentials, risk-based adaptive authentication, and the ability to export user authentication event logs for threat analysis.

Pricing for the Lite, Essentials and Plus tiers is based on monthly active users. Customers currently using the advanced security features of Amazon Cognito should consider the Plus tier, which includes all the advanced security features, additional capabilities such as passwordless, and up to 60 percent savings as compared to using the standalone advanced security features.

If you want to learn about these new pricing tiers, see the Amazon Cognito pricing page.

Things you need to know

Availability – The Essentials and Plus tier are available in all AWS Regions where Amazon Cognito is available except AWS GovCloud (US) Regions.
Free tier on Lite and Essentials tiers – Customers on the Lite and Essentials tiers can enjoy the free tier each month that does not automatically expire. It is available to both existing and new AWS customers indefinitely. For more details on free tier, please visit the Amazon Cognito pricing page.
Extended pricing benefit for existing customers – Customers are eligible to upgrade their user pools without advanced security features (ASF) in their existing accounts to Essentials and pay the same price as Cognito user pools until November 30, 2025. To be eligible, customers’ accounts must have had at least 1 monthly active user (MAU) in the last 12 months on or before 10:00am Pacific Time, November 22, 2024. These customers are also eligible to create new user pools with Essentials tier at the same price as Cognito users pools in those accounts until November 30, 2025.

With these updates, you can implement secure, scalable, and customizable authentication solutions for your applications with Amazon Cognito.

Happy building,
— Donnie

Introducing new capabilities to AWS CloudTrail Lake to enhance your cloud visibility and investigations

2024-11-21 Esra Kayabali

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/introducing-new-capabilities-to-aws-cloudtrail-lake-to-enhance-your-cloud-visibility-and-investigations/

Today, I’m excited to announce new updates to AWS CloudTrail Lake, which is a managed data lake you can use to aggregate, immutably store, and query events recorded by AWS CloudTrail for auditing, security investigation, and operational troubleshooting.

The new updates in CloudTrail Lake are:

Enhanced filtering options for CloudTrail events
Cross-account sharing of event data stores
General availability of the generative AI–powered natural language query generation
AI-powered query results summarization capability in preview
Comprehensive dashboard capabilities, including a high-level overview dashboard with AI-powered insights (AI-powered insights is in preview), a suite of 14 pre-built dashboards for various use cases, and the ability to create custom dashboards with scheduled refreshes

Let’s look into the new features one by one.

Enhanced filtering options for CloudTrail events ingested into event data stores
Enhanced event filtering capabilities give you greater control over which CloudTrail events are ingested into your event data stores. These enhanced filtering options provide tighter control over your AWS activity data, improving the efficiency and precision of security, compliance, and operational investigations. Additionally, the new filtering options help you reduce your analysis workflow costs by ingesting only the most relevant event data into your CloudTrail Lake event data stores.

You can filter both management and data events based on attributes such as eventSource, eventType, eventName, userIdentity.arn, and sessionCredentialFromConsole.

I go to the AWS CloudTrail console and choose Event data stores under Lake in the navigation pane. I choose Create event data store. In the first step, I enter a name in the Event data store name field. For this demo, I leave other fields as default. You can choose the pricing and retention options that suit your needs. In the next step, I choose Managements events and Data events under CloudTrail events. You can include all the options you need under CloudTrail events. You also have the option to choose ingestion options. I choose Ingest events to start ingesting when it’s created. There may be scenarios, when you want to deselect the Ingest events option to stop an event data store from ingesting events. For example, you may be copying trail events to the event data store and do not want the event data store to collect any future events. You can also choose to enable ingestion for all accounts in your organization or include only the current region in your event data store.

The following example shows an out of the box template for filtering, which excludes any management events that are initiated by an AWS Service. I choose Advanced event collection under the Management events. I choose Exclude AWS service-initiated events from the Log selector template dropdown. You can also expand the JSON view to see how the filters actually apply.

Under the Data events, the following example creates a filter to include DynamoDB data events initiated by a certain user, helping me to log events based on an IAM principal. I choose DynamoDB as Resource type. I choose Custom as Log selector template. Under the Advanced event selector, I choose userIdentity.arn as Field and equals as Operator. I enter the user’s ARN as Value. I choose Next and choose Create event data store in the final step.

Now, I have my event data store that gives me granular control over the ingested CloudTrail data.

This expanded set of filtering options helps you to be more selective in capturing only the most relevant events for your security, compliance, and operational needs.

Cross-account sharing of event data stores
You can use the cross-account sharing feature of event data stores to enhance collaborative analysis within organizations. It enables secure sharing of event data stores with selected AWS principals through Resource-Based Policies (RBP). This functionality allows authorized entities to query shared event data stores within the same AWS Region where they were created.

To use this feature, I go to the AWS CloudTrail console and choose Event data stores under Lake in the navigation pane. I choose an event data store from the list and navigate to its details page. I choose Edit in the Resource policy section. The following example policy includes a statement that allows root users in accounts 111111111111, 222222222222, and 333333333333 to run queries and get query results on the event data store owned by account ID 999999999999. I choose Save changes to save the policy.

Generative AI–powered natural language query generation in CloudTrail Lake is now generally available
In June, we announced this feature for CloudTrail Lake in preview. With this launch, you can generate SQL queries using natural language questions to easily explore and analyze AWS activity logs (only management, data, and network activity events) without needing technical SQL expertise. The feature uses generative AI to convert natural language questions into ready-to-use SQL queries you can run directly in the CloudTrail Lake console. This simplifies the process of exploring event data stores and retrieving insights such as error counts, top services used, and the causes of errors. This feature is also accessible through the AWS Command Line Interface (AWS CLI), providing additional flexibility for users who prefer command-line operations. The preview blog post provides step-by-step instructions on how to get started with the natural language query generation feature in CloudTrail Lake.

CloudTrail Lake generative AI–powered query results summarization capability in preview
Building on the capability of natural language query generation, we’re introducing a new AI-powered query results summarization feature in preview to further simplify the process of analyzing AWS account activity. With this feature, you can easily extract valuable insights from your AWS activity logs (only management, data, and network activity events) by automatically summarizing the key points from your query results in natural language, reducing the time and effort required to understand the information.

To try this feature, I go to the AWS CloudTrail console and choose Query under Lake in the navigation pane. I choose an event data store for my CloudTrail Lake query from the dropdown list in Event data store. You can use summarization regardless of whether the query was written manually or generated by generative AI. For this example, I will use the natural language query generation capability. In the Query generator, I enter the following prompt in the Prompt field using natural language:

How many errors were logged during the past month for each service and what was the cause of each error?

Then, I choose Generate query. The following SQL query is automatically generated:

SELECT eventsource,
    errorcode,
    errormessage,
    count(*) as errorcount
FROM a0******
WHERE eventtime >= '2024-10-14 00:00:00'
    AND eventtime <= '2024-11-14 23:59:59'
    AND (
        errorcode IS NOT NULL
        OR errormessage IS NOT NULL
    )
GROUP BY 1,
    2,
    3
ORDER BY 4 DESC;

I choose Run to get the results. To use the summarization capability, I choose Summarize results in the Query results tab. CloudTrail automatically analyzes the query results and provides a natural language summary of the key insights. It’s important to note that there’s a monthly quota of 3 MB for query results that can be summarized.

This new summarization capability can save you time and effort in understanding your AWS activity data by automatically generating meaningful summaries of the key findings.

Comprehensive dashboard capabilities
Lastly, let me tell you about the new dashboard capabilities of CloudTrail Lake to enhance visibility and analysis across your AWS environments.

The first one is a Highlights dashboard that provides you with an easy-to-view summary of the data captured in your CloudTrail Lake management and data events stored in event data stores. This dashboard makes it easier to quickly identify and understand important insights, such as the top failed API calls, trends in failed login attempts, and spikes in resource creation. It surfaces any anomalies or unusual trends in the data.

I go to the AWS CloudTrail console and choose Dashboard under Lake in the navigation pane to check out the Highlights dashboard. First, I enable Highlights dashboard by choosing Agree and enable Highlights.

I check out the Highlights dashboard once it populates with data.

The second addition to the new dashboard capabilities is a suite of 14 pre-built dashboards. These dashboards are designed for different personas and use cases. For example, the security-focused dashboards help you to track and analyze key security indicators, such as top access denied events, failed console login attempts, and users who have disabled multi-factor authentication (MFA). There are also pre-built dashboards for operational monitoring, highlighting trends in errors and availability issues, such as top APIs with throttling errors and top users with errors. You can also use the dashboards focused on specific AWS services such as Amazon EC2 and Amazon DynamoDB, which help you identify security risks or operational problems within those particular service environments.

You can create your own dashboards and optionally set schedules for refreshing them. This level of customization helps you tailor the CloudTrail Lake analysis capabilities to your precise monitoring and investigative needs across your AWS environments.

I switch to the Managed and custom dashboards to observe the custom and pre-built dashboards.

I choose IAM activity dashboard pre-built dashboard to observe overall IAM activity. You can choose Save as new dashboard to customize this dashboard.

To create a custom dashboard from scratch, I go to Dashboard under Lake in the navigation pane and choose Build my own dashboard. I enter a name in the Enter a name for the dashboard field and choose event data stores under Permissions, to visualize the events. Next, I choose Create dashboard.

Now, I can add widgets to my dashboard. You have the flexibility to customize your dashboards in multiple ways. You can select from a list of pre-built sample widgets using Add sample widget, or you can create your own custom widgets using Create new widget. For each widget, you can choose the type of visualization you prefer, such as a line graph, bar graph, or other options to best represent your data.

Now available
The new features in AWS CloudTrail Lake represent a major advancement in providing a comprehensive audit logging and analysis solution. These enhancements provide the ability to gain more profound understanding and conduct investigations more rapidly, assisting with more preventative monitoring and faster incident handling across your entire AWS environments.

You can now start using generative AI–powered natural language query generation in CloudTrail Lake in US East (N. Virginia), US West (Oregon), Asia Pacific (Mumbai), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), and Europe (London) AWS Regions.

CloudTrail Lake generative AI–powered query results summarization capability is available in preview in US East (N. Virginia), US West (Oregon), and Asia Pacific (Tokyo) Regions.

Enhanced filtering options, cross-account sharing of event data stores and dashboards are available in all the Regions where CloudTrail Lake is available, with the exception of generative AI–powered summarization feature on the Highlights dashboard being available only in US East (N. Virginia), US West (Oregon), and Asia Pacific (Tokyo) Regions.

Running queries will incur CloudTrail Lake query charges. For more details on pricing, visit AWS CloudTrail pricing.

— Esra

Track performance of serverless applications built using AWS Lambda with Application Signals

2024-11-21 Veliswa Boya

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/track-performance-of-serverless-applications-built-using-aws-lambda-with-application-signals/

In November 2023, we announced Amazon CloudWatch Application Signals, an AWS built-in application performance monitoring (APM) solution, to solve the complexity associated with monitoring performance of distributed systems for applications hosted on Amazon EKS, Amazon ECS, and Amazon EC2. Application Signals automatically correlates telemetry across metrics, traces, and logs, to speed up troubleshooting and reduce application disruption. By providing an integrated experience for analyzing performance in the context of your applications, Application Signals gives you improved productivity focusing on the applications that support your most critical business functions.

Today we’re announcing the availability of Application Signals for AWS Lambda to eliminate the complexities of manual setup and performance issues required to assess application health for Lambda functions. With CloudWatch Application Signals for Lambda, you can now collect application golden metrics (the incoming and outgoing volume of requests, latency, faults, and errors).

AWS Lambda abstracts away the complexity of the underlying infrastructure, enabling you to focus on building your application without having to monitor server health. This allows you to shift your focus toward monitoring the performance and health of your applications, which is necessary to operate your applications at peak performance and availability. This requires deep visibility into performance insights such as volume of transactions, latency spikes, availability drops, and errors for your critical business operations and application programming interfaces (APIs).

Previously, you had to spend signiﬁcant time correlating disjointed logs, metrics, and traces across multiple tools to establish the root cause of anomalies, increasing mean time to recovery (MTTR) and operational costs. Additionally, building your own APM solutions with custom code or manual instrumentation using open source (OSS) libraries was time-consuming, complex, operationally expensive, and often resulted in increased cold start times and deployment challenges when managing large ﬂeets of Lambda functions. Now, you can use Application Signals to seamlessly monitor and troubleshoot health and performance issues in serverless applications, without requiring any manual instrumentation or code changes from your application developers.

How it works
Using the pre-built, standardized dashboards of Application Signals, you can identify the root cause of performance anomalies in just a few clicks by drilling down into performance metrics for critical business operations and APIs. This helps you visualize application topology which shows interactions between the function and its dependencies. In addition, you can define Service Level Objectives (SLOs) on your applications to monitor specific operations that matter most to you. An example of an SLO could be to set a goal that a webpage should render within 2000 ms 99.9 percent of the time in a rolling 28-day interval.

Application Signals auto-instruments your Lambda function using enhanced AWS Distro for OpenTelemetry (ADOT) libraries. This delivers better performance such as lower cold start latency,
memory consumption, and function invocation duration, so you can quickly monitor your applications.

I have an existing Lambda function appsignals1 and I will configure Application Signals in the Lambda Console to collect various telemetry on this application.

In the Configuration tab of the function I select Monitoring and operations tools to enable both the Application signals and the Lambda service traces.

I have an application myAppSignalsApp that has this Lambda function attached as a resource. I’ve defined an SLO for my application to monitor specific operations that matter most to me. I’ve defined a goal that states that the application executes within 10 ms 99.9 percent of the time in a rolling 1-day interval.

It can take 5-10 minutes for Application Signals to discover the function after it’s been invoked. As a result you’ll need to refresh the Services page before you can see the service.

Now I’m in the Services page and I can see a list of all my Lambda functions that have been discovered by Application Signals. Any telemetry that is emitted will be displayed here.

I can then visualize the complete application topology from the Service Map and quickly spot anomalies across my service’s individual operations and dependencies, using the newly collected metrics of volume of requests, latency, faults, and errors. To troubleshoot, I can click into any point in time for any application metric graph to discover correlated traces and logs related to that metric, to quickly identify if issues impacting end users are isolated to an individual task or deployment.

Available now
Amazon CloudWatch Application Signals for Lambda is now generally available and you can start using it today in all AWS Regions where Lambda and Application Signals are available. Today, Application Signals is available for Lambda functions that use Python and Node.js managed runtimes. We’ll continue to add support for other Lambda runtimes in near future.

To learn more, visit the AWS Lambda developer guide and Application Signals developer guide. You can submit your questions to AWS re:Post for Amazon CloudWatch, or through your usual AWS Support contacts.

– Veliswa.

Noise

Tag Archives: news

AWS Database Migration Service now automates time-intensive schema conversion tasks using generative AI

Simplify governance with declarative policies

AWS Verified Access now supports secure access to resources over non-HTTP(S) protocols (in preview)

Announcing AWS Transfer Family web apps for fully managed Amazon S3 file transfers

Introducing Amazon OpenSearch Service and Amazon Security Lake integration to simplify security analytics

Use your on-premises infrastructure in Amazon EKS clusters with Amazon EKS Hybrid Nodes

Streamline Kubernetes cluster management with new Amazon EKS Auto Mode

Introducing storage optimized Amazon EC2 I8g instances powered by AWS Graviton4 processors and 3rd gen AWS Nitro SSDs

Now available: Storage optimized Amazon EC2 I7ie instances

New Amazon CloudWatch Database Insights: Comprehensive database observability from fleets to instances

New Amazon CloudWatch and Amazon OpenSearch Service launch an integrated analytics experience

Amazon FSx for Lustre increases throughput to GPU instances by up to 12x

Time-based snapshot copy for Amazon EBS

Announcing future-dated Amazon EC2 On-Demand Capacity Reservations

AWS Weekly Roundup: 197 new launches, AI training partnership with Anthropic, and join AWS re:Invent virtually (Nov 25, 2024)

Introducing a new experience for AWS Systems Manager

AWS named as a leader again in the Gartner Magic Quadrant for Distributed Hybrid Infrastructure

Improve your app authentication workflow with new Amazon Cognito features

Introducing new capabilities to AWS CloudTrail Lake to enhance your cloud visibility and investigations

Track performance of serverless applications built using AWS Lambda with Application Signals

The collective thoughts of the interwebz