Tag Archives: launch

Introducing a new experience for AWS Systems Manager

Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/introducing-a-new-experience-for-aws-system-manager/

Today, I’m excited to introduce a new and improved version of AWS Systems Manager that brings a highly requested cross-account, and cross-Region experience for managing nodes at scale.

The new System Manager experience provides centralized visibility of all your managed nodes which include various infrastructure types, such as Amazon Elastic Compute Cloud (EC2) instances, containers, virtual machines on other cloud providers, on-premise servers, and edge Internet of Things (IoT) devices. They are referred to as “managed nodes” when they have the Systems Manager Agent (SSM Agent) installed and are connected to Systems Manager.

If an SSM Agent stops working on a node for whatever reason, then Systems Manager loses connection to it and that node is then referred to as an “unmanaged node.” With the new update, Systems Manager can also help you to easily discover and troubleshoot unmanaged nodes. You can run and even schedule an automated diagnosis that provides you with recommended runbooks that you can execute to fix any issues and reestablish connection so they become managed nodes again.

Systems Manager is also now integrated with Amazon Q Developer, the most capable generative AI–powered assistant for software development. You can ask questions about your managed nodes to Amazon Q Developer using natural language and it will provide you with rapid insights plus links straight to Systems Manager where you can perform actions or continue to explore further.

With this release, you can also use AWS Organizations, to allow a delegated administrator to centrally manage nodes across the organization thanks to the new integration with Systems Manager.

the new systems manager experience

Let’s examine a quick example that helps to demonstrate some of these new capabilities.

Imagine a scenario where you are a cloud platform engineer leading a migration plan aiming to replace all nodes running Windows Server 2016 Datacenter in the organization. Let’s use the new Systems Manager experience to quickly gather information about all the nodes that needs to be included in our plan.

Step 1 – Asking Amazon Q Developer
The easiest starting point is using Amazon Q Developer to ask what you want to find using natural language. Using the AWS Console, I open the Amazon Q chatbot and type Find all of my managed nodes running Microsoft Windows Server 2016 Datacenter in my organization.

Amazon Q quickly comes back with an answer: it tells us that there are ten nodes that fit the criteria and provides a list with an overview of each one.

There is also a link that redirects to the new Explore nodes page in System Manager where we can learn more information. Let’s follow it.

Step 2 – Reviewing our infrastructure
The Explore nodes page provides a comprehensive overview of all managed nodes across your organization, with options to group and filter results for quick access. In this case, we can see that the results are already filtered by Operating system name providing us with a list of all the nodes that are running Microsoft Windows Server 2016 Datacenter.

This is a great start! We could just finish here by downloading the report and add those nodes to our migration plan, however, this page only shows you information about your managed nodes. Could it be that there are unmanaged nodes that need to included in our plan? Let’s find out.

Step 3 – Handling unmanaged nodes
Open the menu, and navigate to the Review node insights page. Here you can see a dashboard with widgets that provide insightful interactive charts that you can use to drill down and discover more information about your nodes or even take actions. For example, the Managed node types pie chart shows the types of managed nodes we have whereas the SSM Agent versions graph provides us with an overview of all the different versions of SSM Agent running on them. You can also customize this view by adding and replacing widgets.

We want to investigate any unmanaged nodes to make sure we don’t miss any that may need to be added to our migration plan. The Node summary widget clearly shows that there are two unmanaged nodes. This could mean that these nodes don’t have the SSM Agent installed in which case we will need to investigate them manually. However, it could also just mean there are issues with the SSM agent permissions or network connectivity preventing Systems Manager from managing these nodes and treating them like any other managed node. The new Systems Manager experience allows you easily troubleshoot and remediate SSM Agents issues so let’s attempt to do this now.

Start by selecting the piece of the chart displaying our unmanaged nodes. This pops up an option to initiate a comprehensive diagnosis of all our unmanaged nodes with only one click. Let’s run this.

The diagnosis reviews key configurations such as missing virtual private cloud (VPC) endpoints, misconfigured VPC DNS settings, and misconfigured instance security groups that may be preventing the SSM Agent from connecting to Systems Manager. After the scanning is complete, we can see that it displays two Misconfigured VPC endpoint findings. It also gives you a link that you can use to open a side panel containing a recommended runbook that you can execute to solve the issues as well as links to relevant documentation.

Choosing to execute the recommended runbook presents you with a detailed preview of the changes which include a thorough overview of the actions it’s going to take in addition to the input parameters used, a link to view a breakdown of the steps involved, and the target nodes for this execution.

Let’s choose to go ahead and select Execute. Keep in mind that this may incur costs, so make sure to review them before executing. You can keep an eye on progress on this page as it goes through the steps to attempt to fix the issues on each node.

Aha! After the remediation is complete, we can see that Systems Manager has found and corrected issues with the SSM Agent with two nodes. This means that Systems Manager is able to connect with the SSM Agent running in those nodes successfully making them “managed nodes.” We can verify this by returning to the Explore nodes page and noticing that the count of “unmanaged nodes” has been reduced to zero now.

Now that all of our nodes are managed, we’re ready to get a full list of all of those that need to be added to our migration plan.

Step 4 – Downloading a report
Back on the Explore nodes page we can see that the count for nodes running Microsoft Windows Server 2016 Datacenter has gone up from ten to twelve! That means that those previously unmanaged nodes that we fixed through the automated diagnosis are indeed running our target operating system.

This is exactly what we need so we choose to download a Report. You give it a file name, and then choose from a few options such as which columns to include. In this case, we choose to download a CSV file with a row containing the column names.

That’s it! We have our CSV with detailed information about the nodes that need upgrading across our entire infrastructure. And the best part? You can also use Systems Manager to automate the upgrade once you’re ready to go ahead with the migration.

Conclusion
Systems Manager is a critical tool for gaining visibility and control over your compute infrastructure and performing operational actions at scale. The new experience offers a centralized cross-account, cross-Region view of all your nodes in your AWS accounts, on-premises, and multicloud environments through a centralized dashboard, offering integration with Amazon Q Developer for natural language queries, and one-click SSM Agent troubleshooting. You can enable the new experience at no extra cost by navigating to the Systems Manager console and following the straightforward instructions.

To learn more, see the documentation for more detail about the new Systems Manager experience.

Check out this interactive demo for a full visual tour of this experience.

Improve your app authentication workflow with new Amazon Cognito features

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/improve-your-app-authentication-workflow-with-new-amazon-cognito-features/

Introduced 10 years ago, Amazon Cognito is a service that helps you implement customer identity and access management (CIAM) in your web and mobile applications. You can use Amazon Cognito for various use cases, from providing your customers to quickly add sign-in and sign-up experiences to your applications and authorization to securing machine-to-machine authentication and enabling role-based access to AWS resources.

Today, I’m excited to share a series of significant updates to Amazon Cognito. These enhancements aim to provide you with more flexibility, improved security, and a better user experience for your applications.

Here’s a quick summary:

A new developer-focused console experience
Amazon Cognito now offers a streamlined getting-started experience featuring a quick wizard and use case-specific recommendations. This new approach helps you set up configurations and reach your end users faster and more efficiently than ever before.

This is the new Amazon Cognito flow to help you quickly set up your application. You can get started in three steps:

  1. Choose the type of application you need to build
  2. Configure the sign-in options according to the type of your application
  3. Follow the instructions to integrate the sign-in and sign-up pages with your application

Then, select Create.

Amazon Cognito then automatically creates your application and a new user pool, which is a user directory for authentication and authorization. From here, you can review your sign-in page by selecting View login page or get started with the example code for your application. Furthermore, Amazon Cognito supports major application frameworks and offers detailed instructions for integrating them using standard OpenID Connect (OIDC) and OAuth open source libraries.

This is the new overview dashboard for your application. The user pool dashboard now provides important information in the Details section, as well as a set of Recommendations to help you continue your development journey.

On this page, you can customize your users’ sign-in and sign-up experience with the Managed Login feature. This is a good segue for me to provide you with a quick overview of the next new feature.

Introducing Managed Login
The introduction of Managed Login brings a new level of customization to Amazon Cognito. Managed Login handles the heavy lifting of availability, scaling, and security for your company. Once integrated, you automatically get all the new security patches and future features without further code changes.

This feature allows you to create personalized sign-up and sign-in experiences that are a seamless part of your company’s application for your end users.

Before you can use Managed Login, you need to assign a domain. There are two ways to do this: use a prefix domain, a randomly generated sub-domain of Amazon Cognito domain, or use your own custom domain to provide your users with a familiar domain name.

Then, you can choose your Branding version, selecting either Managed login or classic Hosted UI.

If you’re an existing Amazon Cognito user, you might be familiar with the classic Hosted UI feature. Managed Login is the improved version of Hosted UI, offering a new collection of web interfaces for sign-up and sign-in, built-in responsiveness for different screen sizes, multi-factor authentication, and password-reset activities in your user pool.

With Managed Login, you can use the new branding designer, a no-code visual editor for managed login assets and style, and a set of API operations for programmatic configuration or deployment via infrastructure-as-code with AWS CloudFormation.

With the branding designer, you have the flexibility to customize the look and feel of the entire user journey, from sign up and sign in to password recovery and multi-factor authentication. This feature provides a real time preview and convenient shortcuts to preview screens in different screen sizes and display modes before you launch it.

You can learn more about Managed Login by visiting the Managed Login documentation page.

Passwordless login support
The Managed Login feature also offers pre-built integrations for passwordless authentication methods, including signing in with passkeys, email OTP (one-time-password) and SMS OTP. Passkey support allows users to authenticate using cryptographic keys stored securely on their devices, offering better security compared to traditional passwords. This capability helps you implement low-friction and secure authentication methods without the need to understand and implement WebAuthn related protocols.

By reducing the friction associated with traditional password-based sign-ins, this feature simplifies application access for your users while maintaining high security standards.

Visit the user pools authentication flow documentation page to learn more about the passwordless login support.

More options on pricing tiers: Lite, Essentials and Plus
Amazon Cognito has introduced new user pool feature tiers: Lite, Essentials, and Plus. These tiers are designed to cater to different customer needs and use cases with the Essentials tier being the default tier for new users pools created by customers. This new tier structure also allows you to choose the most appropriate option based on your application requirements, with the flexibility to switch between tiers as needed.

To check your current tier, you can go to your application dashboard and select Feature plan. You can also select Settings from the navigation menu.

On this page, you’ll get detailed information for each tier and the option to downgrade or upgrade your plan.

Here’s a quick overview of each tier:

  1. Lite tier: Existing features such as user registration, password-based authentication, and social identity provider integration are now packaged in this tier. If you’re an existing Amazon Cognito user, you can continue using these features without making changes to your user pools. 

  2. Essentials tier: Offers comprehensive authentication and access control features, allowing you to implement secure, scalable, and customized sign-up and sign-in experiences for your application within minutes. It includes all capabilities in Lite along with supporting Managed Login and passwordless login options using passkeys, email, or SMS. Essentials also supports customizing access tokens and disallowing password reuse.

  3. Plus tier: Builds upon the Essentials tier, focusing on elevated security needs. It includes all Essentials features plus threat protection capabilities against suspicious login activity, detection of compromised credentials, risk-based adaptive authentication, and the ability to export user authentication event logs for threat analysis.

Pricing for the Lite, Essentials and Plus tiers is based on monthly active users. Customers currently using the advanced security features of Amazon Cognito should consider the Plus tier, which includes all the advanced security features, additional capabilities such as passwordless, and up to 60 percent savings as compared to using the standalone advanced security features.

If you want to learn about these new pricing tiers, see the Amazon Cognito pricing page.

Things you need to know

  • Availability – The Essentials and Plus tier are available in all AWS Regions where Amazon Cognito is available except AWS GovCloud (US) Regions.

  • Free tier on Lite and Essentials tiers – Customers on the Lite and Essentials tiers can enjoy the free tier each month that does not automatically expire. It is available to both existing and new AWS customers indefinitely. For more details on free tier, please visit the Amazon Cognito pricing page.

  • Extended pricing benefit for existing customers – Customers are eligible to upgrade their user pools without advanced security features (ASF) in their existing accounts to Essentials and pay the same price as Cognito user pools until November 30, 2025. To be eligible, customers’ accounts must have had at least 1 monthly active user (MAU) in the last 12 months on or before 10:00am Pacific Time, November 22, 2024. These customers are also eligible to create new user pools with Essentials tier at the same price as Cognito users pools in those accounts until November 30, 2025.

With these updates, you can implement secure, scalable, and customizable authentication solutions for your applications with Amazon Cognito.

Happy building,
— Donnie

Introducing new capabilities to AWS CloudTrail Lake to enhance your cloud visibility and investigations

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/introducing-new-capabilities-to-aws-cloudtrail-lake-to-enhance-your-cloud-visibility-and-investigations/

Today, I’m excited to announce new updates to AWS CloudTrail Lake, which is a managed data lake you can use to aggregate, immutably store, and query events recorded by AWS CloudTrail for auditing, security investigation, and operational troubleshooting.

The new updates in CloudTrail Lake are:

  • Enhanced filtering options for CloudTrail events
  • Cross-account sharing of event data stores
  • General availability of the generative AI–powered natural language query generation
  • AI-powered query results summarization capability in preview
  • Comprehensive dashboard capabilities, including a high-level overview dashboard with AI-powered insights (AI-powered insights is in preview), a suite of 14 pre-built dashboards for various use cases, and the ability to create custom dashboards with scheduled refreshes

Let’s look into the new features one by one.

Enhanced filtering options for CloudTrail events ingested into event data stores
Enhanced event filtering capabilities give you greater control over which CloudTrail events are ingested into your event data stores. These enhanced filtering options provide tighter control over your AWS activity data, improving the efficiency and precision of security, compliance, and operational investigations. Additionally, the new filtering options help you reduce your analysis workflow costs by ingesting only the most relevant event data into your CloudTrail Lake event data stores.

You can filter both management and data events based on attributes such as eventSource, eventType, eventName, userIdentity.arn, and sessionCredentialFromConsole.

I go to the AWS CloudTrail console and choose Event data stores under Lake in the navigation pane. I choose Create event data store. In the first step, I enter a name in the Event data store name field. For this demo, I leave other fields as default. You can choose the pricing and retention options that suit your needs. In the next step, I choose Managements events and Data events under CloudTrail events. You can include all the options you need under CloudTrail events. You also have the option to choose ingestion options. I choose Ingest events to start ingesting when it’s created. There may be scenarios, when you want to deselect the Ingest events option to stop an event data store from ingesting events. For example, you may be copying trail events to the event data store and do not want the event data store to collect any future events. You can also choose to enable ingestion for all accounts in your organization or include only the current region in your event data store.

The following example shows an out of the box template for filtering, which excludes any management events that are initiated by an AWS Service. I choose Advanced event collection under the Management events. I choose Exclude AWS service-initiated events from the Log selector template dropdown. You can also expand the JSON view to see how the filters actually apply.

Under the Data events, the following example creates a filter to include DynamoDB data events initiated by a certain user, helping me to log events based on an IAM principal. I choose DynamoDB as Resource type. I choose Custom as Log selector template. Under the Advanced event selector, I choose userIdentity.arn as Field and equals as Operator. I enter the user’s ARN as Value. I choose Next and choose Create event data store in the final step.

Now, I have my event data store that gives me granular control over the ingested CloudTrail data.

This expanded set of filtering options helps you to be more selective in capturing only the most relevant events for your security, compliance, and operational needs.

Cross-account sharing of event data stores
You can use the cross-account sharing feature of event data stores to enhance collaborative analysis within organizations. It enables secure sharing of event data stores with selected AWS principals through Resource-Based Policies (RBP). This functionality allows authorized entities to query shared event data stores within the same AWS Region where they were created. 

To use this feature, I go to the AWS CloudTrail console and choose Event data stores under Lake in the navigation pane. I choose an event data store from the list and navigate to its details page. I choose Edit in the Resource policy section. The following example policy includes a statement that allows root users in accounts 111111111111, 222222222222, and 333333333333 to run queries and get query results on the event data store owned by account ID 999999999999. I choose Save changes to save the policy.

Generative AI–powered natural language query generation in CloudTrail Lake is now generally available
In June, we announced this feature for CloudTrail Lake in preview. With this launch, you can generate SQL queries using natural language questions to easily explore and analyze AWS activity logs (only management, data, and network activity events) without needing technical SQL expertise. The feature uses generative AI to convert natural language questions into ready-to-use SQL queries you can run directly in the CloudTrail Lake console. This simplifies the process of exploring event data stores and retrieving insights such as error counts, top services used, and the causes of errors. This feature is also accessible through the AWS Command Line Interface (AWS CLI), providing additional flexibility for users who prefer command-line operations. The preview blog post provides step-by-step instructions on how to get started with the natural language query generation feature in CloudTrail Lake.

CloudTrail Lake generative AI–powered query results summarization capability in preview
Building on the capability of natural language query generation, we’re introducing a new AI-powered query results summarization feature in preview to further simplify the process of analyzing AWS account activity. With this feature, you can easily extract valuable insights from your AWS activity logs (only management, data, and network activity events) by automatically summarizing the key points from your query results in natural language, reducing the time and effort required to understand the information.

To try this feature, I go to the AWS CloudTrail console and choose Query under Lake in the navigation pane. I choose an event data store for my CloudTrail Lake query from the dropdown list in Event data store. You can use summarization regardless of whether the query was written manually or generated by generative AI. For this example, I will use the natural language query generation capability. In the Query generator, I enter the following prompt in the Prompt field using natural language:

How many errors were logged during the past month for each service and what was the cause of each error?

Then, I choose Generate query. The following SQL query is automatically generated:

SELECT eventsource,
    errorcode,
    errormessage,
    count(*) as errorcount
FROM a0******
WHERE eventtime >= '2024-10-14 00:00:00'
    AND eventtime <= '2024-11-14 23:59:59'
    AND (
        errorcode IS NOT NULL
        OR errormessage IS NOT NULL
    )
GROUP BY 1,
    2,
    3
ORDER BY 4 DESC;

I choose Run to get the results. To use the summarization capability, I choose Summarize results in the Query results tab. CloudTrail automatically analyzes the query results and provides a natural language summary of the key insights. It’s important to note that there’s a monthly quota of 3 MB for query results that can be summarized.

This new summarization capability can save you time and effort in understanding your AWS activity data by automatically generating meaningful summaries of the key findings.

Comprehensive dashboard capabilities
Lastly, let me tell you about the new dashboard capabilities of CloudTrail Lake to enhance visibility and analysis across your AWS environments.

The first one is a Highlights dashboard that provides you with an easy-to-view summary of the data captured in your CloudTrail Lake management and data events stored in event data stores. This dashboard makes it easier to quickly identify and understand important insights, such as the top failed API calls, trends in failed login attempts, and spikes in resource creation. It surfaces any anomalies or unusual trends in the data.

I go to the AWS CloudTrail console and choose Dashboard under Lake in the navigation pane to check out the Highlights dashboard. First, I enable Highlights dashboard by choosing Agree and enable Highlights.

I check out the Highlights dashboard once it populates with data.

The second addition to the new dashboard capabilities is a suite of 14 pre-built dashboards. These dashboards are designed for different personas and use cases. For example, the security-focused dashboards help you to track and analyze key security indicators, such as top access denied events, failed console login attempts, and users who have disabled multi-factor authentication (MFA). There are also pre-built dashboards for operational monitoring, highlighting trends in errors and availability issues, such as top APIs with throttling errors and top users with errors. You can also use the dashboards focused on specific AWS services such as Amazon EC2 and Amazon DynamoDB, which help you identify security risks or operational problems within those particular service environments.

You can create your own dashboards and optionally set schedules for refreshing them. This level of customization helps you tailor the CloudTrail Lake analysis capabilities to your precise monitoring and investigative needs across your AWS environments.

I switch to the Managed and custom dashboards to observe the custom and pre-built dashboards.

I choose IAM activity dashboard pre-built dashboard to observe overall IAM activity. You can choose Save as new dashboard to customize this dashboard.

To create a custom dashboard from scratch, I go to Dashboard under Lake in the navigation pane and choose Build my own dashboard. I enter a name in the Enter a name for the dashboard field and choose event data stores under Permissions, to visualize the events. Next, I choose Create dashboard.

Now, I can add widgets to my dashboard. You have the flexibility to customize your dashboards in multiple ways. You can select from a list of pre-built sample widgets using Add sample widget, or you can create your own custom widgets using Create new widget. For each widget, you can choose the type of visualization you prefer, such as a line graph, bar graph, or other options to best represent your data.

Now available
The new features in AWS CloudTrail Lake represent a major advancement in providing a comprehensive audit logging and analysis solution. These enhancements provide the ability to gain more profound understanding and conduct investigations more rapidly, assisting with more preventative monitoring and faster incident handling across your entire AWS environments.

You can now start using generative AI–powered natural language query generation in CloudTrail Lake in US East (N. Virginia), US West (Oregon), Asia Pacific (Mumbai), Asia Pacific (Sydney), Asia Pacific (Tokyo), Canada (Central), and Europe (London) AWS Regions.

CloudTrail Lake generative AI–powered query results summarization capability is available in preview in US East (N. Virginia), US West (Oregon), and Asia Pacific (Tokyo) Regions.

Enhanced filtering options, cross-account sharing of event data stores and dashboards are available in all the Regions where CloudTrail Lake is available, with the exception of generative AI–powered summarization feature on the Highlights dashboard being available only in US East (N. Virginia), US West (Oregon), and Asia Pacific (Tokyo) Regions.

Running queries will incur CloudTrail Lake query charges. For more details on pricing, visit AWS CloudTrail pricing.

— Esra

Track performance of serverless applications built using AWS Lambda with Application Signals

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/track-performance-of-serverless-applications-built-using-aws-lambda-with-application-signals/

In November 2023, we announced Amazon CloudWatch Application Signals, an AWS built-in application performance monitoring (APM) solution, to solve the complexity associated with monitoring performance of distributed systems for applications hosted on Amazon EKS, Amazon ECS, and Amazon EC2. Application Signals automatically correlates telemetry across metrics, traces, and logs, to speed up troubleshooting and reduce application disruption. By providing an integrated experience for analyzing performance in the context of your applications, Application Signals gives you improved productivity focusing on the applications that support your most critical business functions.

Today we’re announcing the availability of Application Signals for AWS Lambda to eliminate the complexities of manual setup and performance issues required to assess application health for Lambda functions. With CloudWatch Application Signals for Lambda, you can now collect application golden metrics (the incoming and outgoing volume of requests, latency, faults, and errors).

AWS Lambda abstracts away the complexity of the underlying infrastructure, enabling you to focus on building your application without having to monitor server health. This allows you to shift your focus toward monitoring the performance and health of your applications, which is necessary to operate your applications at peak performance and availability. This requires deep visibility into performance insights such as volume of transactions, latency spikes, availability drops, and errors for your critical business operations and application programming interfaces (APIs).

Previously, you had to spend significant time correlating disjointed logs, metrics, and traces across multiple tools to establish the root cause of anomalies, increasing mean time to recovery (MTTR) and operational costs. Additionally, building your own APM solutions with custom code or manual instrumentation using open source (OSS) libraries was time-consuming, complex, operationally expensive, and often resulted in increased cold start times and deployment challenges when managing large fleets of Lambda functions. Now, you can use Application Signals to seamlessly monitor and troubleshoot health and performance issues in serverless applications, without requiring any manual instrumentation or code changes from your application developers.

How it works
Using the pre-built, standardized dashboards of Application Signals, you can identify the root cause of performance anomalies in just a few clicks by drilling down into performance metrics for critical business operations and APIs. This helps you visualize application topology which shows interactions between the function and its dependencies. In addition, you can define Service Level Objectives (SLOs) on your applications to monitor specific operations that matter most to you. An example of an SLO could be to set a goal that a webpage should render within 2000 ms 99.9 percent of the time in a rolling 28-day interval.

Application Signals auto-instruments your Lambda function using enhanced AWS Distro for OpenTelemetry (ADOT) libraries. This delivers better performance such as lower cold start latency,
memory consumption, and function invocation duration, so you can quickly monitor your applications.

I have an existing Lambda function appsignals1 and I will configure Application Signals in the Lambda Console to collect various telemetry on this application.

In the Configuration tab of the function I select Monitoring and operations tools to enable both the Application signals and the Lambda service traces.

I have an application myAppSignalsApp that has this Lambda function attached as a resource. I’ve defined an SLO for my application to monitor specific operations that matter most to me. I’ve defined a goal that states that the application executes within 10 ms 99.9 percent of the time in a rolling 1-day interval.

It can take 5-10 minutes for Application Signals to discover the function after it’s been invoked. As a result you’ll need to refresh the Services page before you can see the service.

Now I’m in the Services page and I can see a list of all my Lambda functions that have been discovered by Application Signals. Any telemetry that is emitted will be displayed here.

I can then visualize the complete application topology from the Service Map and quickly spot anomalies across my service’s individual operations and dependencies, using the newly collected metrics of volume of requests, latency, faults, and errors. To troubleshoot, I can click into any point in time for any application metric graph to discover correlated traces and logs related to that metric, to quickly identify if issues impacting end users are isolated to an individual task or deployment.

Available now
Amazon CloudWatch Application Signals for Lambda is now generally available and you can start using it today in all AWS Regions where Lambda and Application Signals are available. Today, Application Signals is available for Lambda functions that use Python and Node.js managed runtimes. We’ll continue to add support for other Lambda runtimes in near future.

To learn more, visit the AWS Lambda developer guide and Application Signals developer guide. You can submit your questions to AWS re:Post for Amazon CloudWatch, or through your usual AWS Support contacts.

– Veliswa.

Announcing a visual update to the AWS Management Console (preview)

Post Syndicated from Prasad Rao original https://aws.amazon.com/blogs/aws/announcing-a-visual-update-to-the-aws-management-console-preview/

Today, we are announcing a visual update to the AWS Management Console in preview. We are rolling out this update by using the latest version of Cloudscape, the Amazon Web Services (AWS) design system used to build intuitive, inclusive, and meaningful AWS experiences at scale.

In this post, I describe how the visual update makes it easier for you to scan content, focus on the key information, and find what you are looking for more effectively while preserving the familiar and consistent experience of the AWS Management Console.

AWS Management console home page - previous

AWS Management console home page - Visual Update

Improved readability
A revised typography scale and improved treatment of headings result in a stronger visual hierarchy, which helps you to better locate and understand your data. A refined use of color and weight across text elements help you differentiate key pieces of information faster. For example, you’ll see that labels in form fields are now more prominent, which eases scanning. The same applies to keys in key-value pairs and sections across components, such as service navigation, expandable elements, and tabs.

Cloudfront distribution console screenshot - Previous vs Visual Update

We improved the color palette, made it more vibrant, and simplified the color treatment of interactive elements. For example, secondary buttons, links, tokens, and interactive states for numerous interface elements are now blue, making it easier for you to interact with the content on the screen and contributing to improving task efficiency.

Screenshot showing improved color - Previous vs Visual Update

Improved focus in light and dark mode
Reduced visual complexity supports user focus. We replaced drop shadows with a new thinner stroke on main content wrappers, such as cards, panels, and containers, and unified the use of border styles across components. This reduces visual noise and optimizes the space inside the layout. Shadows are now reserved to add emphasis on specific interactive and transient elements, which helps simplify visual depth and improves the overall content hierarchy.

Screenshot showing improved focus - Previous vs Visual Update

We also released updates to dark mode to address the need for clearer differentiation between elements on the page. These changes include an update to the color ramp and improved contrast between interactive states across components.

Screenshot comparing dark mode of AWS Management Console home page - Previous vs Visual Update

Modernized interface
We modernized the interface while retaining familiarity to continue to offer predictable and recognizable experiences across AWS. The user experience is now easier on the eyes, thanks to the use of rounder shapes, brighter colors, and improved layout treatment. These updates create a smoother, more natural appearance, making the interface more visually pleasing.

To deliver a more delightful experience and support visual storytelling, we also introduced a whole new family of illustrations and motion while still offering the highest accessibility standards.

Example of an illustration introduced

Improved information density
We optimized information density by reducing unused space, leading to more content visible on the screen. Related data is now displayed closer together, reinforcing visual grouping. Space within content wrappers such as cards and containers has been minimized, so you can consume more information at once. The new layout is centered and wider, optimizing the experience to serve larger screen sizes than before. The visual update makes it easier to consume information, which creates a better and friendlier experience within the AWS Management Console.

Showing Improved information density on AWS Lambda Create Function Screen - Previous vs Visual Update

Showing Improved information density in tabular format - Previous vs Visual Update

Additionally, we introduced Toolbar, a new way to navigate and access contextual tools and features. This helps you perform your tasks while maximizing the amount of content available on screen.

Screenshot of toolbar introduced

Improved consistency
The interface is now more distinctive and consistent. Refreshed colors, iconography, and shapes help deliver a more dynamic and expressive experience while reinforcing a unified and cohesive journey across all AWS experiences.

Available now
You can start experiencing the visual update now in selected consoles across all AWS Regions by visiting the AWS Management Console. We’ll be extending the update across all services. Thanks to the new visual treatment, you can now benefit from an experience that’s more readable and intuitive and that contributes to improved overall task efficiency.

Introducing Amazon CloudFront VPC origins: Enhanced security and streamlined operations for your applications

Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/introducing-amazon-cloudfront-vpc-origins-enhanced-security-and-streamlined-operations-for-your-applications/

I’m happy to introduce the release of Amazon CloudFront Virtual Private Cloud (VPC) origins, a new feature that enables content delivery from applications hosted in private subnets within their Amazon Virtual Private Cloud (Amazon VPC). This makes it easy to secure web applications, allowing you to focus on growing your businesses while improving security and maintaining high-performance and global scalability with CloudFront.

Customers serving content from Amazon Simple Storage Solution (Amazon S3), AWS Elemental Services and AWS Lambda Function URLs can use Origin Access Control as a managed solution to secure their origins, and make CloudFront the single front-door to your application. However, this was more difficult to achieve for applications that are hosted on Amazon Elastic Compute Cloud (Amazon EC2) or using load balancers, because you had to create your own solution to achieve the same result. You would have to use a combination of methods such as using access control lists (ACLs), managing firewall rules, or using logic such as header validation and a few other techniques to ensure that the endpoint remained exclusive to CloudFront.

CloudFront VPC origins removes the need for this kind of undifferentiated work by offering a managed solution that can be used to point CloudFront distributions directly to Application Load Balancers (ALBs), Network Load Balancers (NLBs), or EC2 instances inside your private subnets. This ensures that CloudFront becomes the sole ingress point for those resources with minimum configuration effort, providing you with improved performance and a cost-saving opportunity because it also eliminates the need for public IP addresses.

Configuring a CloudFront VPC origin
CloudFront VPC origins is available at no additional cost, making it an accessible option for all AWS customers. It can be integrated with new or existing CloudFront distributions using the Amazon CloudFront console or the AWS Command Line Interface (AWS CLI).

Imagine that you have an application hosted privately on an AWS Fargate for Amazon ECS fronted through an ALB. Let’s create a CloudFront distribution that uses the ALB directly inside the private subnet.

Start by navigating to the CloudFront console and select the new menu option: VPC origins.

vpc origins page

Creating a new VPC origin is straightforward. You only need to select from a few options. In the Origin ARN, you can search for available resources that are hosted in private subnets or enter it directly. You select the resources that you want, choose a friendly name for your VPC origin alongside some security options, and then confirm. Please note that, at launch, the VPC origin resource must be in the same AWS Account as the CloudFront distribution, although support for resources across all accounts is coming soon.

creating a vpc origin

After the creation process is complete, your VPC origin will be deployed and ready to go! You can check its status on the VPC origins page.

With this, we have created a CloudFront distribution that serves content directly from a resource hosted on a private subnet in a few clicks! After your VPC origin is created, you can navigate to your Distribution window, and add the VPC origin to your Distribution by either selecting the ARN from the dropdown or copy-pasting the ARN manually.

Remember, though, that it’s important to still continue to layer your application’s security by using services such as AWS Web Application Firewall (WAF) to protect from web exploits, or AWS Shield for managed DDos protection, and other services to achieve a full-spectrum protection.

Conclusion
CloudFront VPC Origins offers a new way for organizations to deliver secure, high-performance applications by enabling CloudFront distributions to serve content directly from resources hosted within private subnets. This reduces the complexity and cost of maintaining public-facing origins while ensuring that your application remains secure.

To learn more, see the getting started guide.

Matheus Guimaraes | @codingmatheus

Amazon CloudFront now accepts your applications’ gRPC calls

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/amazon-cloudfront-now-accepts-your-applications-grpc-calls/

Starting today, you can deploy Amazon CloudFront, our global content delivery network (CDN), in front of your gRPC API endpoints.

gRPC is a modern, efficient, and language-agnostic framework for building APIs. It uses Protocol Buffers (protobuf) as its interface definition language (IDL), which enable you to define services and message types in a platform-independent manner. With gRPC, communication between services is achieved through lightweight and high-performance remote procedure calls (RPCs) over HTTP/2. This promotes efficient and low-latency communication across services, making it ideal for microservices architectures.

gRPC offers features such as bidirectional streaming, flow control, and automatic code generation for a variety of programming languages. It’s well-suited for scenarios in which you require high performance, efficient communication, and real-time data streaming. If your application needs to handle a large amount of data or requires low-latency communication between client and server, gRPC can be a good choice. However, gRPC might be more challenging to learn compared to REST. For example, gRPC relies on the protobuf serialization format, which requires developers to define their data structures and service methods in .proto files.

I see two benefits of deploying CloudFront in front of your gRPC API endpoints.

First, it allows the reduction of latency between the client application and your API implementation. CloudFront offers a global network of over 600+ edge locations with intelligent routing to the closest edge. Edge locations provide TLS termination and optional caching for your static content. CloudFront transfers client application requests to your gRPC origin through the fully managed, low-latency, and high-bandwidth private AWS network.

Secondly, your applications benefit from additional security services deployed on edge locations, such as traffic encryption, the validation of the HTTP headers through AWS Web Application Firewall, and AWS Shield Standard protection against distributed denial of service (DDoS) attacks.

Let’s see it in action
To start this demo, I use the gRPC route-guide demo from the official gRPC code repository. I deploy this example application in a container for ease of deployment (but any other deployment option is supported too).

I use this Dockerfile

FROM python:3.7
RUN pip install protobuf grpcio
COPY ./grpc/examples/python/route_guide .
CMD python route_guide_server.py
EXPOSE 50051

I also use the AWS Copilot command line to deploy my container on Amazon Elastic Container Service (Amazon ECS). The Copilot command prompts me to collect the information it requires to build and deploy the container. Then, it creates the ECS cluster, the ECS service, and the ECS task automatically. It also creates a TLS certificate and the load balancer for me. I test the client application by modifying line 122 to use the DNS name of the load balancer listener endpoint. I also change the client application code to use grpc.secure_channel instead of grpc.insecure_channel because the load balancer provides the application with an HTTPS endpoint.

gRPC client application demo - source code with ALB

When I’m confident my API is correctly deployed and working, I proceed and configure CloudFront.

First, in the CloudFront section of the AWS Management Console, I select Create distribution.

Under Origin, I enter my gRPC endpoint DNS name as Origin domain. I enable HTTPS only as Protocol and leave the HTTPS port as is (443). Then I choose a Name for the distribution.

CloudFront - Add origin and name

Under Viewer, I select HTTPS only as Viewer protocol policy. Then, I select GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE as Allowed HTTP methods. I select Enable for Allow gRPC requests over HTTP/2.

CloudFront - Viewer Policy

Under Cache key and origin requests, I select AllViewer as Origin request policy.

The default cache policy is CacheOptimized, but gRPC isn’t cacheable API traffic. Therefore, I select CachingDisabled as Cache policy.

CloudFront - Cache policy

AWS WAF helps protect you against common web exploits and bots that can affect availability, compromise security, or consume excessive resources. For gRPC traffic, AWS WAF can inspect the HTTP headers of the request and enforce access control. It doesn’t inspect the request body in protobuf format.

For this demo, I choose to not use AWS WAF. Under Web Application Firewall (WAF), I select Do not enable security protections.

CloudFront - Security

I also keep all the other options with their default value. HTTP/2 support is selected by default. Do not disable it because it is required for gRPC.

Finally, I select Create distribution.

CloudFront - Create distribution

There is only one switch to enable gRPC on top of the usual setup. When turned on, with HTTP/2 and HTTP POST enabled, CloudFront detects gRPC client traffic and forwards it to your gRPC origin.

After a few minutes, the distribution is ready. I copy and paste the endpoint URL of the CloudFront distribution, and I change the client-side app to make it point to CloudFront instead of the previously created load balancer.

gRPC client application demo - source code

I test the application again, and it works.

gRPC client application demo - execution

Pricing and Availability
gRPC origins are available on all the more than 600 CloudFront edge locations at no additional cost. The usual requests and data transfer fees apply.

Go and point your CloudFront origin to a gRPC endpoint today.

— seb

Expanded resource awareness in Amazon Q Developer

Post Syndicated from Brendan Jenkins original https://aws.amazon.com/blogs/devops/expanded-resource-awareness-in-amazon-q-developer/

Recently, Amazon Q Developer announced expanded support for account resource awareness with Amazon Q in the AWS Management Console along with the general availability of Amazon Q Developer in AWS Chatbot, enabling you to ask questions from Microsoft Teams or Slack. Additionally, Amazon Q will now provide context-aware assistance for your questions about resources in your account depending on where you are in the console. Amazon Q in the console gives you the ability to use natural language with the Amazon Q Developer chat capability to list resources in your AWS account, get specific resource details, and ask about related resources, launched in preview on April 30, 2024.

In this blog, I will highlight the new expanded functionality of this feature in Amazon Q Developer including understanding relationships between account resources, context-awareness, and the general availability of the AWS Chatbot integration with Microsoft Teams and Slack.

Expanded account resource awareness with Amazon Q Developer

Prior to the launch of the expanded support, you could ask Amazon Q Developer to list resources in your AWS Account with prompts such as “List all my EC2 instances in us-east-1” and the service would list all your Amazon Elastic Compute Cloud (Amazon EC2) instances. Now, with the expanded support, you can ask more complex questions about your AWS account resources. I will show a few examples in this section of this post.

For our first example, imagine that you’re a developer who is responsible for maintaining code as a part of the software development lifecycle (SDLC) and you frequently use AWS Lambda for development and Amazon Relational Database Service (RDS) in the backend as a part of your development process. With this new update, a developer could open a new Q chat in the AWS Management Console, and enter a prompt such as: “Which RDS clusters are due for an update?”

User entering prompt Amazon Q Developer chat in the AWS management console about listing all RDS clusters that need updates in their account and Amazon Q listing those Databases.

Figure 1: Amazon Q Developer listing RDS clusters needing an update

As a result, the Amazon Q Developer console chat will return a list of all your Amazon RDS clusters that have available updates as shown in Figure 1 above.

Now, for another example, you want to update any Lambda functions in your AWS account that had a Simple Notification Service (SNS) topic as a trigger due to moving to a new SNS topic you recently created. To identify which SNS topics are still being used, you could enter a prompt such as “List all the SNS topics that trigger a lambda function.”

User entering prompt Amazon Q Developer chat in the AWS management console about listing all SNS topics that trigger a lambda function and Amazon Q listing the SNS topics as an output.

Figure 2: Amazon Q listing SNS topics that are lambda triggers

As shown in the prior example, Amazon Q Developer was able to identify any SNS topics in the form of Amazon resource name (ARN) that was set to trigger a lambda function in the AWS account as intended.

Additionally, you can ask a follow up question in the same chat to investigate more. You can send a prompt such as “Which lambda function uses the arn:aws:sns:us-east-1:76859XXXX:FailoverHealthcheck SNS topic?”

User entering prompting Amazon Q Developer chat with a follow up question in the AWS management console about which Lambda is associated with an SNS topic.

Figure 3: Asking Q Developer a follow up question about a resource

From Figure 3 above, you can see that there is a Lambda function/endpoint associated with the SNS topic resource that Amazon Q Developer was able to identify.

Outside of the examples above, here are some other prompts/examples that can be explored for the expanded support:

– “Do I have any ECS clusters with pending tasks?”

– “Are there any ECS clusters in my account with services in DRAINING status?”

Amazon Q Developer understands where you are in the console

Amazon Q Developer in the AWS Management Console now provides context-aware assistance for your questions about resources in your account. This feature allows you to ask questions directly related to the console page you’re viewing, eliminating the need to specify the service or resource in your query. Q Developer uses the current page as additional context to provide more accurate and relevant responses, streamlining your interaction with AWS services and resources.

Prior to the update, a user would have to prompt, “What is the public IPv4 address of my instance i-08ccXXXXXX?”  Now, if you are viewing an EC2 instance in the console and prompt Amazon Q, “What is the public IPv4 address of my instance?” you will not need to specify the instance you are referring to.

User entering prompt Amazon Q Developer chat in the AWS management console about what the IP address is of the instance on the page.

Figure 4: Asking Amazon Q about an EC2 instance being viewed

In figure 4 above, Amazon Q’s console chat was able to use its context-awareness to pick up on what the IPv4 address was on the console page where I was currently working, despite me not specifying which instance I was referring to.

AWS ChatBot can now answer questions about AWS resources in Microsoft Teams and Slack

Recently, we announced the general availability of Amazon Q Developer in AWS Chatbot, which provides answers to customers’ AWS resource related queries in Microsoft Teams and Slack. This gives teams the ability to quickly find relevant resources to troubleshoot issues using natural language queries in the chat channels of Microsoft Teams or Slack.

For example, you could integrate the AWS Chatbot Service with Amazon Q Developer to allow you to enter a prompt in Slack such as “@aws show EC2 instances in running state in us-east-1”.

User entering prompt in slack to ask the AWS Chatbot about EC2 resources and Amazon Q responding

Figure 5: Amazon Q listing all EC2 resources in Slack

As shown in figure 5 above, Amazon Q was able to list all the EC2 resources and place them into a slack channel showing an example of the functionality in action.

Conclusion

Amazon Q Developer has enhanced its cloud resource management capabilities, enabling more intuitive and intelligent interactions with AWS resources. The new features allow developers to ask complex, context-aware questions about their cloud infrastructure directly through the AWS Management Console, Microsoft Teams, and Slack. Users can now easily discover new details about specific resources with natural language queries that provide precise, contextual information. These improvements represent a significant step forward in simplifying cloud resource management, making it faster and more user-friendly for development teams to understand, track, and maintain their AWS environments. To learn more about chatting with your AWS resources, check out Console documentation and AWS Chatbot documentation.

About the authors

Brendan Jenkins

Brendan Jenkins is a Tech Lead Solutions Architect at Amazon Web Services (AWS) working with Enterprise AWS customers providing them with technical guidance and helping achieve their business goals. He has an area of specialization in DevOps and Machine Learning technology.

Streamline container application networking with built-in Amazon ECS support in Amazon VPC Lattice

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/streamline-container-application-networking-with-native-amazon-ecs-support-in-amazon-vpc-lattice/

Since its launch, Amazon VPC Lattice has streamlined complex networking tasks. As a result, my perspective on how to build and connect modern, multi-service applications has changed. As my colleague Danilo wrote in his post announcing the general availability of VPC Lattice:

“By using VPC Lattice, you can focus on your application logic and improve productivity and deployment flexibility with consistent support for instances, containers, and serverless computing.”

Today, we’re announcing Amazon VPC Lattice built-in support for Amazon Elastic Container Service (Amazon ECS). With this new built-in integration, Amazon ECS services can now be directly associated with VPC Lattice target groups without the need for intermediate load balancers.

Here’s a quick look at how you can find Amazon VPC Lattice integration while creating an Amazon ECS service:

The Amazon VPC Lattice integration with Amazon ECS works by registering and deregistering IP addresses from ECS tasks within a service as targets in a VPC Lattice target group. As ECS tasks for the service are launched, Amazon ECS will automatically register those tasks to the VPC Lattice target group.

Furthermore, if ECS tasks fail VPC Lattice health checks, Amazon ECS will automatically replace the tasks. Also, if any task is terminated or scales down, it’s removed from the target group.

Using the Amazon VPC Lattice integration
Let me walk you through how to use this new integration. In the following demo, I will deploy a simple application server running as an ECS service and configure the integration with VPC Lattice. Then, I’ll test the application server by connecting to the VPC Lattice domain name without having to configure additional load balancers on Amazon ECS.

Before I can start with this integration, I need to make sure Amazon ECS will have the required permissions to register and deregister targets into VPC Lattice. To learn more, please visit the Amazon ECS infrastructure IAM role documentation page.

To use the integration with VPC Lattice, I need to define a task definition with at least one container and one port mapping. This is an example of my task definition.

{
    "containerDefinitions": [
        {
            "name": "webserver",
            "image": "public.ecr.aws/ecs-sample-image/amazon-ecs-sample:latest",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "web-80-tcp",
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            ...
            *redacted for brevity*
}

Then, I navigate to my ECS cluster and choose Create.

Next, I need to select the task definition and assign the service name.

In the VPC Lattice integration section, I choose Turn on VPC Lattice to start configuring the target group for VPC Lattice. I don’t need to specify a load balancer because I’ll use VPC Lattice. By default, VPC Lattice will use a round-robin routing algorithm to route requests to healthy targets.

Now, I can start defining the integration for my ECS service in VPC Lattice. First, I select the infrastructure role for Amazon ECS. Then, I need to select the virtual private cloud (VPC) where I want my service to run. After that, I need to define the Target groups that will receive traffic. After I’m done configuring the service with VPC Lattice integration, I create this service.

After a few minutes, I have my ECS service ready. I navigate to the service and choose Configuration and networking. If I scroll down to the VPC Lattice section, I can see the VPC Lattice target group created.

To get more information on this target group, I select the target group name, which will redirect me to the VPC Lattice target group page. Here, I can see that Amazon ECS successfully registered the IP address of the running task.

Now, I need to create a VPC Lattice service and service network. My preference is always to create the VPC Lattice service then associate with the VPC Lattice service network later on. So, let’s do that.

I choose Services under the VPC Lattice section and choose Create service.

I fill in all the details required to create a VPC Lattice service and choose Next.

Then, I add a listener, and for the Forward to target group on the Listener default action, I select the newly created target group.

On the next page, because I’m going to create the VPC Lattice service network later, I skip this step and choose Next, review the configurations, and create the service.

With VPC Lattice service created, now it’s time to create VPC Lattice service networks. I navigate to Service networks under the VPC Lattice section and choose Create service network.

First, I fill the VPC Lattice service network name.

Then, on the Service associations page, I select the service that I have created.

I associate this service network to my VPC as well as the security group.

For the simplicity of this demo, I set None for the Auth type. However, I highly recommend you to read how you can use IAM to manage access to VPC Lattice. Then, I choose Create service network.

At this stage, we have everything setup for this integration. My VPC Lattice service network is now associated with my VPC Lattice service and my VPC.

With everything set up, I copy the Domain name from my VPC Lattice service page.

Then, to access the service, I log in to the instance in the same VPC and call the service by using the domain name from VPC Lattice.

[ec2-user@ ~]$ curl http://service-a-XYZ.XYZ.vpc-lattice-svcs.XYZ.on.aws

"Hello there! I'm Amazon ECS."

One thing to note is if you’re not receiving traffic to your Amazon ECS workloads, check the security groups as described in the Control traffic in VPC Lattice using security groups documentation page.

I’m personally excited about this integration because it unlocks various possibilities while streamlining application architectures and improving overall system reliability. Now that all AWS compute types are inherently supported in VPC Lattice, I can unify services across all my ECS clusters, AWS accounts, and VPCs.

Things to know
Here are a couple of important points to note:

Try this new capability of Amazon VPC Lattice today and see how it can streamline your container application communication running on Amazon ECS.

Happy building!

— Donnie Prakoso

AWS Lambda SnapStart for Python and .NET functions is now generally available

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/aws-lambda-snapstart-for-python-and-net-functions-is-now-generally-available/

Today, we’re announcing the general availability of AWS Lambda SnapStart for Python and .NET functions that delivers faster function startup performance, from several seconds to as low as sub-second, typically with minimal or no code changes in Python, C#, F#, and Powershell.

In November 28, 2022, we introduced Lambda SnapStart for Java functions to improve startup performance by up to 10 times. With Lambda SnapStart, you can reduce outlier latencies that come from initializing functions, without having to provision resources or spend time implementing complex performance optimizations.

Lambda SnapStart works by caching and reusing the snapshotted memory and disk state of any one-time initialization code, or code that runs only the first time a Lambda function is invoked. Lambda takes a Firecracker microVM snapshot of the memory and disk state of the initialized execution environment, encrypts the snapshot, and caches it for low-latency access.

When you invoke the function version for the first time, and as the invocations scale up, Lambda resumes new execution environments from the cached snapshot instead of initializing them from scratch, improving startup latency. Lambda SnapStart makes it easy to build highly scalable and responsive applications in Python and .NET using AWS Lambda.

For Python functions, startup latency from initialization code can be several seconds long. Some scenarios where this can occur are – loading dependencies (such as LangChain, Numpy, Pandas, and DuckDB) or using frameworks (such as Flask or Django). Many functions also perform machine learning (ML) inference using Lambda, and need to load ML models during initialization – a process that can take tens of seconds depending on the size of the model used. Using Lambda SnapStart can reduce startup latency from several seconds to as low as sub-second for these scenarios.

For .NET functions, we expect most use cases to benefit because .NET just-in-time (JIT) compilation takes up to several seconds. Latency variability associated with initialization of Lambda functions has been a long-standing barrier for customers to use .NET for AWS Lambda. SnapStart enables functions to resume quickly by caching a snapshot of their memory and disk state. Therefore, most .NET functions will experience significant improvement in latency variability with Lambda SnapStart.

Getting started with Lambda SnapStart for Python and .NET
To get started, you can use the AWS Management Console, AWS Command Line Interface (AWS CLI) or AWS SDKs to activate, update, and delete SnapStart for Python and .NET functions.

On the AWS Lambda console, go to the Functions page and choose your function to use Lambda SnapStart. Select Configuration, choose General configuration, and then choose Edit. You can see SnapStart settings on the Edit basic settings page.

You can activate Lambda functions using Python 3.12 and higher, and .NET 8 and higher managed runtimes. Choose Published versions and then choose Save.

When you publish a new version of your function, Lambda initializes your code, creates a snapshot of the initialized execution environment, and then caches the snapshot for low-latency access. You can invoke the function to confirm activation of SnapStart.

Here is an AWS CLI command to update the function configuration by running the update-function-configuration command with the --snap-start option.

aws lambda update-function-configuration \
  --function-name lambda-python-snapstart-test \
  --snap-start ApplyOn=PublishedVersions

Publish a function version with the publish-version command.

aws lambda publish-version \
  --function-name lambda-python-snapstart-test

Confirm that SnapStart is activated for the function version by running the get-function-configuration command and specifying the version number.

aws lambda get-function-configuration \
  --function-name lambda-python-snapstart-test:1

If the response shows that OptimizationStatus is On and State is Active, then SnapStart is activated, and a snapshot is available for the specified function version.

"SnapStart": { 
    "ApplyOn": "PublishedVersions",
    "OptimizationStatus": "On"
 },
 "State": "Active",

To learn more about activating, updating, and deleting a snapshot with AWS SDKs, AWS CloudFormation, AWS Serverless Application Model (AWS SAM), and AWS Cloud Development Kit (AWS CDK), visit Activating and managing Lambda SnapStart in the AWS Lambda Developer Guide.

Runtime hooks
You can use runtime hooks to run code executed before Lambda creates a snapshot or after Lambda resumes a function from a snapshot. Runtime hooks are useful to perform cleanup or resource release operations, dynamically update configuration or other metadata, integrate with external services or systems, such as sending notifications or updating external state or to fine-tune your function’s startup sequence, such as by preloading dependencies.

Python runtime hooks are available as part of the open source Snapshot Restore for Python library, which is included in Python managed runtime. This library provides two decorators @register_before_snapshot to run before Lambda creates a snapshot and @register_after_restore to run when Lambda resumes a function from a snapshot. To learn more, visit Lambda SnapStart runtime hooks for Python in the AWS Lambda Developer Guide.

Here is an example Python handler to show how to run code before checkpointing and after restoring:

from snapshot_restore_py import register_before_snapshot, register_after_restore

def lambda_handler(event, context):
    # handler code

@register_before_snapshot
def before_checkpoint():
    # Logic to be executed before taking snapshots

@register_after_restore
def after_restore():
    # Logic to be executed after restore

You can also use .NET runtime hooks available as part of the Amazon.Lambda.Core package (version 2.5 or later) from NuGet. This library provides two methods RegisterBeforeSnapshot() to run before snapshot creation and RegisterAfterRestore() to run after resuming a function from a snapshot. To learn more, visit Lambda SnapStart runtime hooks for .NET in the AWS Lambda Developer Guide.

Here is an example C# handler to show how to run code before checkpointing and after restoring:

public class SampleClass
{
    public SampleClass()
    { 
        Amazon.Lambda.Core.SnapshotRestore.RegisterBeforeSnapshot(BeforeCheckpoint); 
        Amazon.Lambda.Core.SnapshotRestore.RegisterAfterRestore(AfterRestore);
    }
    
    private ValueTask BeforeCheckpoint()
    {
        // Add logic to be executed before taking the snapshot
        return ValueTask.CompletedTask;
    }

    private ValueTask AfterRestore()
    {
        // Add logic to be executed after restoring the snapshot
        return ValueTask.CompletedTask;
    }

    public APIGatewayProxyResponse FunctionHandler(APIGatewayProxyRequest request, ILambdaContext context)
    {
        // INSERT business logic
        return new APIGatewayProxyResponse
        {
            StatusCode = 200
        };
    }
}

To learn how to implement runtime hooks for your preferred runtime, visit Implement code before or after Lambda function snapshots in the AWS Lambda Developer Guide.

Things to know
Here are some things that you should know about Lambda SnapStart:

  • Handling uniqueness – If your initialization code generates unique content that is included in the snapshot, then the content will not be unique when it’s reused across execution environments. To maintain uniqueness when using SnapStart, you must generate unique content after initialization, such as if your code uses custom random number generation that doesn’t rely on built-in-libraries or caches any information such as DNS entries that might expire during initialization. To learn how to restore uniqueness, visit Handling uniqueness with Lambda SnapStart in the AWS Lambda Developer Guide.
  • Performance tuning – To maximize the performance, we recommend that you preload dependencies and initialize resources that contribute to startup latency in your initialization code instead of in the function handler. This moves the latency associated with heavy class loading out of the invocation path, optimizing startup performance with SnapStart.
  • Networking best practices –The state of connections that your function establishes during the initialization phase isn’t guaranteed when Lambda resumes your function from a snapshot. In most cases, network connections that an AWS SDK establishes automatically resume. For other connections, review the Maximize Lambda SnapStart performance in the AWS Lambda Developer Guide.
  • Monitoring functions – You can monitor your SnapStart functions using Amazon CloudWatch log stream, AWS X-Ray active tracing, and accessing real-time telemetry data for extensions using the Telemetry API, Amazon API Gateway and function URL metrics. To learn more about differences for SnapStart functions, visit Monitoring for Lambda SnapStart in the AWS Lambda Developer Guide.

Now available
AWS Lambda SnapStart for Python and .NET functions are available today in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), Europe (Ireland), and Europe (Stockholm) AWS Regions.

With the Python and .NET managed runtimes, there are two types of SnapStart charges: the cost of caching a snapshot per function version that you publish with SnapStart enabled, and the cost of restoration each time a function instance is restored from a snapshot. So, delete unused function versions to reduce your SnapStart cache costs. To learn more, visit the AWS Lambda pricing page.

Give Lambda SnapStart for Python and .NET a try in the AWS Lambda console. To learn more, visit Lambda SnapStart page and send feedback through AWS re:Post for AWS Lambda or your usual AWS Support contacts.

— Channy

Build and modify apps using natural language with AWS App Studio, now generally available

Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/build-and-modify-apps-using-natural-language-with-aws-app-studio-now-generally-available/

Announced as preview in July, AWS App Studio is a generative AI-powered application development service that enables users to create applications using natural language, without the need for professional software development skills. In that post, I covered how AWS App Studio helps you build secure, scalable applications and eliminates operational overhead by fully managing each application.

App Studio empowers a new set of builders to create business applications. Whether you are an IT Project Manager, Data Engineer, Enterprise Architect, or Solution Architect, simply describe your requirements in natural language, within minutes, App Studio generates fully functional applications complete with multipage UIs, data models, and custom business logic.

Today, we’re excited to announce that AWS App Studio is now generally available in the US West (Oregon) and Europe (Ireland) AWS Regions.

Building on feedback from the preview, we are introducing several new features to enhance your app building experience:

Modify your applications with natural language
During the preview period, customers shared with us that they enjoy and appreciate generating fully functional applications using natural language prompts. However, the development journey usually doesn’t stop there, and they asked if they could extend or modify their apps using natural language.

Now, with App Studio, you can modify your applications using natural language. After you’ve generated your applications, you can now describe your desired changes and the assistant will propose updates for you to review. Upon confirmation, it will automatically make the change. This feature makes it even faster and easier to customize your application.

Let’s see how it works in my IT inventory management application that I built with App Studio.

With this new feature, I can chat with the assistant to modify my applications.

To modify my application, I can provide a prompt to add another feature to my app. In this case, I need to add another text input for the web URL to get details of requested hardware, and I need to another text area to store notes.

The generative AI assistant will then process my input and provide a proposal. I can review this proposal and select Confirm to proceed.

Then, the assistant will automatically add the components and modify my application.

Add intelligence to your app with a new generative AI component
We’re also introducing a new component to make it even easier to add generative AI capabilities such as text summarization, content generation, and file analysis to your applications.

There are two ways to use this feature. First, with my canvas open, I can select the Gen AI component and drag and drop it onto the canvas. Then, while selecting the component, I can use the assistant to customize it.

Another way is to use the assistant directly. Let’s say I need a feature to analyze repair notes and provide a summary to make it easier for me to review. I can type what I need in the chat box or use the suggested prompts.

Then, the assistant will process my input and provide a proposal. I can review the proposal and select Confirm to proceed. 

App Studio will automatically add the required components. On the canvas, I see there’s a button that triggers an automation. If I need to change the underlying prompt, I can select the link that will redirect me to the respective automation. 

Under the hood, the Gen AI component is powered by a new action step called Gen AI Prompt. This new component provides an easy way to modify the prompt and input parameters to customize the output generated by the large language model (LLM).

Here’s my published app with the newly added generative AI feature to summarize repair notes.

Generate and add custom business logic with natural language
I can also use the assistant to help me add custom business logic with JavaScript in my automation.

Let’s say that I need a custom business logic to calculate repair duration and notify my stakeholders through email. Here’s the multi-step automation that I created. To add the custom logic to my automation, I choose the JavaScript component and then drag and drop it into the right spot.

Next, I need to select the action and, in the Properties panel, I select the Expand editor icon.

With this feature, I can now generate JavaScript code with natural language. Here, I provide a prompt and App Studio generates the source code for me along with comments. This generated source code provides a foundation that I can customize to suit my requirements. 

Next, I need to add the Send Email action into my automation to complete the flow.

Customize your app’s theme and style
Now, you can customize the look and feel of your application with App themes. With this feature, you can change the appearance of your application to Light mode or Dark mode. Additionally, you can specify custom colors for your app to match your company’s brand. To enable this feature, you need to turn on the Customize toggle.

Available today
Start building secure, intelligent, and scalable business applications with App Studio today. It’s free to build, and you’ll receive a 60-day (250 user hour) free trial.

Learn more about all these features and others in the AWS App Studio documentation and join the conversation in the #aws-app-studio channel in the AWS Developers Slack workspace.

Happy building,

— Donnie

AWS Lambda turns ten – looking back and looking ahead

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-lambda-turns-ten-the-first-decade-of-serverless-innovation/

I have a very vague memory of a 2013-era meeting with my then-colleague Tim Wagner. The term serverless did not exist, but we chatted about various ways to allow developers to focus on code instead of on infrastructure. At some I recall throwing my arms skyward and indicating that it would be cool to simply toss the code into the air and have the cloud grab, store, and run it. After many more such meetings, Tim wrote a PRFAQ proposing that we build a platform that did just that, and in 2014 I was able to announce AWS Lambda – Run Code in the Cloud.

From Startup to Enterprise
It is often the case that startups, with no installed base to worry about and the need to innovate, are the first to take a chance on something new such as Lambda. While that certainly did happen, I was pleasantly surprised to find that established companies—up to and including enterprises—were just as quick to jump in. After a bit of experimentation, they quickly found ways to build event-driven applications that supported critical internal use cases. I took this as an early indicator that Lambda would be a success. It was easy to see how quickly our customers felt a new sense of empowerment: they could move from idea to implementation, and from there to realizing business value, more quickly than ever, while still building their systems in a scalable and composable way.

Today, over 1.5 million Lambda users collectively make tens of trillion function invocations per month. These customers use Lambda for file processing, stream processing (in conjunction with Amazon Kinesis and/or Amazon MSK), web applications, IoT backends, mobile backends (often using Amazon API Gateway and AWS Amplify as well) and to support and power many other use cases.

The First Decade of Serverless Innovation
Let’s roll back the calendar and take a look at a few of the more significant Lambda launches of the past decade:

2014 – The preview launch of AWS Lambda ahead of AWS re:Invent 2014 with support for Node.js and the ability to respond to event triggers from S3 buckets, DynamoDB tables, and Kinesis streams.

2015 – General availability, use of Amazon Simple Notification Service (Amazon SNS) notifications as triggers, and support for Lambda functions written in Java.

2016 – Support for DynamoDB Streams, Support for Python, and an increase in the function duration to 5 minutes (this was later raised to 15 minutes). Access to resources in a VPC, the power to call Lambda functions from Amazon Aurora stored procedures, environment variables, and the Serverless Application Model. This year also saw the introduction of Step Functions, which gave you the power to compose multiple Lambda functions to build more complex applications.

2017 – Support for AWS X-Ray, launches of AWS SAM Local and the Serverless Application Repository.

2018 – Support for Amazon SQS as an event trigger, the power to Extend AWS CloudFormation with Lambda-powered macros, and the ability to write your Lambda functions in any programming language.

2019 – Support for provisioned concurrency to give you additional control over performance.

2020 – Access to Savings Plans to save up to 17%, the ability for Lambda functions to access a shared file system, support for AWS PrivateLink to access your functions over a private network, code signing, billing at 1 ms granularity, functions that can use up to 10 MB of memory and 6 vCPUs, and support for container images.

2021 – Amazon S3 Object Lambda to let you process data as it is being retrieved from S3, AWS Lambda Extensions, support for running Lambda functions on Graviton processors.

2022 – Support for up to 10 GB of ephemeral storage per function invocation, HTTPS endpoints for Lambda functions, and Lambda SnapStart to make function invocation faster and more predictable.

2023 – Amazon S3 Object Lambda support for CloudFront, response streaming, and 12x faster function scaling when handling an unpredictable volume of requests.

2024 -New controls to make it easier to capture and search your Lambda function logs, SnapStart support for Java functions that use the ARM64 architecture, recursive loop detection, a new console editor based on VS Code, and an enhanced local IDE experience. The last two launches were designed to improve the developer experience by making it easier to build, test, debug, and deploy Lambda functions.

Again, this is just a subset of what we have launched. If you want to find even more launches, check out the Lambda category tag and search the What’s New for Lambda.

The Next Decade of Serverless
From the start, the vision for severless has been about helping developers to move from idea to business value more quickly. With that in mind, here are a couple of trends that seem clear to me as I look at Lambda’s direction over the first decade:

Default Choice – The serverless model is definitely here to stay, and will likely become the default operating model over time.

Continued Shift Toward Composability – Over time I can see that serverless applications will continue to make increasing use of reusable, prebuilt components. Aided by AI-powered development tools, a lot of new code will focus on connecting exiting components together in new and powerful ways. This will also boost consistency and reliability across applications.

Automated, AI-Optimized Infrastructure Management– We have already seen that Lambda reduces the amount of time and effort needed for managing infrastructure. Going forward, I can see that Machine Learning and other forms of AI will help to optimize costs and performance by allocating resources optimally with minimal human intervention. Applications will run on infrastructure that is automated, self-healing, and fault tolerant.

Extensibility and Integration – As a consequence of the two previous items, applications should be able to grow and adapt to changing conditions more easily than ever.

Security – Automated infrastructure management, real-time monitoring and other forms of threat detection, and AI-assisted remediation will work together to make serverless applications even more secure.

Some Lambda Resources
If you are already using Lambda to build serverless apps, great! If you are ready to get started, here are some resources to help out:

Serverless Training – Enroll in the free Serverless Learning Plan to learn about serverless concepts, common patterns, and best practices. Read the Serverless Ramp-Up Guide, and look at our extensive (in both topic and language) selection of digital training courses and in-person classroom training:

Case Studies – Review the AWS Serverless Customer Success stories to learn how AWS customers are building and innovating with Lambda and other serverless technologies.

re:Invent 2024 Sessions -Browse the re:Invent 2024 Session Catalog to find nearly 200 sessions focused on Serverless Compute & Containers:

Podcast – Listen to Episode 137 (AWS Lambda: A Decade of Transformation) of the AWS Developers Podcast to hear Marc Brooker and Julian Wood discuss the origins, evolution, and impact of Lambda.

New Books – Take a peek at some of the newest books on serverless development and architecture:

I hope that you have enjoyed this not-so-brief look at the past, present, and future of AWS Lambda. Leave me a comment and let me know what you think!

— Jeff;

Replicate changes from databases to Apache Iceberg tables using Amazon Data Firehose (in preview)

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/replicate-changes-from-databases-to-apache-iceberg-tables-using-amazon-data-firehose/

Today, we’re announcing the availability, in preview, of a new capability in Amazon Data Firehose that captures changes made in databases such as PostgreSQL and MySQL and replicates the updates to Apache Iceberg tables on Amazon Simple Storage Service (Amazon S3).

Apache Iceberg is a high-performance open-source table format for performing big data analytics. Apache Iceberg brings the reliability and simplicity of SQL tables to S3 data lakes and makes it possible for open source analytics engines such as Apache Spark, Apache Flink, Trino, Apache Hive, and Apache Impala to concurrently work with the same data.

This new capability provides a simple, end-to-end solution to stream database updates without impacting transaction performance of database applications. You can set up a Data Firehose stream in minutes to deliver change data capture (CDC) updates from your database. Now, you can easily replicate data from different databases into Iceberg tables on Amazon S3 and use up-to-date data for large-scale analytics and machine learning (ML) applications.

Typical Amazon Web Services (AWS) enterprise customers use hundreds of databases for transactional applications. To perform large scale analytics and ML on the latest data, they want to capture changes made in databases, such as when records in a table are inserted, modified, or deleted, and deliver the updates to their data warehouse or Amazon S3 data lake in open source table formats such as Apache Iceberg.

To do so, many customers develop extract, transform, and load (ETL) jobs to periodically read from databases. However, ETL readers impact database transaction performance, and batch jobs can add several hours of delay before data is available for analytics. To mitigate impact on database transaction performance, customers want the ability to stream changes made in the database. This stream is referred to as a change data capture (CDC) stream.

I met multiple customers that use open source distributed systems, such as Debezium, with connectors to popular databases, an Apache Kafka Connect cluster, and Kafka Connect Sink to read the events and deliver them to the destination. The initial configuration and test of such systems involves installing and configuring multiple open source components. It might take days or weeks. After setup, engineers have to monitor and manage clusters, and validate and apply open source updates, which adds to the operational overhead.

With this new data streaming capability, Amazon Data Firehose adds the ability to acquire and continually replicate CDC streams from databases to Apache Iceberg tables on Amazon S3. You set up a Data Firehose stream by specifying the source and destination. Data Firehose captures and continually replicates an initial data snapshot and then all subsequent changes made to the selected database tables as a data stream. To acquire CDC streams, Data Firehose uses the database replication log, which reduces impact on database transaction performance. When the volume of database updates increases or decreases, Data Firehose automatically partitions the data, and persists records until they’re delivered to the destination. You don’t have to provision capacity or manage and fine-tune clusters. In addition to the data itself, Data Firehose can automatically create Apache Iceberg tables using the same schema as the database tables as part of the initial Data Firehose stream creation and automatically evolve the target schema, such as new column addition, based on source schema changes.

Since Data Firehose is a fully managed service, you don’t have to rely on open source components, apply software updates, or incur operational overhead.

The continual replication of database changes to Apache Iceberg tables in Amazon S3 using Amazon Data Firehose provides you with a simple, scalable, end-to-end managed solution to deliver CDC streams into your data lake or data warehouse, where you can run large-scale analysis and ML applications.

Let’ see how to configure a new pipeline
To show you how to create a new CDC pipeline, I setup a Data Firehose stream using the AWS Management Console. As usual, I also have the choice to use the AWS Command Line Interface (AWS CLI), AWS SDKs, AWS CloudFormation, or Terraform.

For this demo, I choose a MySQL database on Amazon Relational Database Service (Amazon RDS) as source. Data Firehose also works with self-managed databases on Amazon Elastic Compute Cloud (Amazon EC2). To establish connectivity between my virtual private cloud (VPC)—where the database is deployed—and the RDS API without exposing the traffic to the internet, I create an AWS PrivateLink VPC service endpoint. You can learn how to create a VPC service endpoint for RDS API by following instructions in the Amazon RDS documentation.

I also have an S3 bucket to host the Iceberg table, and I have an AWS Identity and Access Management (IAM) role setup with correct permissions. You can refer to the list of prerequisites in the Data Firehose documentation.

To get started, I open the console and navigate to the Amazon Data Firehose section. I can see the stream already created. To create a new one, I select Create Firehose stream.

Create Firehose Stream

I select a Source and Destination. In this example: a MySQL database and Apache Iceberg Tables. I also enter a Firehose stream name for my stream.

Create Firehose Stream - screen 1

I enter the fully qualified DNS name of my Database endpoint and the Database VPC endpoint service name. I verify that Enable SSL is checked and, under Secret name, I select the name of the secret in AWS Secrets Manager where the database username and password are securely stored.

Create Firehose Stream - screen 2

Next, I configure Data Firehose to capture specific data by specifying databases, tables, and columns using explicit names or regular expressions.

I must create a watermark table. A watermark, in this context, is a marker used by Data Firehose to track the progress of incremental snapshots of database tables. It helps Data Firehose identify which parts of the table have already been captured and which parts still need to be processed. I can create the watermark table manually or let Data Firehose automatically create it for me. In that case, the database credentials passed to Data Firehose must have permissions to create a table in the source database.

Create Firehose Stream - screen 3

Next, I configure the S3 bucket Region and name to use. Data Firehose can automatically create the Iceberg tables when they don’t exist yet. Similarly, it can update the Iceberg table schema when detecting a change in your database schema.

Create Firehose Stream - screen 4

As a final step, it’s important to enable Amazon CloudWatch error logging to get feedback about the stream progress and the eventual errors. You can configure a short retention period on the CloudWatch log group to reduce the cost of log storage.

After having reviewed my configuration, I select Create Firehose stream.

Create Firehose Stream - screen 5

Once the stream is created, it will start to replicate the data. I can monitor the stream’s status and check for eventual errors.

Create Firehose Stream - screen 6

Now, it’s time to test the stream.

I open a connection to the database and insert a new line in a table.

Firehose - MySQL

Then, I navigate to the S3 bucket configured as the destination and I observe that a file has been created to store the data from the table.

View parquet files on S3 bucket

I download the file and inspect its content with the parq command (you can install that command with pip install parquet-cli)

Parquet file content

Of course, downloading and inspecting Parquet files is something I do only for demos. In real life, you’re going to use AWS Glue and Amazon Athena to manage your data catalog and to run SQL queries on your data.

Things to know
Here are a few additional things to know.

This new capability supports self-managed PostgreSQL and MySQL databases on Amazon EC2 and the following databases on Amazon RDS:

The team will continue to add support for additional databases during the preview period and after general availability. They told me they are already working on supporting SQL Server, Oracle, and MongoDB databases.

Data Firehose uses AWS PrivateLink to connect to databases in your Amazon Virtual Private Cloud (Amazon VPC).

When setting up an Amazon Data Firehose delivery stream, you can either specify specific tables and columns or use wildcards to specify a class of tables and columns. When you use wildcards, if new tables and columns are added to the database after the Data Firehose stream is created and if they match the wildcard, Data Firehose will automatically create those tables and columns in the destination.

Pricing and availability
The new data streaming capability is available today in all AWS Regions except China Regions, AWS GovCloud (US) Regions, and Asia Pacific (Malaysia) Regions. We want you to evaluate this new capability and provide us with feedback. There are no charges for your usage at the beginning of the preview. At some point in the future, it will be priced based on your actual usage, for example, based on the quantity of bytes read and delivered. There are no commitments or upfront investments. Make sure to read the pricing page to get the details.

Now, go configure your first continual database replication to Apache Iceberg tables on Amazon S3 and visit http://aws.amazon.com/firehose.

— seb

Centrally managing root access for customers using AWS Organizations

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/centrally-managing-root-access-for-customers-using-aws-organizations/

AWS Identity and Access Management (IAM) is launching a new capability allowing security teams to centrally manage root access for member accounts in AWS Organizations. You can now easily manage root credentials and perform highly privileged actions.

Managing root user credentials at scale
For a long time, Amazon Web Services (AWS) accounts were provisioned with highly privileged root user credentials, which had unrestricted access to the account. This root access, while powerful, also posed significant security risks. Each AWS account’s root user had to be secured by adding layers of protection like multi-factor authentication (MFA). Security teams were required to manage and secure these root credentials manually. The process involved rotating credentials periodically, storing them securely, and making sure that the credentials complied with security policies.

As our customers expanded their AWS environments, this manual approach became cumbersome and prone to error. For example, large enterprises operating hundreds or thousands of member accounts struggled to secure root access consistently across all accounts. The manual intervention not only added operational overhead but also created a lag in account provisioning, preventing full automation and increasing security risks. Root access, if not properly secured, could lead to account takeovers and unauthorized access to sensitive resources.

Furthermore, whenever specific root actions such as unlocking an Amazon Simple Storage Service (Amazon S3) bucket policy or an Amazon Simple Queue Service (Amazon SQS) resource policy were required, security teams had to retrieve and use root credentials, which only increased the attack surface. Even with rigorous monitoring and strong security policies, maintaining long-term root credentials opened doors to potential mismanagement, compliance risks, and manual errors.

Security teams began seeking a more automated, scalable solution. They needed a way to not only centralize the management of root credentials but also programmatically manage root access without needing long-term credentials in the first place.

Centrally manage root access
With the new ability to centrally manage root access, we address the longstanding challenge of managing root credentials across multiple accounts. This new capability introduces two essential capabilities: the central management of root credentials and root sessions. Together, they offer security teams a secure, scalable, and compliant way to manage root access across AWS Organizations member accounts.

Let’s first discuss the central management of root credentials. With this capability, you can now centrally manage and secure privileged root credentials across all accounts in AWS Organizations. Root credentials management allows you to:

  • Remove long-term root credentials – Security teams can now programmatically remove root user credentials from member accounts, confirming that no long-term privileged credentials are left vulnerable to misuse.
  • Prevent credential recovery – It not only removes the credentials but also prevents their recovery, safeguarding against any unintended or unauthorized root access in the future.
  • Provision secure-by-default accounts – Because you can now create member accounts without root credentials from the start, you no longer need to apply additional security measures like MFA after account provisioning. Accounts are secure by default, which drastically reduces security risks associated with long-term root access and helps simplify the entire provisioning process.
  • Help to stay compliant – Root credentials management allows security teams to demonstrate compliance by centrally discovering and monitoring the status of root credentials across all member accounts. This automated visibility confirms that no long-term root credentials exist, making it easier to meet security policies and regulatory requirements.

But how can we make sure it remains possible to perform selected root actions on the accounts? This is the second capability we launch today: root sessions. It offers a secure alternative to maintaining long-term root access. Instead of manually accessing root credentials whenever privileged actions are required, security teams can now gain short-term, task-scoped root access to member accounts. This capability makes sure that actions such as unlocking S3 bucket policies or SQS queue policies can be performed securely without the need for long-term root credentials.

Root sessions key benefits include:

  • Task-scoped root access – AWS enables short-term root access for specific actions, adhering to the best practices of least privilege. This limits the scope of what can be done and minimizes the duration of access, reducing potential risks.
  • Centralized management – You can now perform privileged root actions from a central account without needing to log in to each member account individually. This streamlines the process and reduces the operational burden on security teams, allowing them to focus on higher-level tasks.
  • Alignment with AWS best practices – By using short-term credentials, organizations align themselves with AWS security best practices, which emphasize the principle of least privilege and the use of short-term, temporary access where possible.

This new capability does not grant full root access. It provides temporary credentials for performing one of these five specific actions. The first three actions are possible with central management of root accounts. The last two come when enabling root sessions.

  • Auditing root user credentials – Read-only access to review root user information
  • Re-enabling account recovery – Reactivating account recovery without root credentials
  • Deleting root user credentials – Removing console passwords, access keys, signing certificates, and MFA devices
  • Unlocking an S3 bucket policy – Editing or deleting an S3 bucket policy that denies all principals
  • Unlocking an SQS queue policy – Editing or deleting an Amazon SQS resource policy that denies all principals

How to obtain root credentials on a member account
In this demo, I show you how to prepare your management account, create a member account without root credentials, and obtain temporary root credentials to make one of the five authorized API call on the member account. I assume you have an organization already created.

First, I create a member account.

aws organizations create-account    \
     --email [email protected] \
     --account-name 'Root Accounts Demo account'
{
    "CreateAccountStatus": {
        "Id": "car-695abd4ee1ca4b85a34e5dcdcd1b944f",
        "AccountName": "Root Accounts Demo account",
        "State": "IN_PROGRESS",
        "RequestedTimestamp": "2024-09-04T20:04:09.960000+00:00"
    }
}

Then, I enable the two new capabilities on my management account. Don’t worry, these commands don’t alter the behavior of the accounts in any way other than enabling use of the new capability.

➜  aws organizations enable-aws-service-access \
        --service-principal iam.amazonaws.com

➜  aws iam enable-organizations-root-credentials-management
{
    "OrganizationId": "o-rlrup7z3ao",
    "EnabledFeatures": [
        "RootCredentialsManagement"
    ]
}

➜  aws iam enable-organizations-root-sessions
{
    "OrganizationId": "o-rlrup7z3ao",
    "EnabledFeatures": [
        "RootSessions",
        "RootCredentialsManagement"
    ]
}

Alternatively, I can also use the console on the management account. Under Access management, I select Account settings.

Root Access Management

Now, I’m ready to make requests to obtain temporary root credentials. I have to pass one of the five managed IAM policies to scope down the credentials to a specific action.

➜  aws sts assume-root \
       --target-principal <my member account id> \
       --task-policy-arn arn=arn:aws:iam::aws:policy/root-task/S3UnlockBucketPolicy 

{
    "Credentials": {
        "AccessKeyId": "AS....XIG",
        "SecretAccessKey": "ao...QxG",
        "SessionToken": "IQ...SS",
        "Expiration": "2024-09-23T17:44:50+00:00"
    }
}

Once I obtain the access key ID, the secret access key, and the session token, I use them as usual with the AWS Command Line Interface (AWS CLI) or an AWS SDKs.

For example, I can pass these three values as environment variables.

$ export AWS_ACCESS_KEY_ID=ASIA356SJWJITG32xxx
$ export AWS_SECRET_ACCESS_KEY=JFZzOAWWLocoq2of5Exxx
$ export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEMb//////////wEaCXVxxxx

Now that I received the temporary credentials, I can make a restricted API call as root on the member account. First, I verify I now have root credentials. The Arn field confirms I’m working with the root account.


# Call get Caller Identity and observe I'm root in the member account
$ aws sts get-caller-identity
{
   "UserId": "012345678901",
   "Account": "012345678901",
   "Arn": "arn:aws:iam::012345678901:root"
}

Then, I use the delete-bucket-policy from S3 to remove an incorrect policy that has been applied to a bucket. The invalid policy removed all bucket access for everybody. Removing such policy requires root credentials.

aws s3api delete-bucket-policy --bucket my_bucket_with_incorrect_policy

When there is no output, it means the operation is successful. I can now apply a correct access policy to this bucket.

Credentials are valid only for 15 minutes. I wrote a short shell script to automate the process of getting the credentials as JSON, exporting the correct environment variables, and issuing the command I want to run as root.

Availability
Central management of root access is available at no additional cost in all AWS Regions except AWS GovCloud (US) and AWS China Regions, where there is no root account. Root sessions are available everywhere.

You can start using it through the IAM console, AWS CLI or AWS SDK. For more information, visit AWS account root user in our documentation and follow best practices for securing your AWS accounts.

— seb

Introducing resource control policies (RCPs), a new type of authorization policy in AWS Organizations

Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/introducing-resource-control-policies-rcps-a-new-authorization-policy/

Today, I am happy to introduce resource control policies (RCPs) – a new authorization policy managed in AWS Organizations that can be used to set the maximum available permissions on resources within your entire organization. They are a type of preventative control that help you establish a data perimeter in your AWS environment and restrict external access to resources at scale. Enforced centrally within Organizations, RCPs provide confidence to the central governance and security teams that access to resources within their AWS accounts conforms to their organization’s access control guidelines.

RCPs are available in all commercial AWS Regions and, at launch, the following services are supported: Amazon Simple Storage Service (Amazon S3), AWS Security Token Service (AWS STS), AWS Key Management Service (AWS KMS), Amazon Simple Queue Service (Amazon SQS), and AWS Secrets Manager.

There are no additional charges for enabling and using RCPs.

How are they different from service control policies (SCPs)?
While similar in nature, RCPs complement service control policies (SCPs), but they work independently.

Service control policies (SCPs) allow you to limit the permissions granted to principals within your organization such as AWS Identity and Access Management (IAM) roles. By constraining these permissions centrally within Organizations you can restrict access to AWS services, specific resources and even under what conditions principals can make requests across multiple AWS accounts.

RCPs, on the other hand, allow you to limit the permissions granted to resources in your organization. Because you implement RCPs centrally within Organizations, you can enforce consistent access controls on resources across multiple AWS accounts. For instance, you can restrict access to S3 buckets in your accounts so that they can only be accessed by principals that belong to your organization. RCPs are evaluated when your resources are being accessed irrespective of who is making the API request.

Keep in mind that neither SCPs nor RCPs grant any permissions. They only set the maximum permissions available to principals and resources in your organization. You still need to grant permissions with appropriate IAM policies, such as identity-based or resource-based policies.

Quotas for SCPs and RCPs are completely independent from each other. Each RCP can contain up to 5,120 characters. You can have up to five RCPs attached to the organization root, each OU, and account, and you can create and store up to 1000 RCPs in an organization.

How to get started
To start using RCPs you must first enable them. You can do this using the Organizations console, an AWS SDK, or by using the AWS Command Line Interface (AWS CLI). Make sure you are using the Organizations management account or a delegated administrator because those are the only accounts that can enable or disable policy types.

Make sure that you are using Organizations with the “all features” option. If you are using the “Consolidated billing features” mode, then you need to migrate to using all features before you can enable RCPs.

For console users, first head to the Organizations console. Choose Policies and you should see the option to enable Resource control policies.

enabling RCPs in the AWS Organizations console

After RCPs are enabled, you will notice in the Resource control policies page that a new policy called RCPFullAWSAccess is now available. This is an AWS managed policy that is automatically created and attached to every entity in your organization including the root, each OU, and AWS account.

the RCPFullAWSAccessPolicy can be seen on the console once RCPs are enabled

This policy allows all principals to perform any action against the organization’s resources, which means that until you start creating and attaching your own RCPs, all of your existing IAM permissions continue to operate as they did.

This is how it looks:

{
  "Version": "2012-10-17",
  "Statement": [
    { 
        "Effect": "Allow", 
        "Principal": "*", 
        "Action": "*", 
        "Resource": "*" 
    }
  ]
}

Creating an RCP

Now, we are ready to create our first RCP! Let’s look at an example.

By default, AWS resources do not permit access to external principals; resource owners must explicitly grant such access by configuring their policies. While developers have the flexibility to set resource-based policies according to their application needs, RCPs enable central IT teams to maintain control over the effective permissions across resources in their organization. This ensures that even if developers grant broad access, external identities are still restricted access in accordance with the organization’s security requirements.

Let’s create an RCP to restrict access to our S3 buckets so that only principals within our organization can access them.

On the Resources control policies page, choose Create policy which will take you to the page where you can author a new policy.

create a new resource control policy pageI am going to call this policy EnforceOrgIdentities. I recommend you enter a clear description so it is easy to understand at first glance what this policy does and to tag it appropriately.

The next section is where you can edit your policy statement. I replace the pre-populated policy template with my own:

create policy - policy syntaxHere is the full JSON policy document:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EnforceOrgIdentities",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "*",
            "Condition": {
                "StringNotEqualsIfExists": {
                    "aws:PrincipalOrgID": "[MY ORG ID]"
                },
                "BoolIfExists": {
                    "aws:PrincipalIsAWSService": "false"
                }
            }
        }
    ]
}

Let’s break this down:

Version – This is a standard and required element of IAM policies. AWS maintains backwards compatibility, so using the latest version, currently 2012-10-17, doesn’t break existing policies but allows you to use newer features.

Statement – An array that can contain one or more statement objects. Each of these statement objects defines a single permission or set of permissions.

Sid – This is an optional field that can be helpful for policy management and troubleshooting. It needs to be unique within the scope of this JSON policy document.

Effect – As you might remember from earlier, by default we have an RCP that allows access to every AWS principal, action, and resource attached to every entity in our organization. Therefore, you should use Deny to apply restrictions.

Principal – For an RCP, this field must always be set to "*". Use the Condition field if you want this policy to apply only to specific principals.

Action – Specifies the AWS service and the actions that this policy applies to. In this case, we want to deny all interactions with Amazon S3 if they don’t meet our access control guidelines.

Resource – Specifies the resources that the RCP applies to.

Condition – A collection of conditions that will be evaluated to determine whether the policy should be applied or not for each request.

It’s important to remember that all conditions must evaluate to true for the RCP to be applied. In this case, we are applying two conditions:

1. Was the request made by an external principal?

"StringNotEqualsIfExists": 
 { 
   "aws:PrincipalOrgID": "[MY ORG ID]" 
 }

This condition first checks if the key aws:PrincipalOrgID is present in the request. If it’s not, then this condition evaluates to true without further evaluation.

If it does exist, then it compares the value to our organization ID. If the value is the same then it evaluates to false which means that the RCP will not be applied because all conditions must evaluate to true. This is the intended behaviour because we don’t want to deny access to principals within our organization.

However, if the value doesn’t match our organization ID, that means the request was made by a principal who is external to our organization. The condition evaluates to true which means that the RCP can still potentially be applied as long as the second condition also evaluates to true.

2. Is the request coming from an AWS service?

"BoolIfExists": 
   { 
     "aws:PrincipalIsAWSService": "false"
   }

This condition tests if the request contains a special key called aws:PrincipalIsAWSService which is automatically injected into the request context for all signed API requests and is set to true when it originates from an AWS service such as AWS CloudTrail writing events to your S3 bucket. If the key is not present, then this condition evaluates to true.

If it does exist, then it will compare the value to what we declare in our statement. In this case, we are testing if the value is equal to false. If it is, then we return true since that would mean that the request was not made by an AWS service and could potentially still have been made by someone outside of our organization. Otherwise, we return false.

In other words, if the request did not originate from a principal within our organization and it did not originate from an AWS service, then access to the S3 bucket is denied.

This policy is just a sample and you should tailor it to meet your unique business and security objectives. For instance, you might want to customize this policy to allow access by your business partners or to restrict access to AWS services so that they can access your resources only on your behalf. See Establishing a data perimeter on AWS: Allow only trusted identities to access company data for more details.

Attaching an RCP
The process of attaching an RCP is similar to an SCP. As previously mentioned, you can attach it to the root of your organization, to an OU, or to specific AWS accounts within your organization.

attaching a policy

After the RCP is attached, access requests to affected AWS resources must comply with the RCP restrictions. We recommend that you thoroughly test the impact that the RCP has on resources in your accounts before enforcing it at scale. You can begin by attaching RCPs to individual test accounts or test OUs.

Seeing it in action
I have now created and attached my RCP, so I’m ready to see it in practice! Let’s assume that a developer attached a resource-based policy to an S3 bucket in our organization and they explicitly gave access to identities in an external account:

bucket policy with external account id

RCPs do not prevent users from saving resource-based policies that are more permissive than the RCP allows. However, the RCP will be evaluated as part of the authorization process, as we’ve seen previously, so the request by external identities will be denied regardless.

We can prove this by trying to access the bucket with this external account, this time from the AWS CLI:


$ aws s3api get-object —bucket 123124ffeiufskdjfgbwer \
  --key sensitivefile.txt \
  --region us-east-1 local_file

An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

Scaling the deployment of RCPs in your environment
So far, we have seen how we can manage RCPs using the console. However, for large-scale control management you should look into configuring them as infrastructure as code and make sure they are integrated into your existing continuous integration and continuous delivery (CI/CD) pipelines and processes.

If you use AWS Control Tower, you can deploy RCP-based controls in addition to SCP-based controls. For instance, you can use AWS Control Tower to deploy an RCP similar to that we created in the preceding example which prevents external principals from accessing S3 buckets in our organization. This ensures that RCPs are consistently applied to resources in managed accounts, streamlining and centralizing access control governance at scale.

Additionally, similar to SCPs, AWS Control Tower also supports drift detection for RCPs. If an RCP is modified or removed outside of AWS Control Tower, you will be notified of the drift and provided with steps for remediation.

Conclusion
Resource control policies (RCPs) offer centralized management over the maximum permissions available to AWS resources in your organization. Along with SCPs, RCPs help you to centrally establish a data perimeter across your AWS environment and prevent unintended access at scale. SCPs and RCPs are independent controls that allow you to achieve a distinct set of security objectives. You can choose to enable only SCPs or RCPs, or use both policy types together to establish a comprehensive security baseline as part of the defense-in-depth security model.

To learn more, see Resource control policies (RCPs) in the AWS Organizations User Guide.

Matheus Guimaraes | @codingmatheus

AWS BuilderCards second edition at re:Invent 2024

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-buildercards-second-edition-available-at-reinvent-2024-and-online/

I have been following the progress of AWS BuilderCards for several years. Players of all skill levels use the cards to learn about AWS in a fun and engaging way, competing (in a friendly fashion) to combine their cards to form architectures, gaining knowledge and scoring points as they play:

To date, more than 15,000 sets of BuilderCards have been printed and put to use over the course of three re:Invents, fifteen AWS Summits, and multiple community events. For example, here is a group of AWS enthusiasts having a good time in Tokyo during JAWS Days 2024:

Feedback from players has been incredibly positive, with a 4.8 star customer satisfaction score (CSAT) across more than 1500 replies.

Second Edition Now Available
I am happy to announce that the second edition of AWS BuilderCards will be distributed at re:Invent 2024 and will soon be available for purchase online. In response to feedback on the first edition, we have made many changes to the second edition. Here’s a summary:

Design — The revised design focuses on the contents of the card, with additional gradients and patterns to make the cards more attractive and playful. The font size was increased, game effects are now represented as icons, and the QR codes now point to the BuilderCards site:

Game Mechanics — The mechanics have been revised to improve balance, simplify play, and remove some bias. There are also some starter cards that represent a common on-premises data center environment.

Generative AI Add-On Pack — This new deck includes 50 additional BuilderCards designed to show how Generative AI and AWS architecture align. This add-on pack also includes a set of Mission cards. These cards add context, with QR codes that link to published best practices or solutions. The cards are larger, with additional text and an architecture diagram. Use is optional, with players earning extra points for completing a mission. Each deck also includes five blank BuilderCards and two blank Mission cards to support customization:

Get Your BuilderCards
If you will be at re:Invent 2024, make plans to visit the BuilderCards play area on Level 1 of Caesar’s Forum next to the PeerTalk area:

Sunday – 10 AM to 6 PM
Monday – 9 AM to 5 PM
Tuesday – 9 AM to 5 PM
Wednesday – 9 AM to 5 PM
Thursday – 9 AM to 4 PM

You can play against other re:Invent attendees and also get your own game and add-on packs to take home (we’ll give away 1,000 per day). Thanks to hard work by Masanori Yamaguchi (AWS Hero) and Kosuke Enomoto (AWS Community Builder) we will also have BuilderCards in Japanese for on-site play. You can read Kosuke’s blog post (AWS BuilderCards Japanese Edition for JAWS DAYS) to learn more about the translation process.

If you won’t be able to attend re:Invent, you will soon be able to purchase a deck of your own. Check back here for more info!

Stay Tuned
The BuilderCards team is working on additional goodies including expansion packs and additional language packs.

— Jeff;

AWS Weekly Roundup: 20 years of AWS News Blog, Express brokers for Amazon MSK, Windows Server 2025 images on EC2, and more (Nov 11, 2024)

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-20-years-of-aws-news-blog-express-brokers-for-amazon-msk-windows-server-2025-images-on-ec2-and-more-nov-11-2024/

Happy 20th Anniversary of the AWS News Blog! 🎉🥳🎊 On November 9, 2004, Jeff Barr published his first blog post. At the time, he started a personal blog site using TypePad. He wanted to speak to his readers with his personal voice, not the company or team.

On April 29, 2014, we created a new AWS blog site and migrated all posts to that page. There are currently over 4,300 posts on the AWS News Blog, with Jeff contributing over 3,200 of them.

Since December 2016, the AWS News Blog has added new writers, but we are still following Jeff’s leadership principals for AWS News Bloggers in accordance with Day One. What’s unique about the AWS News Blog is that the blog writers get to use the features of the product team in advance, following the Customer Obsession leadership principle, and focus on walk-throughs of how customers can quickly use them to save time, with the Frugality principle.

I am very grateful for Jeff’s fundamental and pivotal role over the past 20 years, and I look forward to the next 20 years!

Last week’s launches
Here are some launches that got my attention:

New Express brokers for Amazon MSK – Express brokers are a new broker type for Amazon MSK Provisioned designed to deliver up to three times more throughput per broker, scale up to 20 times faster, and reduce recovery time by 90 percent as compared to standard Apache Kafka brokers. Express brokers come preconfigured with Kafka best practices by default, support all Kafka APIs, and provide the same low-latency performance, so you can continue using existing client applications without any changes.

New Amazon Kinesis Client Library 3.0 – You can now reduce compute costs to process streaming data by up to 33 percent with Kinesis Client Library (KCL) 3.0, compared to previous KCL versions. KCL 3.0 introduces an enhanced load balancing algorithm that continuously monitors resource utilization of the stream processing workers and automatically redistributes the load from overutilized workers to other underutilized workers. To learn more, read the AWS Big Data Blog post.

Microsoft Windows Server 2025 images on Amazon EC2 – We now support Microsoft Windows Server 2025 with License Included (LI) Amazon Machine Images (AMIs), providing customers with an easy and flexible way to launch the latest version of Windows Server. By running Windows Server 2025 on Amazon EC2, customers can take advantage of the security, performance, and reliability of AWS with the latest Windows Server features. To learn more about running Windows Server 2025 on Amazon EC2, visit Windows Workloads on AWS.

Anthropic’s Claude 3.5 Haiku model in Amazon Bedrock – Claude 3.5 Haiku is the next generation of Anthropic’s fastest model, combining rapid response times with improved reasoning capabilities, making it ideal for tasks that require both speed and intelligence. Claude 3.5 Haiku improves across every skill set and surpasses even Claude 3 Opus, the largest model in Anthropic’s previous generation, on many intelligence benchmarks—including coding. To learn more, read the AWS News Blog post.

Amazon Bedrock Prompt Management GA – You can simplify the creation, testing, versioning, and sharing of prompts in Amazon Bedrock Prompt Management. At general availability, we added new features that provide enhanced options for configuring your prompts and enabling seamless integration for invoking them in your generative AI applications, such as structured prompts and Converse and InvokeModel API integration. To learn more, read the AWS Machine Learning blog post.

Six new synthetic generative voices for Amazon Polly – The generative engine is Amazon Polly’s most advanced text-to-speech (TTS) model leveraging the generative AI technology. We added six new synthetic female-sounding generative voices: Ayanda (South African English), Léa (French), Lucia (European Spanish), Lupe (American Spanish), Mía (Mexican Spanish), and Vicki (German). This extends thirteen voices and nine locales to provide you with more options of highly expressive and engaging voices.

Amazon OpenSearch Service Extended Support – We announce the end of Standard Support and Extended Support timelines for legacy Elasticsearch versions and OpenSearch Versions. Standard Support ends on Nov 7, 2025, for legacy Elasticsearch versions up to 6.7, Elasticsearch versions 7.1 through 7.8, OpenSearch versions from 1.0 through 1.2, and OpenSearch versions 2.3 through 2.9. With Extended Support, for an incremental flat fee over regular instance pricing, you continue to get critical security updates beyond the end of Standard Support. To learn more, read the AWS Big Data Blog post.

For a full list of AWS announcements, be sure to keep an eye on the What’s New at AWS page.

Other AWS news
Here are some additional news items that you might find interesting:

CEO’s visiting at AWS data center – Matt Garman, CEO of AWS, had a great time visiting one of our AWS data centers recently, and was able to get a look at the continuous innovation delivered by the team. Of course, it’s no surprise that Amazon’s senior executives visit fulfillment centers, contact centers, and data centers, to do real work for customers. AWS data centers are designed for customers in every aspect, for maximum resilience, performance, and energy efficiency.

AWS supports small businesses, creates jobs, sets up sustainability initiatives, and develops educational programs near AWS data centers. Get the latest updates – AWS in your community: Here’s what’s happening near data centers across the US on About Amazon News.

Amazon Q Business at Amazon – I introduced an Amazon story to use Code transformation in Amazon Q Developer to migrate more than old 30,000 Java applications to Java 17 version. It saved over 4,500 developer years of effort compared to previous manual jobs and saved the company $260 million in annual by moving to the latest Java version.

Here is another dogfooding story of Amazon Q Business at Amazon. Amazon built an internal chatbot with Amazon Q Business and it has resolved over 1 million internal Amazon developer questions, reducing time spent churning on manual technical investigations by more than 450,000 hours.

Our team onboarded Amazon Q Business with millions of internal documents and integrated Q Business into the tools our team use every day. Now, instead of waiting hours for responses to complex technical questions on Q&A boards or Slack channels, developers can get answers in seconds.

TOURCast at PGA TOUR – If you enjoy golf, this news will be of interest to you. The PGA TOUR debuted TOURCast in Japan at the 2024 ZOZO Championship to capture and disseminate better statistical data and bring fans closer to the game based on new scoring system called ShotLink, powered by CDW. This marks the first time the PGA TOUR has been able to bring this technology to Asia, leveraging the flexibility and scalability of AWS to overcome unique challenges.


PGA TOUR volunteer setting up GPS equipment on the fairway at ZOZO championship that will input specific shot data and feed back to Shotlink Select Plus. [IMAGE: PGA TOUR]

They’ve completely rebuilt their scoring system over the past two years on a new cloud stack. With AWS cloud, whether data comes from high-tech radar systems, cameras, or manual input, the system processes it all seamlessly.

Upcoming AWS events
Check your calendars and sign up for these AWS events:

AWS GenAI Lofts – AWS GenAI Lofts are about more than just the tech, they bring together startups, developers, investors, and industry experts. Whether you’re looking to gain deep insights, or get your questions answered by generative AI pros, our GenAI Lofts have you covered, and provide everything you need to start building your next innovation. Join events in São Paulo (through November 20), and Paris (through November 25).

AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Jakarta, Indonesia (November 23), Kochi, India (December 14).

AWS re:Invent – You can still register for the annual learning event, taking place December 2–6 in Las Vegas. Surprisingly Andy Jassy, CEO of Amazon said he will come back and participate in AWS re:Invent this year. He said “As always, the priority is to make this a learning event so customers can take nuggets back and change their own customer experiences and businesses. We’ll also have a bunch of goodies for you that we’ll announce and that we think folks will like.” Let’s meet there!

You can browse all upcoming in-person and virtual events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Channy

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Announcing new APIs for Amazon Location Service Routes, Places, and Maps

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/announcing-new-apis-for-amazon-location-service-routes-places-and-maps/

Today, Amazon Location Service released 17 new and enhanced APIs that expand and improve capabilities for the Routes, Places, and Maps functionalities, creating a more cohesive and streamlined experience for developers. By introducing enhanced features and offering simplified migration, these updates make Amazon Location Service more accessible and useful for a wide range of applications.

You can now access advanced route optimization, toll cost calculations, GPS traces snapping, and a variety of map styles with static and dynamic rendering options, and perform proximity-based search and predictive suggestions, with rich, detailed information on points of interest.

At Amazon, the vast majority of our roadmaps are driven by customer feedback. Many customers building applications with Amazon Location Service have shared that they need purpose-built APIs and more granular details, such as contact information and business hours, when working with location-based data. Although the current API set has provided valuable tools for many customers, developers have expressed a desire for additional capabilities, such as detailed route planning, proximity-based searches, additional places details, and static map images. These new APIs address these requests and provide a more comprehensive, out-of-the-box location solution.

New and enhanced capabilities
Today’s launch introduces 10 updated APIs and seven entirely new APIs, responding directly to your feedback. Each service—Routes, Places, and Maps—receives specific enhancements designed to support a broader range of use cases.

Routes
The Amazon Location Routes API now supports advanced route planning and customization options, allowing users to:

  • CalculateIsolines to identify service areas within specific travel time, or distance
  • OptimizeWaypoints to determine the most efficient sequence of waypoints, helping to minimize either travel time or distance
  • Calculate Toll Costs to provide precise cost estimates for routes involving toll roads
  • SnapToRoads, to enable precise matching of GPS traces by snapping points to the road network

With these capabilities, you can design more accurate and dynamic route experiences for your users. For example, a logistics company could optimize driver routes in real-time, factoring in live traffic and minimizing travel costs for deliveries.

Maps
The updated Amazon Location Maps API includes more purpose-built map styles crafted by skilled cartographers. These map styles offer professional designs that accelerate time to market and eliminate the need for custom map creation. Additionally, the Static Map Image feature allows developers to integrate static maps within applications, reducing the need for continuous data streaming and improving performance in use cases where interactivity isn’t necessary.

Key features of the Maps API include:

  • GetTile, to download a tile from a tileset, with a specified X, Y, and Z axis values
  • GetStyleDescriptor, to return information about the style
  • GetStaticMap, which enables the rendering of non-interactive maps for reporting or visualization purposes

Places
The Amazon Location Places API enhancements allow more detailed search capabilities, addressing requests for increased granularity in location data. The new capabilities include:

  • SearchNearby and Autocomplete, which support proximity-based queries and enable predictive text features for better user experiences
  • Enhanced business details with categories such as Business Hours, Contact Information, and additional attributes for points of interest

These features are especially useful for applications where users need detailed information about nearby locations, such as food delivery services or retail applications. Imagine that a customer opens a food delivery application, searches for nearby restaurants using SearchNearby, and retrieves restaurant details such as business hours and contact information to confirm availability. Once multiple delivery orders are assigned to a driver, the application uses OptimizeWaypoints to suggest the most efficient route for pickups and deliveries. As the driver follows the route, SnaptoRoads provides precise visualization of their location, enhancing the customer’s real-time tracking experience.

Enhanced Location Service in action
Calling the API is straightforward. You can use the AWS Command Line Interface (AWS CLI), one of our AWS SDKs, or the plain REST API. However, displaying the information on a map in a web or mobile app requires some additional setup. Although the process is well documented, it’s too detailed to cover fully here. In this demo, I’ll focus on using the API.

Amazon Location Service allows API calls to be authenticated in two ways: through AWS API authentication (AWS Sigv4 authentication) or through API keys. API keys can be more convenient for developers of mobile applications where the end user is not authenticated or when integrating with Amazon Cognito is not feasible. This is the recommended authentication method for front-end applications.

To demonstrate the versatility of the APIs and how easily you can integrate inside your applications, I use a combination of the AWS CLI, cURL, and a graphical REST API client for each step of the demo.

Step 1: Create an API key

First, I create an API key for my application using the AWS CLI. You can also manage API keys in the AWS Management Console.

REGION=eu-central-1
KEYNAME=geo-key-seb

aws location create-key --region ${REGION} --key-name ${KEYNAME} --restrictions \
AllowActions="geo-routes:*","geo-places:*","geo-maps:*",\
AllowResources="arn:aws:geo-routes:${REGION}::provider/default",\
"arn:aws:geo-places:${REGION}::provider/default",\
"arn:aws:geo-maps:${REGION}::provider/default" \
--no-expiry 

{
    "Key": "v1.public.ey...cy",
    "KeyArn": "arn:aws:geo:eu-central-1:02345678901:api-key/geo-key-seb",
    "KeyName": "geo-key-seb",
    "CreateTime": "2024-09-29T09:35:53.115000+00:00"
}

This command generates the API key, which I can now use to call Amazon Location APIs.

Step 2: Get geographic coordinates

Next, I use cURL to retrieve the geographic coordinates (a longitude and latitude) for the city center of Lille, France, by calling GeoCode and passing an address in the QueryText parameter.

$ curl --silent -X "POST" "https://places.geo.eu-central-1.amazonaws.com/v2/geocode?key=v1.public.ey...cy" \
       -d $'{ "QueryText": "Grand Place, Lille, France" }' 
 
{"ResultItems":[{"PlaceId":"AQ...5U","PlaceType":"Street","Title":"Grand'Place, 59800 Lille, France",
                 "Address":{"Label":"Grand'Place, 59800 Lille, France",
                 "Country":{"Code2":"FR","Code3":"FRA","Name":"France"},
                 "Region":{"Code":"HDF","Name":"Hauts-de-France"},"SubRegion":{"Name":"Nord"},
                 "Locality":"Lille","District":"Centre","PostalCode":"59800",
                 "Street":"Grand'Place","StreetComponents":[{"BaseName":"Grand'Place","Language":"fr"}]},
                 "Position":[3.06361,50.63706],
                 "MapView":[3.0628,50.6367,3.06413,50.63729],
                 "MatchScores":{"Overall":1,"Components":{"Address":{"Country":1,"Locality":1,"Intersection":[1]}}}}]}

This returns several data points, including the GPS coordinates for the city center: [3.06361, 50.63706].

Step 3: Search for nearby places

Using the coordinates retrieved, I use a REST API client tool to call the SearchNearby API to find places of interest around Lille’s city center.

Amazon Location Service - API call for places of interest

On the right side of the screen, I can read the API response: a list of nearby places, such as restaurants, banks, and parking areas. I can further refine the search by specifying categories or restricting the search area.

The SearchNearby API accepts an optional Filter parameter that helps you restrict the search within a bounding box or to include or exclude business chains, categories, countries, or food types.

"Filter": {
   "BoundingBox": [ number ],
   "ExcludeBusinessChains": [ "string" ],
   "ExcludeCategories": [ "string" ],
   "ExcludeFoodTypes": [ "string" ],
   "IncludeBusinessChains": [ "string" ],
   "IncludeCategories": [ "string" ],
   "IncludeCountries": [ "string" ],
   "IncludeFoodTypes": [ "string" ]
},

In my search for nearby points of interest, one of the results returned was a McDonald’s, a well-known international reference 🍔.

Amazon Location Service - SearchNearby result

Step 4: Get driving directions

Finally, I use the AWS CLI to calculate driving directions between two city centers: Brussels, Belgium, and Lille, France.

aws location calculate-routes    \
    --origin 4.35278 50.84687      \
    --destination 3.06361 50.63706 \
    --key "v1.public.ey...cy"

The response includes a polyline for rendering the path on a map and a step-by-step list of driving directions.

...
          "TravelMode": "Car",
          "Type": "Vehicle",
          "VehicleLegDetails": {
            "TravelSteps": [
              {
                "Duration": 15,
                "Distance": 75,
                "ExitNumber": [],
                "GeometryOffset": 0,
                "Type": "Depart"
              },
              {
                "Duration": 10,
                "Distance": 8,
                "ExitNumber": [],
                "GeometryOffset": 2,
                "Type": "Turn",
                "TurnStepDetails": {
                  "Intersection": [],
                  "SteeringDirection": "Right",
                  "TurnIntensity": "Typical"
                }
              },
...

Step 5: Displaying the driving directions on a map

To visualize the route on a map, I use the MapLibre library, which is a rendering engine for displaying maps in web and mobile applications. Following the Amazon Location Service Developer Guide, I built a basic app to display the route.

Amazon Location Service - Map with route

In addition to MapLibre, you can use AWS Amplify to integrate and display Amazon Location data in your applications.

Getting started
With these new and updated APIs, Amazon Location Service offers a more comprehensive suite of mapping and location data for your business needs. These will help to accelerate your development lifecyle by increasing developers’ agility and scalability.

To get started, explore the updated Amazon Location Service Developer Guide and begin integrating these features today. You can also visit the Amazon Location Service page to learn more or try out the APIs with your favorite AWS SDKs to see how they can enhance your applications.

— seb

Introducing Express brokers for Amazon MSK to deliver high throughput and faster scaling for your Kafka clusters

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-express-brokers-for-amazon-msk-to-deliver-high-throughput-and-faster-scaling-for-your-kafka-clusters/

Today, we’re announcing the general availability of Express brokers, a new broker type for Amazon Managed Streaming for Apache Kafka (Amazon MSK). It’s designed to deliver up to three times more throughput per-broker, scale up to 20 times faster, and reduce recovery time by 90 percent as compared to Standard brokers running Apache Kafka. Express brokers come preconfigured with Kafka best practices by default, support Kafka APIs, and provide the same low latency performance that Amazon MSK customers expect, so they can continue using existing client applications without any changes.

Express brokers provide improved compute and storage elasticity for Kafka applications when using Amazon MSK provisioned clusters. Amazon MSK is a fully-managed AWS service that makes it easier for you to build and run highly available and scalable applications based on Apache Kafka.

Let’s dive deeper into some of the key features that Express brokers have and the benefits they provide:

  • Easier operations with hands-free storage management – Express brokers offer unlimited storage without preprovisioning, eliminating disk-related bottlenecks. Cluster sizing is simpler, requiring only ingress and egress throughput divided by recommended per-broker throughput. This removes the need for proactive disk capacity monitoring and scaling, simplifying cluster management and improving resilience by eliminating a potential failure source.
  • Fewer brokers with up to three times throughput per broker – Higher throughput per broker allows for smaller clusters for the same workload. Standard brokers’ throughput must account for client traffic and background operations, with m7g.16xl Standard brokers safely handling 154 MBps ingress. Express brokers use opinionated settings and resource isolation, enabling m7g.16xl size instances to safely manage up to 500 MBps ingress without compromising performance or availability during cluster events.
  • Higher utilization with 20 times faster scaling – Express brokers reduce data movement during scaling, making them up to 20 times faster than Standard brokers. This allows for more quicker and reliable cluster resizing. You can monitor each broker’s ingress throughput capacity and add brokers within minutes, eliminating the need for over-provisioning in anticipation of traffic spikes.
  • Higher resilience with 90 percent faster recovery – Express brokers are designed for mission-critical applications requiring high resilience. They come preconfigured with best-practice defaults, including 3-way replication (RF=3), which reduce failures due to misconfiguration. Express brokers also recover 90 percent faster from transient failures compared to standard Apache Kafka brokers. Express brokers’ rebalancing and recovery use minimal cluster resources, simplifying capacity planning. This eliminates the risk of increased resource utilization and the need for continuous monitoring when right-sizing clusters.

You have choice options in Amazon MSK depending on your workload and preference:

MSK provisioned MSK Serverless
Standard brokers Express brokers
Configuration range Most flexible Flexible Least flexible
Cluster rebalancing Customer managed Customer managed
but up to 20x faster		 MSK managed
Capacity management Yes Yes (compute only) No
Storage management Yes No No

Express brokers lower costs, provide higher resiliency, and lower operational overhead, making them the best choice for all Kafka workloads. If you prefer to use Kafka without managing any aspect of its capacity, its configuration, or how it scales, then you can choose Amazon MSK Serverless. This provides a fully abstracted Apache Kafka experience that eliminates the need for any infrastructure management, scales automatically, and charges you on a pay-per-use consumption model that doesn’t require you to optimize resource utilization.

Getting started with Express brokers in Amazon MSK
To get started with Express brokers, you can use the Sizing and Pricing worksheet that Amazon MSK provides. This worksheet helps you estimate the cluster size you’ll need to accommodate your workload and also gives you a rough estimate of the total monthly cost you’ll incur.

The throughput requirements of your workload are the primary factor in the size of your cluster. You should also consider other factors, such as partition and connection count to arrive at the size and number of brokers you’ll need for your cluster. For example, if your streaming application needs 30 MBps of data ingress (write) and 80 MBps data egress (read) capacity, you can use three express.m7g.large brokers to meet your throughput needs (assuming the partition count for your workload is within the maximum number of partitions that Amazon MSK recommends for an m7g.large instance).

The following table shows the recommended maximum ingress, egress, and partition counts per instance size for sustainable and safe operations. You can learn more about these recommendations in the Best practices section of Amazon MSK Developer Guide.

Instance size Ingress (MBps) Egress (MBps)
express.m7g.large 15.6 31.2
express.m7g.4xlarge 124.9 249.8
express.m7g.16xlarge 500.0 1000.0

Once you have decided the number and size of Express brokers you’ll need for your workload, go to the AWS Management Console or use the CreateCluster API to create an Amazon MSK provisioned cluster.

When you create a new cluster on the Amazon MSK console, in the Broker type option, choose Express brokers and then select the mount of compute capacity that you want to provision for the broker. As you can see in the screen shot, you can use Apache Kafka 3.6.0 version and Graviton-based instances for Express brokers. You don’t need to preprovision storage for Express brokers.

You can also customize some of these configurations to further fine-tune the performance of your clusters according to your own preferences. To learn more, visit Express broker configurations in the Amazon MSK developer guide.

To create an MSK cluster in the AWS Command Line Interface (AWS CLI), use the create-cluster command.

aws kafka create-cluster \
    --cluster-name "channy-express-cluster" \
    --kafka-version "3.6.0" \
    --number-of-broker-nodes 3 \
    --broker-node-group-info file://brokernodegroupinfo.json

A JSON file named brokernodegroupinfo.json specifies the three subnets over which you want Amazon MSK to distribute the broker nodes.

{
    "InstanceType": "express.m7g.large",
    "BrokerAZDistribution": "DEFAULT",
    "ClientSubnets": [
        "subnet-0123456789111abcd",
        "subnet-0123456789222abcd",
        "subnet-0123456789333abcd"
    ]
}

Once the cluster is created, you can use the bootstrap connection string to connect your clients to the cluster endpoints.

With Express brokers, you can scale vertically (changing instance size) or horizontally (adding brokers). Vertical scaling doubles throughput without requiring partition reassignment. Horizontal scaling adds brokers in sets of three and and allows you to create more partitions, but it requires partition reassignment for new brokers to serve traffic.

A major benefit of Express brokers is that you can add brokers and rebalance partitions within minutes. On the other hand, rebalancing partitions after adding Standard brokers can take several hours. The graph below shows the time it took to rebalance partitions after adding 3 Express brokers to a cluster and reassigning 2000 partitions to each of the new brokers.

As you can see, it took approximately 10 minutes to reassign these partitions and utilize the additional capacity of the new brokers. When we ran the same experiment on an equivalent cluster comprising of Standard brokers, partition reassignment took over 24hours.

To learn more about the partition reassignment, visit Expanding your cluster in the Apache Kafka documentation.

Things to know
Here are some things you should know about Express brokers:

  • Data migration – You can migrate the data in your existing Kafka or MSK cluster to a cluster composed of Express brokers using Amazon MSK Replicator, which copies both the data and the metadata of your cluster to a new cluster.
  • Monitoring – You can monitor your cluster composed of Express brokers in the cluster and at the broker level with Amazon CloudWatch metrics and enable open monitoring with Prometheus to expose metrics using the JMX Exporter and the Node Exporter.
  • Security – Just like with other broker types, Amazon MSK integrates with AWS Key Management Service (AWS KMS) to offer transparent server-side encryption for the storage in Express brokers. When you create an MSK cluster with Express brokers, you can specify the AWS KMS key that you want Amazon MSK to use to encrypt your data at rest. If you don’t specify a KMS key, Amazon MSK creates an AWS managed key for you and uses it on your behalf.

Now available
The Express broker type is available today in the US East (Ohio), US East (N. Virginia), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland), and Europe (Stockholm) Regions.

You pay an hourly rate for Apache Kafka broker instance usage (billed at one-second resolution) for Express brokers, with varying fees depending on the size of the broker instance and active brokers in your MSK clusters. You also pay a per-GB rate for data written to an Express broker (billed at per-byte resolution). To learn more, visit the Amazon MSK pricing page.

Give Express brokers for Amazon MSK a try in the Amazon MSK console. For more information, visit the Amazon MSK Developer Guide and send feedback to AWS re:Post for Amazon MSK or through your usual AWS support contacts.

— Channy

Amazon OpenSearch Service launches the next-generation OpenSearch UI

Post Syndicated from Hang Zuo original https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-launches-the-next-generation-opensearch-ui/

Amazon OpenSearch Service launches a modernized operational analytics experience that can provide comprehensive observability spanning multiple data sources, so that you can gain insights from OpenSearch and other integrated data sources in one place. The launch also introduces OpenSearch Workspaces that provides tailored experience for popular use cases and supports access control, so that you can create a private space for your use case and share it only to your collaborators. With the next-generation user interface (UI), the Discover experience has been improved to simplify interactive analysis, so that you can easily utilize features such as natural language query generation to gain insights from your data.

Multiple Data Source: You might have already used OpenSearch Dashboards to provide an operational analytics experience for your OpenSearch clusters. OpenSearch Dashboards is co-located with a cluster, so that each OpenSearch Dashboards can only work with one cluster. And as you scale up your workload across multiple clusters, there is not a unified experience to analyze your data in one place. In comparison, the next-generation OpenSearch UI is designed to work across multiple OpenSearch clusters to aggregate the comprehensive insights in one place. An OpenSearch application is an instance of the next-generation OpenSearch UI. Currently, OpenSearch applications can be associated with multiple OpenSearch clusters (above version 1.3), Amazon OpenSearch Service Serverless collections, and integrated data sources such as Amazon S3. Each OpenSearch cluster can be associated with multiple OpenSearch applications, in addition to its co-located OpenSearch Dashboards that will remain functional.

Workspace: With workspaces, you can easily create your use case specific contents in a private space and manage the permissions in team collaboration. Workspace provide curated experiences for popular use cases such as Observability, Security Analytics and Search, so that you can find it straightforward to build contents for your use case. Workspace supports collaborator management, so that you can share your workspace only to your intended collaborators, and manage the permissions for each collaborator.

Discover: The improved Discover feature now provides a unified log exploration experience that adds the support for SQL and Piped Processing Language (PPL), in addition to the existing support for DQL and Lucene. Discover features a new data selector to support multiple data sources, a new visual design, query autocomplete and natural language query generation for improved usability. With the enhanced Discover interface, you can now analyze data from multiple sources without switching tools, reducing complexity and improving efficiency.

Solution Overview

The following diagram illustrates architecture of the OpenSearch Dashboards.

The following diagram illustrates the next-generation OpenSearch UI architecture.

In the following sections, we discuss the following topics

  1. The process of creating an application
  2. Setting up and using the new Workspaces functionality
  3. The enhanced Discover experience

We’ll demonstrate how these improvements streamline data analysis, foster collaboration, and empower you to extract insights more efficiently across various use cases.

Create an application:

To begin using the next-generation OpenSearch UI, you can first create an application. An application is an instance of the OpenSearch UI (Dashboards), and you have the flexibility to create multiple applications within a single account. To create a new application, complete the following steps:

  1. On the Amazon OpenSearch Service console, choose OpenSearch UI (Dashboards)under Central management in the navigation panel.
  2. Choose Create application.
  3. For application name, enter a descriptive name for your new application.
  4. AWS Identity and Access Management (IAM) is the default authentication mechanism. Optionally, you can select Authentication with IAM Identity Center (IDC), so that you can use credentials and access management from your existing identity providers to manage user access.
  5. For OpenSearch application admins, specify the IAM principals or IDC users that will have permissions to update or delete the application configuration. You will automatically be set as the first admin.

This page lists all the existing applications under your account in the current AWS region. You can create new application from this page.

This page is the create application workflow. You can specify the application name, enable/disable IDC and define application admins to create an application.

After you configured these settings and created an application, your new OpenSearch application will be ready for you to associate data sources and start using the enhanced UI capabilities.

Associating data sources:

After you create your new OpenSearch application, the next step is to associate the relevant data sources. This allows you to connect the application to the necessary OpenSearch domains, collections, and other data sources.

  1. On the application details page, choose Manage data sources.

You will be presented with a list of all the OpenSearch data sources you have access to, including managed domains and serverless collections.

  1. Select the data sources you want to associate with this application.

OpenSearch domains below version v1.3 will not be compatible with the next-generation UI, and will be grayed out in the list. Additionally, if you need to connect to a domain within a virtual private cloud (VPC), you will need to authorize OpenSearch application as a new principal under its security configurations. If you need to connect to a collection within a VPC, you will need to configure its network policy to Private, enable AWS service private access with OpenSearch application.

  1. Choose Save to finalize the data source association.

Your OpenSearch application is now ready to use, with access to the connected data.

Working with the OpenSearch application:

To access your new OpenSearch application, you can either choose the application URL or choose Launch application on the application details page. After you’ve successfully logged in either with IAM or IDC, you’ll be directed to the application’s homepage. From here, you can choose to create a new workspace or navigate to an existing one that you have access to.

Creating a new workspace:

A workspace is a tailored experience for your use case and team collaboration. There are five types of workspaces: Observability, Security Analytics, Search, Essentials, and Analytics. You can click on the info button to learn more about each workspace type. Existing workspaces will be listed on the homepage. To create a new workspace, complete the following steps:

  1. Choose Create workspace.
  2. Enter a name for your workspace.
  3. Optionally, you can select a different color for the workspace icon for easier identification.
  4. Select the type of workspace you want to create: Observability, Security Analytics, Search, Essentials, or Analytics
  5. Add at least one data source for this workspace (from the list of data sources you previously associated with the application).

For this post, we create an Observability workspace named MyWorkspace and associate it with one Amazon OpenSearch Serverless collection and one Amazon OpenSearch Service managed cluster. You can always manage the data sources associated with a workspace, even after it has been created.

Invite Collaborators

After you create your new workspace, you can add users or groups as collaborators. Workspace collaborators are the users you want to invite to work with you in this workspace, and there are three available permission levels for collaborators: admin, read/write, and read-only. Read/write permission allows a collaborator to create, edit and delete the dashboards, visualizations, and saved queries within the workspace, whereas collaborators with read-only access can only view the results. Admin level gives a collaborator the same permissions as you to not only read/write but also update the configurations of the workspace or delete it.

To add collaborators to your workspace, complete the following steps:

  1. Choose Collaborators in the navigation panel.
  2. Choose Add collaborators.
  3. Choose the type of users you want to add as collaborators. You can add collaborators by their IAM Amazon Resource Name (ARN) or IDC username.
  4. Select a permission level for the collaborator from the three options: Read only, Read and write, and Admin

If you do not know the ARN of your intended collaborator, follow the instruction to check for their ARN, for example.

Improved navigation:

The improved navigation in workspaces provides a more contextual and purpose-built interface, ensuring that each workspace includes only the tools and features relevant to its use case. With enhanced clarity and better organization, the new navigation system is tailored to help you find the features you need quickly, improving overall productivity and minimizing time spent searching through menus.

Revamped Discover experience

Discover is now revamped to offer improved usability and efficiency. You can access multiple data sources, natural language query generation, a new data selector, and polished design with optimized data density, allowing you to effortlessly navigate and analyze your data:

  • Unified language selector – Discover now offers a unified language selector, allowing users to choose from SQL, PPL, Dashboards Query Language (DQL), or Lucene, making it convenient to work with your preferred query languages in one place.

  • Natural language query generation – Discover now supports natural language query building for PPL. Enter your questions in plain language, and Discover converts them to PPL syntax, making data exploration simpler and more accessible. This new feature empowers users of different skill levels to get insights without needing to fully understand the PPL syntax.

  • Powerful query autocomplete – The enhanced query bar includes autocomplete functionality and natural language query generation support, simplifying query building by offering relevant suggestions as you type, making it faster and more efficient to write complex queries

  • New data selector– The new data selector makes it straightforward to connect to multiple data sources, bringing data from Amazon OpenSearch Service domains and serverless collections, and Amazon S3 into a unified view.

Conclusion

In this post, we discussed the features of the next-generation OpenSearch UI. These improvements streamline data analytics, foster collaboration, and empower you to extract insights more efficiently across various use cases.

You can create your own OpenSearch UI applications today in the US East (N. Virginia), US West (N. California, Oregon), Asia Pacific (Mumbai, Singapore, Sydney, Tokyo), South America (São Paulo), Europe (Frankfurt, Ireland, London, Paris) and Canada (Central) Regions.

About the Authors

Hang (Arthur) Zuo is a Senior Product Manager with Amazon OpenSearch Service. Arthur leads the core experience in the next-gen OpenSearch UI and data migration to Amazon OpenSearch Service. Arthur is passionate about cloud technologies and building data products that help users and businesses gain actionable insights and achieve operational excellence.

Rushabh Vora is a Principal Product Manager for the OpenSearch project of Amazon Web Services. Rushabh leads core experiences in data exploration, dashboards, visualizations, reporting, and data management to help organizations unlock insights at scale. Rushabh is passionate about cloud technologies and building products that enable businesses to make data-driven decisions and achieve operational excellence.

Sohaib Katariwala is a Senior Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service based out of Chicago, IL. His interests are in all things data and analytics. More specifically he loves to help customers use AI in their data strategy to solve modern day challenges.

Arun Lakshmanan is a Search Specialist with Amazon OpenSearch Service based out of Chicago, IL. He works closely with customers on their OpenSearch journey across various use cases including vector search, observability, and security analytics.

Xenia Tupitsyna is a UX Designer at OpenSearch. She is working on user experiences across security analytics solutions, anomaly detection, alerting, and core dashboards.