Tag Archives: generative AI

Unifying metadata governance across Amazon SageMaker and Collibra

Post Syndicated from Vasiliki Nikolopoulou original https://aws.amazon.com/blogs/big-data/unifying-metadata-governance-across-amazon-sagemaker-and-collibra/

This post was co-written with Vasiliki Nikolopoulou from Collibra.

Managing metadata across tools and teams is a growing challenge for organizations building modern data and AI platforms. As data volumes grow and generative AI becomes more central to business strategy, teams need a consistent way to define, discover, and govern their datasets, features, and models.

Collibra is a widely adopted data intelligence platform that helps organizations centralize governance workflows, define business glossaries, and enforce policies across data assets. Teams use Collibra to curate business context, classify sensitive data, and manage access to information in line with compliance requirements.

Amazon SageMaker Catalog, part of the next generation of Amazon SageMaker, provides a unified environment where users can register, search, and govern AI and data assets. It allows organizations to organize datasets, trained models, features, and pipelines and apply metadata such as business terms, classifications, ownership, and usage context. Amazon SageMaker Catalog is designed to support collaboration across roles, including data scientists, engineers, and business stakeholders.

As organizations scale their data and AI initiatives, ensuring consistency and trust in metadata becomes increasingly important. Teams need a unified way to manage glossary terms, asset descriptions, classifications, and access governance across platforms. Without this consistency, it becomes difficult to enforce standards, support compliance, and drive collaboration across teams building and consuming data.

To address this challenge, Amazon Web Services (AWS) and Collibra have built a new integrated solution that demonstrates the integration between the Collibra Platform and the next generation of Amazon SageMaker. Developed collaboratively by both companies, the solution is based on the APIs of both products and is designed to help customers explore what’s possible through hands-on testing. It provides a practical example of how metadata synchronization between Collibra and SageMaker can be accomplished in real-world scenarios. With this integration, you can align business and technical metadata across both platforms, so you can extend your governance workflows to AI and analytics assets managed in Amazon SageMaker.

This solution allows metadata to remain consistent across both platforms, regardless of where it was created. It helps reduce duplication, improve metadata quality, and ensure that business context travels with data and AI assets throughout their lifecycle. The integration supports metadata synchronization, glossary term mapping, and access approval workflows using native APIs and automation.

In this post, we take a closer look at the integration, describe the use cases it enables, walk through the architecture, and show how to implement the solution in your environment.

Solution overview

The integration between Amazon SageMaker Catalog and Collibra offers automated, bidirectional metadata synchronization and access governance across both platforms. It’s built using the built-in APIs of Amazon SageMaker and Collibra Data Governance Center (DGC) to provide a scalable and configurable mechanism for metadata exchange. The solution consists of two main capabilities: metadata synchronization and access subscription workflow integration. The following diagram illustrates the solution architecture.

Metadata synchronization

Many organizations manage business and technical metadata across multiple systems. Without synchronization, glossary terms, asset descriptions, and classifications can become inconsistent, leading to duplicated work and misalignment across teams.

This integration allows metadata to flow between Amazon SageMaker Catalog and Collibra, regardless of where it was created. Key elements such as glossary terms, their hierarchy, associated descriptions, and relationships to assets like datasets or columns are automatically synchronized between platforms.

The solution supports:

  • Bidirectional synchronization of glossary terms and descriptions
  • Preservation of glossary structure, including parent-child relationships
  • Association of terms with data assets such as datasets, tables, and columns
  • Synchronization of additional business metadata, such as classifications and data categories
  • Alignment of technical descriptions for datasets and columns between systems

By keeping metadata consistent, the integration reduces manual work, avoids duplication, and provides users in both platforms with the same trusted context.

Subscription and approval flow

Organizations that rely on Collibra for access governance can now extend those workflows to assets cataloged in Amazon SageMaker. After metadata is synchronized, users can discover and request access to datasets directly from within Collibra, using familiar approval processes.

This integration connects Collibra’s workflow engine with the access control mechanism offered by Amazon SageMaker. When an asset is registered in Amazon SageMaker and shared into Collibra, users can initiate a subscription request in Collibra. When it’s approved, access is granted using Amazon the SageMaker built-in access management, which supports multiple AWS services such as AWS Glue and Amazon Redshift.

Key capabilities include:

  • Discovery and access request initiation from Collibra or Amazon SageMaker
  • Centralized review and approval processes managed within Collibra
  • Access provisioning using the Amazon SageMaker grant mechanism
  • Consistent metadata and asset context available throughout the request lifecycle

This flow helps streamline the experience for both business and technical users while keeping access to governed data traceable, auditable, and aligned with organizational policies.

Prerequisites

To perform the solution, you need the following prerequisites:

Walkthrough

The next section provides a walkthrough that shows how the integration works from start to finish. It highlights how a user discovers a data asset, submits a subscription request, and how that request is reviewed, approved, and fulfilled. Throughout the process, metadata and governance policies remain aligned between Collibra and Amazon SageMaker Catalog. This example helps illustrate what the integration enables and how it fits into a typical data access workflow.

Setup on the Collibra environment

To enable this solution, some initial setup is needed in the Collibra environment. This involves configuring the key components that users will need to discover, request, and manage access to data. The following steps outline the basic setup required to support the overall experience.

Operating Model changes and import workflows in Collibra

The operating model of the Collibra instance needs two new asset types and attribute types as well as two new relations and statuses for the scripts and workflows to work properly. These new asset types are recommended because Amazon SageMaker introduces its own concepts and architecture, such as domains and projects. Using the same names in Collibra makes it easier for users to understand and navigate both systems consistently. In the following diagram, the new asset types are shown with dotted lines along with the corresponding new relations, attributes, and statuses.

In addition to AWS projects, the implementation requires synchronization of AWS users beyond the standard capabilities. This is necessary because in AWS, a user can’t subscribe to an asset directly as an individual. They can only do so as a member of a project. As a result, when a user subscribes to an asset, they must specify which project they’re subscribing through. To support this behavior, membership to projects information for AWS users needs to be maintained and synchronized within Collibra. AWS project to user mapping needs to be maintained in Collibra, which is accessed by administrative users. The metadata information about AWS user membership to projects can be kept in a Collibra environment or community, which isn’t accessible to anyone except authorized users. Steps for implementation of Collibra operating model changes:

  1. Go to Settings, then Operating model, and add two new asset types, AWS Project and AWS User.
  2. In Settings, navigate to Attribute types and add the new attribute types. The new attribute types are: Project id assigned to the AWS Project asset type, Membership to Project assigned to the AWS User, AWS Project id, Consuming Project and Consuming Project Id to be assigned to the existing asset type Data Usage. Refer to the documentation for more details on how to add new attribute types and how to assign them to asset types
  3. In Settings, go to Relation types and add the Asset to be used relation between asset types data usage and data asset. Refer to the documentation for guidance on how to add a new relation to a pair of asset types.
  4. In Settings, go to Statuses and add the new statuses, which are Access granted and Pending, to be assigned to the asset type data usage.
  5. Go back to the Operating model and, for each new asset type, add the newly created relations, attributes, and statuses. Don’t skip this step. If it isn’t completed, the new configurations will won’t take effect.
  6. Create the following domains:
    1. AWS Users – This is a business asset domain where the metadata for AWS user memberships will be stored. Users and their memberships are automatically imported into Collibra through the solution. An example is shown in the screenshot.
    2. AWS Projects – This is also a business asset domain where AWS projects and their metadata will be automatically imported. The following screenshot shows an example of such a domain. The AWS projects, along with their published assets, are brought into Collibra through the solution.
    3. AWS Subscription Requests – This is a domain of type data usage registry. It will hold all new AWS subscription requests along with their context, such as the consuming project and the subscribed data asset. The status of each request is especially important because it drives the integration workflow that users can use to track the current state of their request.

Workflows installation

This solution includes two workflows: one for managing subscription request approvals and another for notifying users when access is granted.

The first workflow handles the full subscription process. It begins by prompting the user to select the consuming project because only projects the user is a member of are eligible for subscriptions. After it’s selected, a new subscription request asset is created in Collibra with a timestamp, the consuming project details, and a status set to Pending.

An approval task is then assigned to the business steward of the requested data asset. If the steward approves the request, the status changes to Approved. This triggers a notification to the requester and signals the AWS solution to pick up the request and grant access. When access is granted, the status is updated to Access granted.

If the steward rejects the request, the status is changed to Rejected and the requester is notified. No further action is taken in that case.

The second workflow notifies the requester that the access was granted. It’s triggered by the functions in AWS when the subscription grant is completed. The steps to deploy the two workflows are as follows:

  1. Go to Settings, then select Workflows followed by Definitions, as shown in the following screenshot.

  1. Choose Upload a file, as shown in the following screenshot. Then, upload both workflow files from the GitHub directory where all the files are provided. In that GitHub directory, there is a directory with the workflow files called Workflows.

  1. After the workflows are uploaded, complete the following steps for each one, as shown in the following screenshot:
    1. Enable the workflow by choosing Play. When enabled, the button will display a Pause icon.
    2. Under Rules, set it to apply to Assets, then choose Add Rules and choose Asset: Table. You can also use Data Asset for a broader scope, but in this case, published assets in AWS are tables.
    3. Clear This workflow can only run once at the same time on a specific resource. This provides that multiple users can request subscriptions to the same asset simultaneously.

The workflows are now uploaded, enabled, and ready for use.

Add responsibilities

We need to assign business stewards to the ingested AWS assets so that when the workflows are triggered, there is a designated user responsible for approving subscription requests. In this version of the solution, it’s assumed that each asset has only one Business Steward.

To add a Business Steward, follow these steps:

  1. In the domain or community where the AWS data assets have been ingested using the Edge integration, choose Responsibilities. Then choose Add, as shown in the following screenshot

  1. Choose Business Steward from the Role dropdown list, as shown in the following screenshot. From the Users or groups dropdown list, choose the user who will be responsible for approving subscription requests for these assets. This solution allows only one business steward per asset. You can assign a business steward at the community level, and this way this role will be inherited to all assets under this community.

  1. Choose Add, as shown in the following screenshot. This will assign the selected user to the Business Steward role for the specified asset, domain, or community of assets.

Setup on the AWS environment

Now that the configuration on the Collibra side is complete, set up the Amazon SageMaker domain that is used for this walkthrough. We provide the following assets to help users set up this solution

  1. An AWS CloudFormation template in YAML format, called template.yaml
  2. Instructions to generate a lambda zip file that contains all the scripts that the Cloud Formation will run, called lambda_build.zip
  3. Instructions to create a secret using AWS Secrets Manager that will store Collibra credentials.

Create the CloudFormation stack

To support this solution, provision a set of AWS resources that facilitate communication between environments and automate key tasks. In this section, we show how to deploy the foundational infrastructure using AWS CloudFormation, which simplifies resource provisioning and provides consistency across environments.

  1. On the AWS Management Console, navigate to CloudFormation and choose Create stack, then choose With new resources (standard), as shown in the following screenshot.

  1. Choose the provided CloudFormation template and choose Next.

  1. Enter a name for the stack and complete all required parameters below:

  • CollibraAwsProjectAttributeTypeId – The attribute type ID for AWS projects in Collibra.
  • CollibraAwsProjectDomainId – The domain ID for AWS projects in Collibra.
  • CollibraAwsProjectToAssetRelationTypeId – The relation type ID between AWS projects and assets in Collibra.
  • CollibraAwsProjectTypeId – The type ID for AWS projects in Collibra.
  • CollibraAwsUserDomainId – The domain ID for AWS users in Collibra.
  • CollibraAwsUserProjectAttributeTypeId – The attribute type ID for AWS user projects in Collibra.
  • CollibraAwsUserTypeId – The type ID for AWS users in Collibra.
  • CollibraConfigSecretsName – The name of the AWS Secrets Manager secret containing Collibra configuration and credentials.
  • SMUSProducerProjectId – The project ID in SMUS that contains the data assets to be shared (producer side).
  • SMUSConsumerProjectId – The project ID in SMUS where shared data assets will be accessed (consumer side).
  • SMUSDomainId – The unique identifier for the SageMaker Unified Studio (SMUS) domain.
  • CollibraSubscriptionRequestCreationWorkflowId – The unique identifier for the Collibra workflow that creates subscription requests in Collibra.
  • CollibraSubscriptionRequestApprovalWorkflowId – The unique identifier for the Collibra workflow that approves subscription requests in Collibra.
  • LambdaCodeS3Bucket – The S3 bucket containing the Lambda function deployment package.
  • LambdaCodeS3Key – The S3 key (path and filename) of the Lambda function deployment package within the specified bucket.

  1. Select the acknowledgement checkbox, then choose Next, as shown in the following screenshot.

  1. Choose Submit to start the stack deployment. When the process is complete, the stack status will update to CREATE_COMPLETE.

Configure consumer and producer projects

For this post, only two projects are used: one serving as the producer and one as the consumer. Future versions of the solution are planned to support all projects.

  1. On the AWS Management Console, go to the SMUS Domain detail page. Under the Users section, choose Add, then select Add IAM users.

  1. From the dropdown, select the SMUSCollibraIntegrationAdminRole created by the CloudFormation template, then choose Add user(s), as shown in the following screenshot.

  1. Open the Unified Studio portal for this domain and navigate to the Producer Project. Go to the Members tab and choose Add members.
  2. Search for SMUSCollibraIntegrationAdminRole and select it from the results.

  1. Set the role to Owner, then choose Add members.

  1. Repeat the same steps for the Consumer Project. After adding the member, the configuration should look like the example in the following screenshot.

Make sure the producer project has the necessary authorization to create glossary terms in the domain unit it belongs to. For more information, refer to Domain units and authorization policies in Amazon SageMaker Unified Studio in the Amazon SageMaker Unified Studio documentation.

Synchronization of metadata

Metadata synchronization between Collibra and SageMaker Catalog happens on two distinct levels, each serving a specific purpose.The first level focuses on technical metadata. Collibra connects to services such as Amazon Redshift and AWS Glue using JDBC and other supported connection methods. Through these connections, it ingests schema details including tables, columns, and data types. This helps technical teams maintain visibility into the structure of the datasets available in SageMaker Catalog.The second level, which is the focus of this solution, handles business metadata synchronization. Using Collibra APIs, SageMaker Catalog retrieves business glossary terms, column descriptions, asset definitions, and the relationships among them. Additionally, Collibra ingests information about SageMaker projects, the assets published within them, and project membership details. This supports approval workflows and helps manage subscriptions based on project-level access. The following diagram illustrates how these two levels of metadata synchronization work together to bridge technical and business perspectives across both platforms.

For the technical metadata ingestion from AWS to Collibra, follow these steps:

  1. Within the Collibra Edge site, create a new connection for each type of AWS data store you want to ingest metadata from. For detailed instructions, refer to the About Edge and Collibra Cloud site connections in the Collibra Documentation.
    1. Depending on the type of connection, especially if it’s JDBC, you might need to add a capability such as JDBC catalog ingestion. Refer to the official documentation for more details.
    2. So the integration works correctly, name all your AWS connections in Edge with “AWS” at the start of the name. The integration script relies on this naming convention to accurately identify assets that originate from AWS.
  2. In Collibra, go to Catalog, select your connection, configure the rules for your schemas (such as which tables to include or exclude), and run the synchronization. You can also schedule the synchronization to run automatically at intervals defined in the user interface.
  3. When metadata ingestion is complete, go to Catalog, then Data Sources. You can optionally filter by a specific AWS source or keep the default view to view all sources. From there, you can review the schemas, tables, and other metadata imported from AWS, as shown in the following diagram.

These data assets are imported using the JDBC connections that are available from Collibra Edge. The AWS solution we present here, in addition to these data assets, will import AWS projects and will link them to the assets ingested here that are published in these projects.

Technical and business stewardship in Collibra

Collibra provides business glossaries to define business context. These glossaries can also include a hierarchy or taxonomy of business terms based on their interdependence. The following is an example of a glossary used for this post.

An Order includes components such as Order Date, Order ID, and others. In Collibra, Business and Technical Stewards are responsible for linking Business Terms to the columns and tables ingested from AWS, as shown in the following diagram. For detailed guidance on how to perform stewardship activities, refer to the official Collibra documentation.

The entire business glossary with its one-level hierarchy is imported into AWS SageMaker Unified Studio automatically with this solution. Some business terms are also linked to data categories that are associated with data privacy, regulatory policies, and standards. In the example in the following screenshot, customer ID is connected to a data category. This connection between business terms and data categories links the associated data to relevant policies and standards. As a result, a table or column connected to a business term that is linked to a data category will also inherit the associated policy or standard.

The business term customer ID is linked to the data category personally identifiable information (PII). With this relation, all columns or tables that are linked to this business term automatically inherit the PII data category, and therefore the policies linked and associated with it.

The metadata is imported into AWS SageMaker Unified Studio at the asset and schema levels.

All the business metadata described previously is synchronized with AWS using this solution. Descriptions, data categories, tags, business terms are all imported into AWS and linked to respective assets. In the README, the data category is associated with one of the columns and the business term associated with a table or dataset.From Collibra we import into AWS the following:

  • Business terms and their hierarchies and descriptions
  • The link of the business terms to the technical assets
  • Data category of business terms inherited in the technical assets imported in the README section of the technical asset
  • Tags and descriptions of technical data assets

Not only is the business term imported into AWS SageMaker Unified Studio, its taxonomy is imported exactly as it is in Collibra. The following screenshot shows an example where order is imported to have under it the business terms order ID, quantity, and so on.

Subscription to published assets

For the subscription process, the same workflows and series of tasks occur whether the request is initiated from AWS or from Collibra. An overview of these tasks and the end-to-end flow from both platforms is shown in the following diagrams:

This diagram outlines the subscription request flow when initiated from Collibra. A user searches for a business term, locates the related asset, and submits a subscription request. The system creates a corresponding request asset in Collibra. The user then selects the destination project for the data. An approval workflow is triggered, notifying the designated business steward. If the request is approved, SageMaker Catalog automatically provisions access and updates the request status to Access Granted. The user receives a final notification confirming access. This process provides controlled, transparent data sharing across platforms.

The following diagram illustrates the end-to-end subscription flow when the data user initiates the process from within SageMaker Studio. The user begins by searching for data using a business term and selecting the relevant asset. After choosing the appropriate table, they request access, which triggers the creation of a subscription request asset in Collibra. The user then selects a destination project based on their memberships. Collibra sends an approval request to the designated business steward, who reviews and either approves or rejects it. If approved, SageMaker Catalog automatically provisions the subscription and notifies the requester. The subscription request status is then updated to Access Granted, completing the workflow.

For this post, the process is described starting from Collibra, although it functions the same way if initiated from AWS. In this example, a data consumer is searching for data related to AWS orders using the Collibra interface.

In Amazon SageMaker Unified Studio, the data consumer is a member of the Orders and Products project. At this stage, the user has no active subscriptions or access to data assets. The following screenshot is included to illustrate the state before the integration takes effect.

  1. In Collibra, navigate to the Search area and enter a business-friendly term describing what the user is looking for. In this example, enter order.

  1. In the Data Marketplace, filters such as Business Terms can be applied to narrow the results by asset type, as shown in the following screenshot. This approach helps users focus on relevant assets by starting from clear business context, which is especially useful when dealing with many similarly named tables or columns.

  1. In the example shown in the following screenshot, the business term Order is selected, and the Diagram view is opened to display its full logical lineage. The diagram shows that the term is linked to the aws_orders table. Selecting the table in the diagram reveals its metadata details, which appear on the right side of the page. From there, users can navigate directly to the table.

  1. In the aws_orders table asset, access can be requested by initiating an AWS subscription request. From the asset view, selecting Actions reveals the list of available workflows. In this example, the Creation of a new subscription workflow is selected to start the approval process.

  1. The user must select the AWS project to use as the consuming project for the subscription. A list of all projects the user is a member of is displayed to facilitate the selection. After choosing the appropriate project, choose Send to submit the request.

  1. After it’s submitted, the workflow is triggered, and a task is assigned to the business steward of the asset for which the subscription is requested. A new subscription request is also created in the AWS Subscription Requests domain with a status of Pending, and it’s automatically linked to the requested asset.

The new subscription request is also reflected in the lineage of the data asset, as shown in the following screenshot.

  1. The business steward assigned to the asset receives an approval notification.
    1. Choose Tasks button in the top right corner.
    2. Locate the most recent task titled Accept or Reject, which is associated with the aws_orders asset.

  1. The business steward opens the task and chooses either Approve or Reject, depending on the request. In this example, Approve is selected. The task is then marked as complete.

  1. After the business steward approves the subscription request, the corresponding Subscription Request asset is automatically updated to the status Approved.

  1. The requester is notified that the subscription request has been approved. To acknowledge, the requester choose Tasks, locates the approval notification, and chooses Done to confirm receipt, as shown in the following screenshot.

  1. After a subscription request is approved, the integration solution automatically process the request by creating and granting the corresponding subscription in AWS using the asset’s metadata. The user can then confirm the new subscription is reflected in Amazon SageMaker, as shown in the following screenshot.

  1. After the subscription is granted, the status of the Subscription Request is updated to Access Granted.

  1. The requester now receives a new task, which is a notification confirming that the subscription request has been granted. Choose the Send button to acknowledge and complete the task.

  1. In the AWS Subscription Requests domain, all requests and their status are visible. In addition to Approved and Access Granted statuses, Rejected requests are also listed. If a request is rejected by the approver, its status changes to Rejected and no subscription is created in AWS.

Synchronization Interval

The solution keeps Collibra and Amazon SageMaker Catalog in sync through regular updates. Core elements including business metadata of Collibra, user profiles, project information & published assets of Amazon SageMaker Catalog, and subscription requests originating in Collibra are synchronized every 5 minutes. However, when subscription requests are created in Amazon SageMaker Catalog, they are instantly synchronized to Collibra.

Cleanup

To avoid incurring unnecessary costs after testing or exploring the solution, delete the provisioned resources. Follow these steps:

  1. Remove the CloudFormation stack – Go to the AWS CloudFormation console, select the stack you created for this solution, and choose Delete. This will automatically remove the associated AWS resources provisioned by the stack.
  2. Clean up Collibra configurations – In the Collibra environment, remove test domains, projects, or workflows created for this solution to ensure a clean slate for future experiments.
  3. Revoke access tokens or credentials – If you used API credentials or access tokens for integration, ensure they’re revoked or deleted if no longer needed.

Performing these steps ensures your environments stay clean and you avoid unintended resource usage.

Conclusion

The solution connecting Amazon SageMaker Catalog and Collibra gives organizations a simple way to unify metadata and streamline access workflows. It helps reduce duplication, improve governance, and build trust in data for both analytics and AI.We demonstrated how to synchronize metadata and manage access requests using APIs, enabling a shared view of data across teams.Learn more by exploring:

We welcome your feedback as you explore what’s possible with this solution.

About the authors

Vasiliki Nikolopoulou is a Principal Integrations Architect at Collibra, where she is working for the past 11 years. Her extensive career includes roles such as Director, Enterprise Architect at AXA Insurance US, Principal Sales Engineer at Oracle, and Certified Senior IT Professional in technical sales at IBM for over 15 years. She holds numerous technical certifications. Connect with her on LinkedIn.

Divij Bhatia is a Software Development Engineer at AWS. He is passionate about building resilient and scalable cloud-native solutions that solve real-world problems for customers. His free time often takes him outdoors, traveling and shooting landscapes. Connect with him on LinkedIn.

Leonardo Gomez is a Principal Analytics Specialist Solutions Architect at AWS. He has over a decade of experience in data management, helping customers around the globe address their business and technical needs. Connect with him on LinkedIn.

Announcing Amazon Nova customization in Amazon SageMaker AI

Post Syndicated from Betty Zheng (郑予彬) original https://aws.amazon.com/blogs/aws/announcing-amazon-nova-customization-in-amazon-sagemaker-ai/

Today, we’re announcing a suite of customization capabilities for Amazon Nova in Amazon SageMaker AI. Customers can now customize Nova Micro, Nova Lite, and Nova Pro across the model training lifecycle, including pre-training, supervised fine-tuning, and alignment. These techniques are available as ready-to-use Amazon SageMaker recipes with seamless deployment to Amazon Bedrock, supporting both on-demand and provisioned throughput inference.

Amazon Nova foundation models power diverse generative AI use cases across industries. As customers scale deployments, they need models that reflect proprietary knowledge, workflows, and brand requirements. Prompt optimization and retrieval-augmented generation (RAG) work well for integrating general-purpose foundation models into applications, however business-critical workflows require model customization to meet specific accuracy, cost, and latency requirements.

Choosing the right customization technique
Amazon Nova models support a range of customization techniques including: 1) supervised fine-tuning, 2) alignment, 3) continued pre-training, and 4) knowledge distillation. The optimal choice depends on goals, use case complexity, and the availability of data and compute resources. You can also combine multiple techniques to achieve your desired outcomes with the preferred mix of performance, cost, and flexibility.

Supervised fine-tuning (SFT) customizes model parameters using a training dataset of input-output pairs specific to your target tasks and domains. Choose from the following two implementation approaches based on data volume and cost considerations:

  • Parameter-efficient fine-tuning (PEFT) — updates only a subset of model parameters through lightweight adapter layers such as LoRA (Low-Rank Adaptation). It offers faster training and lower compute costs compared to full fine-tuning. PEFT-adapted Nova models are imported to Amazon Bedrock and invoked using on-demand inference.
  • Full fine-tuning (FFT) — updates all the parameters of the model and is ideal for scenarios when you have extensive training datasets (tens of thousands of records). Nova models customized through FFT can also be imported to Amazon Bedrock and invoked for inference with provisioned throughput.

Alignment steers the model output towards desired preferences for product-specific needs and behavior, such as company brand and customer experience requirements. These preferences may be encoded in multiple ways, including empirical examples and policies. Nova models support two preference alignment techniques:

  • Direct preference optimization (DPO) — offers a straightforward way to tune model outputs using preferred/not preferred response pairs. DPO learns from comparative preferences to optimize outputs for subjective requirements such as tone and style. DPO offers both a parameter-efficient version and a full-model update version. The parameter-efficient version supports on-demand inference.
  • Proximal policy optimization (PPO) — uses reinforcement learning to enhance model behavior by optimizing for desired rewards such as helpfulness, safety, or engagement. A reward model guides optimization by scoring outputs, helping the model learn effective behaviors while maintaining previously learned capabilities.

Continued pre-training (CPT) expands foundational model knowledge through self-supervised learning on large quantities of unlabeled proprietary data, including internal documents, transcripts, and business-specific content. CPT followed by SFT and alignment through DPO or PPO provides a comprehensive way to customize Nova models for your applications.

Knowledge distillation transfers knowledge from a larger “teacher” model to a smaller, faster, and more cost-efficient “student” model. Distillation is useful in scenarios where customers do not have adequate reference input-output samples and can leverage a more powerful model to augment the training data. This process creates a customized model of teacher-level accuracy for specific use cases and student-level cost-effectiveness and speed.

Here is a table summarizing the available customization techniques across different modalities and deployment options. Each technique offers specific training and inference capabilities depending on your implementation requirements.

Recipe Modality Training Inference
Amazon Bedrock Amazon SageMaker Amazon Bedrock On-demand Amazon Bedrock Provisioned Throughput
Supervised fine tuning Text, image, video
Parameter-efficient fine-tuning (PEFT) ✅ ✅ ✅ ✅
Full fine-tuning ✅ ✅
Direct preference optimization (DPO)  Text, image, video
Parameter-efficient DPO ✅ ✅ ✅
Full model DPO ✅ ✅
Proximal policy optimization (PPO)  Text-only ✅ ✅
Continuous pre-training  Text-only ✅ ✅
Distillation Text-only ✅ ✅ ✅ ✅

Early access customers, including Cosine AI, Massachusetts Institute of Technology (MIT) Computer Science and Artificial Intelligence Laboratory (CSAIL), Volkswagen, Amazon Customer Service, and Amazon Catalog Systems Service, are already successfully using Amazon Nova customization capabilities.

Customizing Nova models in action
The following walks you through an example of customizing the Nova Micro model using direct preference optimization on an existing preference dataset. To do this, you can use Amazon SageMaker Studio.

Launch your SageMaker Studio in the Amazon SageMaker AI console and choose JumpStart, a machine learning (ML) hub with foundation models, built-in algorithms, and pre-built ML solutions that you can deploy with a few clicks.

Then, choose Nova Micro, a text-only model that delivers the lowest latency responses at the lowest cost per inference among the Nova model family, and then choose Train.

Next, you can choose a fine-tuning recipe to train the model with labeled data to enhance performance on specific tasks and align with desired behaviors. Choosing the Direct Preference Optimization offers a straightforward way to tune model outputs with your preferences.

When you choose Open sample notebook, you have two environment options to run the recipe: either on the SageMaker training jobs or SageMaker Hyperpod:

Choose Run recipe on SageMaker training jobs when you don’t need to create a cluster and train the model with the sample notebook by selecting your JupyterLab space.

Alternately, if you want to have a persistent cluster environment optimized for iterative training processes, choose Run recipe on SageMaker HyperPod. You can choose a HyperPod EKS cluster with at least one restricted instance group (RIG) to provide a specialized isolated environment, which is required for such Nova model training. Then, choose your JupyterLabSpace and Open sample notebook.

This notebook provides an end-to-end walkthrough for creating a SageMaker HyperPod job using a SageMaker Nova model with a recipe and deploying it for inference. With the help of a SageMaker HyperPod recipe, you can streamline complex configurations and seamlessly integrate datasets for optimized training jobs.

In SageMaker Studio, you can see that your SageMaker HyperPod job has been successfully created and you can monitor it for further progress.

After your job completes, you can use a benchmark recipe to evaluate if the customized model performs better on agentic tasks.

For comprehensive documentation and additional example implementations, visit the SageMaker HyperPod recipes repository on GitHub. We continue to expand the recipes based on customer feedback and emerging ML trends, ensuring you have the tools needed for successful AI model customization.

Availability and getting started
Recipes for Amazon Nova on Amazon SageMaker AI are available in US East (N. Virginia). Learn more about this feature by visiting the Amazon Nova customization webpage and Amazon Nova user guide and get started in the Amazon SageMaker AI console.

–Betty

Introducing Amazon Bedrock AgentCore: Securely deploy and operate AI agents at any scale (preview)

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/introducing-amazon-bedrock-agentcore-securely-deploy-and-operate-ai-agents-at-any-scale/

In just a few years, foundation models (FMs) have evolved from being used directly to create content in response to a user’s prompt, to now powering AI agents, a new class of software applications that use FMs to reason, plan, act, learn, and adapt in pursuit of user-defined goals with limited human oversight. This new wave of agentic AI is enabled by the emergence of standardized protocols such as Model Context Protocol (MCP) and Agent2Agent (A2A) that simplify how agents connect with other tools and systems.

In fact, building AI agents that can reliably perform complex tasks has become increasingly accessible thanks to open source frameworks like CrewAI, LangGraph, and Strands Agents. However, moving from a promising proof-of-concept to a production-ready agent that can scale to thousands of users presents significant challenges.

Instead of being able to focus on the core features of the agent, developers and AI engineers have to spend months building foundational infrastructure for session management, identity controls, memory systems, and observability—at the same time supporting security and compliance.

Today, we’re excited to announce the preview of Amazon Bedrock AgentCore, a comprehensive set of enterprise-grade services that help developers quickly and securely deploy and operate AI agents at scale using any framework and model, hosted on Amazon Bedrock or elsewhere.

More specifically, we are introducing today:

AgentCore Runtime – Provides sandboxed low-latency serverless environments with session isolation, supporting any agent framework including popular open source frameworks, tools, and models, and handling multimodal workloads and long-running agents.

AgentCore Memory – Manages session and long-term memory, providing relevant context to models while helping agents learn from past interactions.

AgentCore Observability – Offers step-by-step visualization of agent execution with metadata tagging, custom scoring, trajectory inspection, and troubleshooting/debugging filters.

AgentCore Identity – Enables AI agents to securely access AWS services and third-party tools and services such as GitHub, Salesforce, and Slack, either on behalf of users or by themselves with pre-authorized user consent.

AgentCore Gateway – Transforms existing APIs and AWS Lambda functions into agent-ready tools, offering unified access across protocols, including MCP, and runtime discovery.

AgentCore Browser – Provides managed web browser instances to scale your agents’ web automation workflows.

AgentCore Code Interpreter – Offers an isolated environment to run the code your agents generate.

These services can be used individually and are optimized to work together so developers don’t need to spend time piecing together components. AgentCore can work with open source or custom AI agent frameworks, giving teams the flexibility to maintain their preferred tools while gaining enterprise capabilities. To integrate these services into their existing code, developers can use the AgentCore SDK.

You can now discover, buy, and run pre-built agents and agent tools from AWS Marketplace with AgentCore Runtime. With just a few lines of code, your agents can securely connect to API-based agents and tools from AWS Marketplace with AgentCore Gateway to help you run complex workflows while maintaining compliance and control.

AgentCore eliminates tedious infrastructure work and operational complexity so development teams can bring groundbreaking agentic solutions to market faster.

Let’s see how this works in practice. I’ll share more info on the services as we use them.

Deploying a production-ready customer support assistant with Amazon Bedrock AgentCore (Preview)
When customers reach out with an email, it takes time to provide a reply. Customer support needs to check the validity of the email, find who the actual customer is in the customer relationship management (CRM) system, check their orders, and use product-specific knowledge bases to find the information required to prepare an answer.

An AI agent can simplify that by connecting to the internal systems, retrieve contextual information using a semantic data source, and draft a reply for the support team. For this use case, I built a simple prototype using Strands Agents. For simplicity and to validate the scenario, the internal tools are simulated using Python functions.

When I talk to developers, they tell me that similar prototypes, covering different use cases, are being built in many companies. When these prototypes are demonstrated to the company leadership and receive confirmation to proceed, the development team has to define how to go in production and satisfy the usual requirements for security, performance, availability, and scalability. This is where AgentCore can help.

Step 1 – Deploying to the cloud with AgentCore Runtime

AgentCore Runtime is a new service to securely deploy, run, and scale AI agents, providing isolation so that each user session runs in its own protected environment to help prevent data leakage—a critical requirement for applications handling sensitive data.

To match different security postures, agents can use different network configurations:

Sandbox – To only communicate with allowlisted AWS services.

Public – To run with managed internet access.

VPC-only (coming soon) – This option will allow to access resources hosted in a customer’s VPC or connected via AWS PrivateLink endpoints.

To deploy the agent to the cloud and get a secure, serverless endpoint with AgentCore Runtime, I add to the prototype a few lines of code using the AgentCore SDK to:

  • Import the AgentCore SDK.
  • Create the AgentCore app.
  • Specify which function is the entry point to invoke the agent.

Using a different or custom agent framework is a matter of replacing the agent invocation inside the entry point function.

Here’s the code of the prototype. The three lines I added to use AgentCore Runtime are the ones preceded by a comment.

from strands import Agent, tool
from strands_tools import calculator, current_time

# Import the AgentCore SDK
from bedrock_agentcore.runtime import BedrockAgentCoreApp

WELCOME_MESSAGE = """
Welcome to the Customer Support Assistant! How can I help you today?
"""

SYSTEM_PROMPT = """
You are an helpful customer support assistant.
When provided with a customer email, gather all necessary info and prepare the response email.
When asked about an order, look for it and tell the full description and date of the order to the customer.
Don't mention the customer ID in your reply.
"""

@tool
def get_customer_id(email_address: str):
    if email_address == "[email protected]":
        return { "customer_id": 123 }
    else:
        return { "message": "customer not found" }

@tool
def get_orders(customer_id: int):
    if customer_id == 123:
        return [{
            "order_id": 1234,
            "items": [ "smartphone", "smartphone USB-C charger", "smartphone black cover"],
            "date": "20250607"
        }]
    else:
        return { "message": "no order found" }

@tool
def get_knowledge_base_info(topic: str):
    kb_info = []
    if "smartphone" in topic:
        if "cover" in topic:
            kb_info.append("To put on the cover, insert the bottom first, then push from the back up to the top.")
            kb_info.append("To remove the cover, push the top and bottom of the cover at the same time.")
        if "charger" in topic:
            kb_info.append("Input: 100-240V AC, 50/60Hz")
            kb_info.append("Includes US/UK/EU plug adapters")
    if len(kb_info) > 0:
        return kb_info
    else:
        return { "message": "no info found" }

# Create an AgentCore app
app = BedrockAgentCoreApp()

agent = Agent(
    system_prompt=SYSTEM_PROMPT,
    tools=[calculator, current_time, get_customer_id, get_orders, get_knowledge_base_info]
)

# Specify the entrypoint function invoking the agent
@app.entrypoint
def invoke(payload, context: RequestContext):
    """Handler for agent invocation"""
    user_message = payload.get(
        "prompt", "No prompt found in input, please guide customer to create a json payload with prompt key"
    )
    result = agent(user_message)
    return {"result": result.message}

if __name__ == "__main__":
    app.run()

I install the AgentCore SDK and the starter toolkit in the Python virtual environment:

pip install bedrock-agentcore bedrock-agentcore-starter-toolkit

After I activate the virtual environment, I have access to the AgentCore command line interface (CLI) provided by the starter toolkit.

First, I use agentcore configure --entrypoint my_agent.py -er <IAM_ROLE_ARN> to configure the agent, passing the AWS Identity and Access Management (IAM) role that the agent will assume. In this case, the agent needs access to Amazon Bedrock to invoke the model. The role can give access to other AWS resources used by an agent, such as an Amazon Simple Storage Service (Amazon S3) bucket or a Amazon DynamoDB table.

I launch the agent locally with agentcore launch --local. When running locally, I can interact with the agent using agentcore invoke --local <PAYLOAD>. The payload is passed to the entry point function. Note that the JSON syntax of the invocations is defined in the entry point function. In this case, I look for prompt in the JSON payload, but can use a different syntax depending on your use case.

When I am satisfied by local testing, I use agentcore launch to deploy to the cloud.

After the deployment is succesful and an endpoint has been created, I check the status of the endpoint with agentcore status and invoke the endpoint with agentcore invoke <PAYLOAD>. For example, I pass a customer support request in the invocation:

agentcore invoke '{"prompt": "From: [email protected] – Hi, I bought a smartphone from your store. I am traveling to Europe next week, will I be able to use the charger? Also, I struggle to remove the cover. Thanks, Danilo"}'

Step 2 – Enabling memory for context

After an agent has been deployed in the AgentCore Runtime, the context needs to be persisted to be available for a new invocation. I add AgentCore Memory to maintain session context using its short-term memory capabilities.

First, I create a memory client and the memory store for the conversations:

from bedrock_agentcore.memory import MemoryClient

memory_client = MemoryClient(region_name="us-east-1")

memory = memory_client.create_memory_and_wait(
    name="CustomerSupport", 
    description="Customer support conversations"
)

I can now use create_event to stores agent interactions into short-term memory:

memory_client.create_event(
    memory_id=memory.get("id"), # Identifies the memory store
    actor_id="user-123",        # Identifies the user
    session_id="session-456",   # Identifies the session
    messages=[
        ("Hi, ...", "USER"),
        ("I'm sorry to hear that...", "ASSISTANT"),
        ("get_orders(customer_id='123')", "TOOL"),
        . . .
    ]
)

I can load the most recent turns of a conversations from short-term memory using list_events:

conversations = memory_client.list_events(
    memory_id=memory.get("id"), # Identifies the memory store
    actor_id="user-123",        # Identifies the user 
    session_id="session-456",   # Identifies the session
    max_results=5               # Number of most recent turns to retrieve
)

With this capability, the agent can maintain context during long sessions. But when a users come back with a new session, the conversation starts blank. Using long-term memory, the agent can personalize user experiences by retaining insights across multiple interactions.

To extract memories from a conversation, I can use built-in AgentCore Memory policies for user preferences, summarization, and semantic memory (to capture facts) or create custom policies for specialized needs. Data is stored encrypted using a namespace-based storage for data segmentation.

I change the previous code creating the memory store to include long-term capabilities by passing a semantic memory strategy. Note that an existing memory store can be updated to add strategies. In that case, the new strategies are applied to newer events.

memory = memory_client.create_memory_and_wait(
    name="CustomerSupport", 
    description="Customer support conversations",
    strategies=[{
        "semanticMemoryStrategy": {
            "name": "semanticFacts",
            "namespaces": ["/facts/{actorId}"]
        }
    }]
)

After long-term memory has been configured for a memory store, calling create_event will automatically apply those strategies to extract information from the conversations. I can then retrieve memories extracted from the conversation using a semantic query:

memories = memory_client.retrieve_memories(
    memory_id=memory.get("id"),
    namespace="/facts/user-123",
    query="smartphone model"
)

In this way, I can quickly improve the user experience so that the agent remembers customer preferences and facts that are outside of the scope of the CRM and use this information to improve the replies.

Step 3 – Adding identity and access controls

Without proper identity controls, access from the agent to internal tools always uses the same access level. To follow security requirements, I integrate AgentCore Identity so that the agent can use access controls scoped to the user’s or agent’s identity context.

I set up an identity client and create a workload identity, a unique identifier that represents the agent within the AgentCore Identity system:

from bedrock_agentcore.services.identity import IdentityClient

identity_client = IdentityClient("us-east-1")
workload_identity = identity_client.create_workload_identity(name="my-agent")

Then, I configure the credential providers, for example:

google_provider = identity_client.create_oauth2_credential_provider(
    {
        "name": "google-workspace",
        "credentialProviderVendor": "GoogleOauth2",
        "oauth2ProviderConfigInput": {
            "googleOauth2ProviderConfig": {
                "clientId": "your-google-client-id",
                "clientSecret": "your-google-client-secret",
            }
        },
    }
)

perplexity_provider = identity_client.create_api_key_credential_provider(
    {
        "name": "perplexity-ai",
        "apiKey": "perplexity-api-key"
    }
)

I can then add the @requires_access_token Python decorator (passing the provider name, the scope, and so on) to the functions that need an access token to perform their activities.

Using this approach, the agent can verify the identity through the company’s existing identity infrastructure, operate as a distinct, authenticated identity, act with scoped permissions and integrate across multiple identity providers (such as Amazon Cognito, Okta, or Microsoft Entra ID) and service boundaries including AWS and third-party tools and services (such as Slack, GitHub, and Salesforce).

To offer robust and secure access controls while streamlining end-user and agent builder experiences, AgentCore Identity implements a secure token vault that stores users’ tokens and allows agents to retrieve them securely.

For OAuth 2.0 compatible tools and services, when a user first grants consent for an agent to act on their behalf, AgentCore Identity collects and stores the user’s tokens issued by the tool in its vault, along with securely storing the agent’s OAuth client credentials. Agents, operating with their own distinct identity and when invoked by the user, can then access these tokens as needed, reducing the need for frequent user consent.

When the user token expires, AgentCore Identity triggers a new authorization prompt to the user for the agent to obtain updated user tokens. For tools that use API keys, AgentCore Identity also stores these keys securely and gives agents controlled access to retrieve them when needed. This secure storage streamlines the user experience while maintaining robust access controls, enabling agents to operate effectively across various tools and services.

Step 4 – Expanding agent capabilities with AgentCore Gateway

Until now, all internal tools are simulated in the code. Many agent frameworks, including Strands Agents, natively support MCP to connect to remote tools. To have access to internal systems (such as CRM and order management) via an MCP interface, I use AgentCore Gateway.

With AgentCore Gateway, the agent can access AWS services using Smithy models, Lambda functions, and internal APIs and third-party providers using OpenAPI specifications. It employs a dual authentication model to have secure access control for both incoming requests and outbound connections to target resources. Lambda functions can be used to integrate external systems, particularly applications that lack standard APIs or require multiple steps to retrieve information.

AgentCore Gateway facilitates cross-cutting features that most customers would otherwise need to build themselves, including authentication, authorization, throttling, custom request/response transformation (to match underlying API formats), multitenancy, and tool selection.

The tool selection feature helps find the most relevant tools for a specific agent’s task. AgentCore Gateway brings a uniform MCP interface across all these tools, using AgentCore Identity to provide an OAuth interface for tools that do not support OAuth out of the box like AWS services.

Step 5 – Adding capabilities with AgentCore Code Interpreter and Browser tools

To answer to customer requests, the customer support agent needs to perform calculations. To simplify that, I use the AgentCode SDK to add access to the AgentCore Code Interpreter.

Similarly, some of the integrations required by the agent don’t implement a programmatic API but need to be accessed through a web interface. I give access to the AgentCore Browser to let the agent navigate those web sites autonomously.

Step 6 – Gaining visibility with observability

Now that the agent is in production, I need visibility into its activities and performance. AgentCore provides enhanced observability to help developers effectively debug, audit, and monitor their agent performance in production. It comes with built-in dashboards to track essential operational metrics such as session count, latency, duration, token usage, error rates, and component-level latency and error breakdowns. AgentCore also gives visibility into an agent’s behavior by capturing and visualizing both the end-to-end traces, as well as “spans” that capture each step of the agent workflow including tool invocations, memory

The built-in dashboards offered by this service help reveal performance bottlenecks and identify why certain interactions might fail, enabling continuous improvement and reducing the mean time to detect (MTTD) and mean time to repair (MTTR) in case of issues.

AgentCore supports OpenTelemetry to help integrate agent telemetry data with existing observability platforms, including Amazon CloudWatch, Datadog, LangSmith, and Langfuse.

Step 7 – Conclusion

Through this journey, we transformed a local prototype into a production-ready system. Using AgentCore modular approach, we implemented enterprise requirements incrementally—from basic deployment to sophisticated memory, identity management, and tool integration—all while maintaining the existing agent code.

Things to know
Amazon Bedrock AgentCore is available in preview in US East (N. Virginia), US West (Oregon), Asia Pacific (Sydney), and Europe (Frankfurt). You can start using AgentCore services through the AWS Management Console , the AWS Command Line Interface (AWS CLI), the AWS SDKs, or via the AgentCore SDK.

You can try AgentCore services at no charge until September 16, 2025. Standard AWS pricing applies to any additional AWS Services used as part of using AgentCore (for example, CloudWatch pricing will apply for AgentCore Observability). Starting September 17, 2025, AWS will bill you for AgentCore service usage based on this page.

Whether you’re building customer support agents, workflow automation, or innovative AI-powered experiences, AgentCore provides the foundation you need to move from prototype to production with confidence.

To learn more and start deploying production-ready agents, visit the AgentCore documentation. For code examples and integration guides, check out the AgentCore samples GitHub repo.

Join the AgentCore Preview Discord server to provide feedback and discuss use cases. We’d like to hear from you!

— Danilo

Migrate and modernize VMware workloads with AWS Transform for VMware

Post Syndicated from Kiran Reid original https://aws.amazon.com/blogs/architecture/migrate-and-modernize-vmware-workloads-with-aws-transform-for-vmware/

On May 15, 2025, AWS unveiled a game-changing solution: AWS Transform for VMware. This innovative service tackles head-on the longstanding challenges of cloud migration, ushering in a new era of streamlined, efficient transitions to the AWS Cloud. By significantly reducing manual effort and accelerating the migration of critical VMware workloads, AWS Transform for VMware is set to revolutionize how organizations approach their cloud journey.

Since its general availability announcement, AWS Transform for VMware has ignited enthusiasm across industries, with organizations eager to leverage its capabilities to accelerate their VMware workload migration and modernization initiatives. As we dive into the intricacies of this transformative technology, we’ll uncover how AWS Transform for VMware is not just simplifying migrations, but reshaping the very landscape of cloud adoption and digital transformation.

The VMware migration challenge

Moving enterprise workloads to the cloud isn’t just a technical challenge – it’s a business transformation that demands precision, speed, and minimal disruption. Years of established operational processes have often led to complex environments with poorly documented configurations, inconsistent security practices, and heavy reliance on institutional knowledge. Technical teams must navigate intricate application dependencies, coordinate across multiple stakeholders, and maintain business continuity while executing these transformational projects. The lack of comprehensive documentation and clear understanding of system inter-dependencies frequently results in extended migration timelines and increased project risks. Additionally, the need to balance ongoing operations with migration activities presents challenges. Achieving proper knowledge transfer adds another layer of complexity to these critical initiatives.

Solution overview

Let’s explore how AWS Transform for VMware simplifies application discovery, automates network conversion, and orchestrates complex migrations through its comprehensive architecture in the following diagram.

To understand how these capabilities work together, let’s examine each component of the architecture.

Streamlined discovery and assessment

The journey begins with a thorough discovery and assessment of your VMware environment (1). AWS Transform for VMware (4) supports multiple discovery methods. One option is RVTools for VMware inventory collection. For customers running VMware NSX, there’s optional import/export functionality. Additionally, AWS Application Discovery Service offers both agent-based and agentless discovery options (2) to gather and collect data and dependencies for migration.

The Inventory Discovery capability (5) collects crucial data from your source environment and stores it securely in Amazon Simple Storage Service (Amazon S3) buckets (12) within the AWS Migration Discovery Account (7). This data forms the foundation for informed migration planning and is further processed by AWS Application Discovery Service (15) in the AWS Migration Planning Account. AWS Transform works together with these services to provide a single place to track migration progress and collect server inventory and dependency data, which is essential for successful application grouping and wave planning.

Intelligent network conversion and wave planning

With a comprehensive understanding of your environment, AWS Transform for VMware moves to the next critical phase. The Network Migration capability (19) automates the creation of AWS CloudFormation templates (13, 26) to set up the target network infrastructure. These templates ensure your cloud environment closely mirrors your source setup, simplifying the setup for the migration.

Meanwhile, the Wave Planning capability (6) uses advanced graph neural networks to analyze application dependencies and plan optimal migration waves. This minimizes complex portfolio and application dependency analysis, and provides ready-to-migrate wave plans, resulting in smooth migrations.

Enhanced security and compliance

Security remains paramount throughout the migration process. AWS Key Management Service (AWS KMS) (8, 16, 26) provides robust encryption for stored data, conversation history, and artifacts. By default, AWS managed keys are used, with the option to use customer managed keys (CMKs) for additional control.

AWS Organizations (9) enables centralized management across multiple AWS accounts, and AWS CloudTrail (14, 26) captures and logs API activities for a complete audit trail. Access control is managed through AWS Identity and Access Management (IAM) (26), providing centralized access management across AWS accounts.

Amazon CloudWatch (10, 26) continuously monitors AWS Transform service activities, resource utilization, and operational metrics within the management account, providing full visibility and control throughout the migration process. AWS Identity Center (11) further enhances security by providing centralized access management across all AWS accounts involved in the migration.

Orchestrated migration execution

When it’s time to execute the migration, AWS Transform orchestrates the end-to-end migration by coordinating across various AWS tools and services (20). The AWS Application Migration Service (25) replicates servers from your source environment to Amazon Elastic Compute Cloud (Amazon EC2) instances (21) in the AWS Migration Target Account (18), based on the carefully planned waves and groupings.

The AWS Replication Agent (2) works in tandem with AWS Application Migration Service to ensure efficient and reliable data transfer. Amazon Elastic Block Store (Amazon EBS) (21) provides the necessary storage for the migrated virtual machines, ensuring optimal performance and scalability.

Flexible network configuration

AWS Transform for VMware offers two networking models to suit different requirements:

  • Hub-and-spoke model – AWS Transit Gateway (23) connects virtual private clouds (VPCs) through a central hub VPC with shared NAT gateways. This model is ideal for centralized management and shared services.
  • Isolated model – Each VPC operates independently with no connectivity established. This approach is designed for customers with existing AWS network infrastructure, enabling you to manually connect the new VPCs to your existing network topology.

VPCs (22) created by AWS Transform match your on-premises network segments, providing a seamless transition. NAT gateways (24) provide outbound internet access for private subnets, maintaining security while enabling necessary connectivity. In hub-and-spoke architectures, centralized NAT gateways in the hub VPC can serve multiple spoke VPCs, optimizing costs and simplifying management. For isolated VPC deployments, dedicated NAT gateways must be provisioned within each VPC requiring internet access. In all cases, you must configure route tables to enable egress traffic flow through the NAT gateways

For complete setup instructions and requirements, refer to the AWS Transform User Guide.

Additional considerations

AWS Transform for VMware discovery workspaces are available globally (3). For the most up-to-date information on supported Regions, refer to AWS Services by Region (17).

Throughout the migration process, Amazon S3 buckets (12, 26) in both the AWS Migration Discovery Account and AWS Migration Target Account store key migration artifacts. These include inventory data, dependency mappings, wave plans, and application groupings, as well as Infrastructure as Code templates (AWS CloudFormation and AWS Cloud Development Kit) and per-wave migration plans.

Customers Benefits

AWS Transform for VMware delivers significant advantages:

  • Reduced manual effort – It minimizes human error and frees up valuable IT resources through automation
  • Enhanced accuracy – You can use AI-driven dependency mapping and wave planning for optimal migration strategies
  • Improved collaboration – Centralized management and tracking foster better cross-team coordination
  • Cost optimization – You can right-size instances and take advantage of AWS’s flexible pricing models for immediate and long-term savings
  • Future-proofing – It opens up the opportunity for ongoing modernization and innovation on the AWS Cloud platform

Always review and follow your organization’s security requirements, compliance obligations, and AWS security best practices when implementing any migration solution. For detailed security guidance, consult the AWS Security Documentation and your organization’s security team.

Pricing

AWS Transform accelerates migration and modernization projects for VMware workloads with agentic AI capabilities. Currently, we offer our core features—including assessment and transformation—at no cost* to AWS customers. This allows you to speed up your migration and modernization journey without upfront expenses.

*No cost refers to the AWS Transform service itself. Standard charges apply for AWS services and resources used during migrations.

Summary and Next Steps

AWS Transform for VMware empowers organizations to overcome the complexities of VMware migration and modernization. By providing a comprehensive, automated approach, it enables faster, more reliable transitions to the AWS Cloud. This new service offers the tools and capabilities needed to navigate the changing VMware landscape confidently.

The architecture we explored demonstrates how AWS Transform for VMware tackles key challenges:

  • Streamlines discovery and assessment processes
  • Automates network conversion and intelligent wave planning
  • Orchestrates migration execution with minimal disruption
  • Enhances security and compliance throughout the migration
  • Provides centralized management and monitoring
  • Offers flexible networking options to suit diverse requirements

Ready to accelerate your VMware migration journey? Visit the AWS Transform for VMware product page to learn more and get started today. Check out the following interactive demo of AWS Transform for VMware. If you’re exporting your network configuration from a VMware NSX environment, also refer to Exporting network configuration data with Import/Export for NSX. Our team of experts is ready to guide you through your migration and modernization initiatives, helping you unlock the full potential of the AWS Cloud.

About the authors

Build conversational AI search with Amazon OpenSearch Service

Post Syndicated from Bharav Patel original https://aws.amazon.com/blogs/big-data/build-conversational-ai-search-with-amazon-opensearch-service/

Retrieval Augmented Generation (RAG) is a well-known approach to creating generative AI applications. RAG combines large language models (LLMs) with external world knowledge retrieval and is increasingly popular for adding accuracy and personalization to AI. It retrieves relevant information from external sources, augments the input with this data, and generates responses based on both. This approach reduces hallucinations, improves fact accuracy, and allows for up-to-date, efficient, and explainable AI systems. RAG’s ability to break through classical language model limitations has made it applicable to broad AI use cases.

Amazon OpenSearch Service is a versatile search and analytics tool. It is capable of performing security analytics, searching data, analyzing logs, and many other tasks. It can also work with vector data with a k-nearest neighbors (k-NN) plugin, which makes it helpful for more complex search strategies. Because of this feature, OpenSearch Service can serve as a knowledge base for generative AI applications that integrate language generation with search results.

By preserving context over several exchanges, honing responses, and providing a more seamless user experience, conversational search enhances RAG. It helps with complex information needs, resolves ambiguities, and manages multi-turn reasoning. Conversational search provides a more natural and personalized interaction, yielding more accurate and pertinent results, even though standard RAG performs well for single queries.

In this post, we explore conversational search, its architecture, and various ways to implement it.

Solution overview

Let’s walk through the solution to build conversational search. The following diagram illustrates the solution architecture.

The new OpenSearch feature known as agents and tools is used to create conversational search. To develop sophisticated AI applications, agents coordinate a variety of machine learning (ML) tasks. Every agent has a number of tools; each intended for a particular function. To use agents and tools, you need OpenSearch version 2.13 or later.

Prerequisites

To implement this solution, you need an AWS account. If you don’t have one, you can create an account. You also need an OpenSearch Service domain with OpenSearch version 2.13 or later. You can use an existing domain or create a new domain.

To use the Amazon Titan Text Embedding and Anthropic Claude V1 models in Amazon Bedrock, you need to enable access to these foundation models (FMs). For instructions, refer to Add or remove access to Amazon Bedrock foundation models.

Configure IAM permissions

Complete the following steps to set up an AWS Identity and Access Management (IAM) role and user with appropriate permissions:

  1. Create an IAM role with the following policy that will allow the OpenSearch Service domain to invoke the Amazon Bedrock API: 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "bedrock:InvokeAgent",
                "bedrock:InvokeModel"
            ],
            "Resource": [
                "arn:aws:bedrock:${Region}::foundation-model/amazon.titan-embed-text-v1",
                "arn:aws:bedrock: ${Region}::foundation-model/anthropic.claude-instant-v1"
            ]
        }
    ]
}

Depending on the AWS Region and model you use, specify those in the Resource section.

  1. Add opensearchservice.amazonaws.com as a trusted entity.
  2. Make a note of the IAM role Amazon Resource name (ARN).
  3. Assign the preceding policy to the IAM user that will create a connector.
  4. Create a passRole policy and assign it to IAM user that will create the connector using Python: 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::${AccountId}:role/OpenSearchBedrock"
        }
    ]
}
  5. Map the IAM role you created to the OpenSearch Service domain role using the following steps:
    • Log in to the OpenSearch Dashboard and open the Security page from the navigation menu.
    • Choose Roles and select ml_all_access.
    • Choose Mapped Users and Manage Mapping.
    • Under Users, add the ARN of the IAM user you created.

Establish a connection to the Amazon Bedrock model using the MLCommons plugin

In order to identify patterns and relationships, an embedding model transforms input data—such as words or images—into numerical vectors in a continuous space. Similar objects are grouped together to make it easier for AI systems to comprehend and respond to intricate user enquiries.

Semantic search concentrates on the purpose and meaning of a query. OpenSearch stores data in a vector index for retrieval and transforms it into dense vectors (lists of numbers) using text embedding models. We are using amazon.titan-embed-text-v1 hosted on Amazon Bedrock, but you will need to evaluate and choose the right model for your use case. The amazon.titan-embed-text-v1 model maps sentences and paragraphs to a 1,536-dimensional dense vector space and is optimized for the task of semantic search.

Complete the following steps to establish a connection to the Amazon Bedrock model using the MLCommons plugin:

  1. Establish a connection by using the Python client with the connection blueprint.
  2. Modify the values of the host and region parameters in the provided code block. For this example, we’re running the program in Visual Studio Code with Python version 3.9.6, but newer versions should also work.
  3. For the role ARN, use the ARN you created earlier, and run the following script using the credentials of the IAM user you created: 
    import boto3
import requests 
from requests_aws4auth import AWS4Auth

host = 'https://search-test.us-east-1.es.amazonaws.com/'
region = 'us-east-1'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

path = '_plugins/_ml/connectors/_create'
url = host + path

payload = {
  "name": "Amazon Bedrock Connector: embedding",
  "description": "The connector to bedrock Titan embedding model",
  "version": 1,
  "protocol": "aws_sigv4",
  "parameters": {
    "region": "us-east-1",
    "service_name": "bedrock",
    "model": "amazon.titan-embed-text-v1"
  },
  "credential": {
    "roleArn": "arn:aws:iam::<accountID>:role/opensearch_bedrock_external"
  },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
      "headers": {
        "content-type": "application/json",
        "x-amz-content-sha256": "required"
      },
      "request_body": "{ \"inputText\": \"${parameters.inputText}\" }",
      "pre_process_function": "connector.pre_process.bedrock.embedding",
      "post_process_function": "connector.post_process.bedrock.embedding"
    }
  ]
}

headers = {"Content-Type": "application/json"}

r = requests.post(url, auth=awsauth, json=payload, headers=headers, timeout=15)
print(r.status_code)
print(r.text)
  4. Run the Python program. This will return connector_id. 
    python3 connect_bedrocktitanembedding.py
200
{"connector_id":"nbBe65EByVCe3QrFhrQ2"}
  5. Create a model group against which this model will be registered in the OpenSearch Service domain: 
    POST /_plugins/_ml/model_groups/_register
{
  "name": "embedding_model_group",
  "description": "A model group for bedrock embedding models"
}

    You get the following output:

    {
  "model_group_id": "1rBv65EByVCe3QrFXL6O",
  "status": "CREATED"
}
  6. Register a model using connector_id and model_group_id: 
    POST /_plugins/_ml/models/_register
{
    "name": "titan_text_embedding_bedrock",
    "function_name": "remote",
    "model_group_id": "1rBv65EByVCe3QrFXL6O",
    "description": "test model",
    "connector_id": "nbBe65EByVCe3QrFhrQ2",
   "interface": {}
}

You get the following output:

{
  "task_id": "2LB265EByVCe3QrFAb6R",
  "status": "CREATED",
  "model_id": "2bB265EByVCe3QrFAb60"
}
  1. Deploy a model using the model ID:
POST /_plugins/_ml/models/2bB265EByVCe3QrFAb60/_deploy

You get the following output:

{
  "task_id": "bLB665EByVCe3QrF-slA",
  "task_type": "DEPLOY_MODEL",
  "status": "COMPLETED"
}

Now the model is deployed, and you will see that in OpenSearch Dashboards on the OpenSearch Plugins page.

Create an ingestion pipeline for data indexing

Use the following code to create an ingestion pipeline for data indexing. The pipeline will establish a connection to the embedding model, retrieve the embedding, and then store it in the index.

PUT /_ingest/pipeline/cricket_data_pipeline {
    "description": "batting score summary embedding pipeline",
    "processors": [
        {
            "text_embedding": {
                "model_id": "GQOsUJEByVCe3QrFfUNq",
                "field_map": {
                    "cricket_score": "cricket_score_embedding"
                }
            }
        }
    ]
}

Create an index for storing data

Create an index for storing data (for this example, the cricket achievements of batsmen). This index stores raw text and embeddings of the summary text with 1,536 dimensions and uses the ingest pipeline we created in the previous step.

PUT cricket_data {
    "mappings": {
        "properties": {
            "cricket_score": {
                "type": "text"
            },
            "cricket_score_embedding": {
                "type": "knn_vector",
                "dimension": 1536,
                "space_type": "l2",
                "method": {
                    "name": "hnsw",
                    "engine": "faiss"
                }
            }
        }
    },
    "settings": {
        "index": {
            "knn": "true"
        }
    }
}

Ingest sample data

Use the following code to ingest the sample data for four batsmen:

POST _bulk?pipeline=cricket_data_pipeline
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Sachin Tendulkar, often hailed as the 'God of Cricket,' amassed an extraordinary batting record throughout his 24-year international career. In Test cricket, he played 200 matches, scoring a staggering 15,921 runs at an average of 53.78, including 51 centuries and 68 half-centuries, with a highest score of 248 not out. His One Day International (ODI) career was equally impressive, spanning 463 matches where he scored 18,426 runs at an average of 44.83, notching up 49 centuries and 96 half-centuries, with a top score of 200 not out – the first double century in ODI history. Although he played just one T20 International, scoring 10 runs, his overall batting statistics across formats solidified his status as one of cricket's all-time greats, setting numerous records that stand to this day."}
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Virat Kohli, widely regarded as one of the finest batsmen of his generation, has amassed impressive statistics across all formats of international cricket. As of April 2024, in Test cricket, he has scored over 8,000 runs with an average exceeding 50, including numerous centuries. His One Day International (ODI) record is particularly stellar, with more than 12,000 runs at an average well above 50, featuring over 40 centuries. In T20 Internationals, Kohli has maintained a high average and scored over 3,000 runs. Known for his exceptional ability to chase down targets in limited-overs cricket, Kohli has consistently ranked among the top batsmen in ICC rankings and has broken several batting records throughout his career, cementing his status as a modern cricket legend."}
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Adam Gilchrist, the legendary Australian wicketkeeper-batsman, had an exceptional batting record across formats during his international career from 1996 to 2008. In Test cricket, Gilchrist scored 5,570 runs in 96 matches at an impressive average of 47.60, including 17 centuries and 26 half-centuries, with a highest score of 204 not out. His One Day International (ODI) record was equally remarkable, amassing 9,619 runs in 287 matches at an average of 35.89, with 16 centuries and 55 half-centuries, and a top score of 172. Gilchrist's aggressive batting style and ability to change the course of a game quickly made him one of the most feared batsmen of his era. Although his T20 International career was brief, his overall batting statistics, combined with his wicketkeeping skills, established him as one of cricket's greatest wicketkeeper-batsmen."}
{"index": {"_index": "cricket_data"}}
{"cricket_score": "Brian Lara, the legendary West Indian batsman, had an extraordinary batting record in international cricket during his career from 1990 to 2007. In Test cricket, Lara amassed 11,953 runs in 131 matches at an impressive average of 52.88, including 34 centuries and 48 half-centuries. He holds the record for the highest individual score in a Test innings with 400 not out, as well as the highest first-class score of 501 not out. In One Day Internationals (ODIs), Lara scored 10,405 runs in 299 matches at an average of 40.48, with 19 centuries and 63 half-centuries. His highest ODI score was 169. Known for his elegant batting style and ability to play long innings, Lara's exceptional performances, particularly in Test cricket, cemented his status as one of the greatest batsmen in the history of the game."}

Deploy the LLM for response generation

Use the following code to deploy the LLM for response generation. Modify the values of host, region, and roleArn in the provided code block.

  1. Create a connector by running the following Python program. Run the script using the credentials of the IAM user created earlier. 
    import boto3
import requests 
from requests_aws4auth import AWS4Auth

host = 'https://search-test.us-east-1.es.amazonaws.com/'
region = 'us-east-1'
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

path = '_plugins/_ml/connectors/_create'
url = host + path

payload = {
  "name": "BedRock Claude instant-v1 Connector ",
  "description": "The connector to BedRock service for claude model",
  "version": 1,
  "protocol": "aws_sigv4",
  "parameters": {
    "region": "us-east-1",
    "service_name": "bedrock",
    "anthropic_version": "bedrock-2023-05-31",
    "max_tokens_to_sample": 8000,
    "temperature": 0.0001,
    "response_filter": "$.completion"
  },
   "credential": {
        "roleArn": "arn:aws:iam::accountId:role/opensearch_bedrock_external"
    },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/anthropic.claude-instant-v1/invoke",
      "headers": {
        "content-type": "application/json",
        "x-amz-content-sha256": "required"
      },
      "request_body": "{\"prompt\":\"${parameters.prompt}\", \"max_tokens_to_sample\":${parameters.max_tokens_to_sample}, \"temperature\":${parameters.temperature},  \"anthropic_version\":\"${parameters.anthropic_version}\" }"
    }
  ]
 }
    

headers = {"Content-Type": "application/json"}

r = requests.post(url, auth=awsauth, json=payload, headers=headers, timeout=15)
print(r.status_code)
print(r.text)

If it ran successfully, it would return connector_id and a 200-response code:

200
{"connector_id":"LhLSZ5MBLD0avmh1El6Q"}
  1. Create a model group for this model: 
    POST /_plugins/_ml/model_groups/_register
{
    "name": "claude_model_group",
    "description": "This is an example description"
}

This will return model_group_id; make a note of it:

{
  "model_group_id": "LxLTZ5MBLD0avmh1wV4L",
  "status": "CREATED"
}
  1. Register a model using connection_id and model_group_id: 
    POST /_plugins/_ml/models/_register
{
    "name": "anthropic.claude-v1",
    "function_name": "remote",
    "model_group_id": "LxLTZ5MBLD0avmh1wV4L",
    "description": "LLM model",
    "connector_id": "LhLSZ5MBLD0avmh1El6Q",
    "interface": {}
}

It will return model_id and task_id:

{
  "task_id": "YvbVZ5MBtVAPFbeA7ou7",
  "status": "CREATED",
  "model_id": "Y_bVZ5MBtVAPFbeA7ovb"
}
  1. Finally, deploy the model using an API:
POST /_plugins/_ml/models/Y_bVZ5MBtVAPFbeA7ovb/_deploy

The status will show as COMPLETED. That means the model is successfully deployed.

{
  "task_id": "efbvZ5MBtVAPFbeA7otB",
  "task_type": "DEPLOY_MODEL",
  "status": "COMPLETED"
}

Create an agent in OpenSearch Service

An agent orchestrates and runs ML models and tools. A tool performs a set of specific tasks. For this post, we use the following tools:

  • VectorDBTool – The agent use this tool to retrieve OpenSearch documents relevant to the user question
  • MLModelTool – This tool generates user responses based on prompts and OpenSearch documents

Use the embedding model_id in VectorDBTool and LLM model_id in MLModelTool:

POST /_plugins/_ml/agents/_register {
    "name": "cricket score data analysis agent",
    "type": "conversational_flow",
    "description": "This is a demo agent for cricket data analysis",
    "app_type": "rag",
    "memory": {
        "type": "conversation_index"
    },
    "tools": [
        {
            "type": "VectorDBTool",
            "name": "cricket_knowledge_base",
            "parameters": {
                "model_id": "2bB265EByVCe3QrFAb60",
                "index": "cricket_data",
                "embedding_field": "cricket_score_embedding",
                "source_field": [
                    "cricket_score"
                ],
                "input": "${parameters.question}"
            }
        },
        {
            "type": "MLModelTool",
            "name": "bedrock_claude_model",
            "description": "A general tool to answer any question",
            "parameters": {
                "model_id": "gbcfIpEByVCe3QrFClUp",
                "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\nContext:\n${parameters.cricket_knowledge_base.output:-}\n\n${parameters.chat_history:-}\n\nHuman:${parameters.question}\n\nAssistant:"
            }
        }
    ]
}

This returns an agent ID; take note of the agent ID, which will be used in subsequent APIs.

Query the index

We have batting scores of four batsmen in the index. For the first query, let’s specify the player name:

POST /_plugins/_ml/agents/<agent ID>/_execute {
    "parameters": {
        "question": "What is batting score of Sachin Tendulkar ?"
    }
}

Based on context and available information, it returns the batting score of Sachin Tendulkar. Note the memory_id from the response; you will need it for subsequent questions in the next steps.

We can ask a follow-up question. This time, we don’t specify the player name and expect it to answer based on the earlier question:

POST /_plugins/_ml/agents/<agent ID>/_execute {
    "parameters": {
        "question": " How many T20 international match did he play?",
        "next_action": "then compare with Virat Kohlis score",
        "memory_id": "so-vAJMByVCe3QrFYO7j",
        "message_history_limit": 5,
        "prompt": "\n\nHuman:You are a professional data analysist. You will always answer question based on the given context first. If the answer is not directly shown in the context, you will analyze the data and find the answer. If you don't know the answer, just say don't know. \n\nContext:\n${parameters.population_knowledge_base.output:-}\n\n${parameters.chat_history:-}\n\nHuman:always learn useful information from chat history\nHuman:${parameters.question}, ${parameters.next_action}\n\nAssistant:"
    }
}

In the preceding API, we use the following parameters:

  • Question and Next_action – We also pass the next action to compare Sachin’s score with Virat’s score.
  • Memory_id – This is memory assigned to this conversation. Use the same memory_id for subsequent questions.
  • Prompt – This is the prompt you give to the LLM. It includes the user’s question and the next action. The LLM should answer only using the data indexed in OpenSearch and must not invent any information. This way, you prevent hallucination.

Refer to ML Model tool for more details about setting up these parameters and the GitHub repo for blueprints for remote inferences.

The tool stores the conversation history of the questions and answers in the OpenSearch index, which is used to refine answers by asking follow-up questions.

In real-world scenarios, you can map memory_id against the user’s profile to preserve the context and isolate the user’s conversation history.

We have demonstrated how to create a conversational search application using the built-in features of OpenSearch Service.

Clean up

To avoid incurring future charges, delete the resources created while building this solution:

  1. Delete the OpenSearch Service domain.
  2. Delete the connector.
  3. Delete the index.

Conclusion

In this post, we demonstrated how to use OpenSearch agents and tools to create a RAG pipeline with conversational search. By integrating with ML models, vectorizing questions, and interacting with LLMs to improve prompts, this configuration oversees the entire process. This method allows you to quickly develop AI assistants that are ready for production without having to start from scratch.

If you’re building a RAG pipeline with conversational history to let users ask follow-up questions for more refined answers, give it a try and share your feedback or questions in the comments!

About the author

Bharav Patel is a Specialist Solution Architect, Analytics at Amazon Web Services. He primarily works on Amazon OpenSearch Service and helps customers with key concepts and design principles of running OpenSearch workloads on the cloud. Bharav likes to explore new places and try out different cuisines.

Simplifying sustainability reporting using AWS and generative AI in banking

Post Syndicated from Sachin Kulkarni original https://aws.amazon.com/blogs/architecture/simplifying-sustainability-reporting-using-aws-and-generative-ai-in-banking/

European banks face a new challenge with the European Commission’s transition from the Non-Financial Reporting Directive (NFRD) to the Corporate Sustainability Reporting Directive (CSRD) regulations. This transition represents an expansion in sustainability reporting scope that will affect approximately 50,000 companies, a significant increase from the previous 11,700.

This means that banks themselves need to file sustainability reports because they will now be one of those 50,000 companies, but for their own reporting, they also need to assess their clients’ sustainability reports because they lend or finance those companies.

In this post, you learn how you can use generative AI services on Amazon Web Services (AWS) to automate your sustainability reporting requirements, reduce manual effort, and improve accuracy. You do this by implementing an automated solution for extracting, processing, and validating data from corporate reports.

The challenge

Financial institutions and sustainability teams managing sustainability reporting face three critical challenges:

  • Scale and complexity: Banks and financial institutions must process thousands of annual reports and sustainability documents, often spanning hundreds of pages each. This process requires extensive data extraction, complex EU Taxonomy alignment calculations, and resource-intensive validation steps. Manual processing introduces significant risks of errors and consumes valuable team resources.
  • Regulatory compliance: Banks must now implement detailed CSRD requirements, track specific metrics for turnover, capital expenditure (CapEx), and operating expenses (OpEx), and calculate their Green Asset Ratio (GAR) as well as environmental risks that come with their loans, debt, or equity investments. These new requirements demand robust data collection and processing capabilities.
  • Data management: Processing Green House Gas (GHG) emissions data across Scope 1, 2, and 3 categories requires analyzing complex lending and investment activities. With strict reporting deadlines, organizations need efficient tools to process this expanding volume of sustainability data.

The sustainability team point of view

Banks finance a large variety of counterparties and economic activities. While their carbon footprint is primarily linked to the greenhouse gas (GHG) emissions of their counterparties (Scope 3), The direct GHG emissions (Scope 1) of financial institutions or the GHG emissions linked to their energy consumption (Scope 2) are usually limited. For banks, the most critical key performance indicator (KPI) is the GAR, which measures the proportion of a bank’s taxonomy-aligned balance sheet exposures versus its total eligible exposures, as shown in the following figure.

To calculate their GAR, banks must obtain and use sustainability data from annual reports or sustainability reports of up to 50,000 companies (many of which are subject to NFRD and CSRD reporting), and understand how much of their activities are linked to EU Taxonomy.

The manual process

In the example that follows, we use the Amazon 2023 Annual Report. Some of the data that teams would have to manually extract includes: Revenue, Scope 1, Scope 2, and Scope 3 emissions.

Amazon Annual Report

As you can see from the page count at the top of the preceding figure, people manually searching for this data would have to go through 92 pages to find the parameters they’re looking for. Next, we might determine that some of the data we need (Scope 1, Scope 2, Scope 3) isn’t available in the annual report, so we need to analyze the sustainability report. As shown in the following figure, to manually retrieve the relevant data from this report, we would have to go through 98 pages of information.

amazon sustainability report

To prepare a GAR, we would have to repeat this process across hundreds or even thousands of companies.

A solution using AWS and generative AI

To address these challenges, we propose an automated approach using AWS services. This approach can help banks streamline their sustainability reporting processes.

high level flow

Here’s how this solution works— as shown in the preceding figure:

  1. Upload your counterparties’ reports to Amazon Simple Storage Service (Amazon S3).
  2. Amazon Bedrock automatically:
    1. Determines NFRD eligibility.
    2. Extracts relevant sustainability data.
    3. Organizes information for GAR calculations.
  3. Review and validate the extracted data.
  4. Generate required regulatory reports.

Architecture

We divide the architecture into two areas:

  1. Data ingestion flow
  2. Report generation flow

Data ingestion flow

We use Amazon Bedrock Knowledge Bases to build an automated data ingestion flow. See Prerequisites for your Amazon Bedrock knowledge base data to understand supported document formats and limits for knowledge base data.

Data Ingestion flow

The workflow, shown in the preceding figure, is:

  1.  Annual reports or sustainability reports are uploaded into an S3 bucket.
  2. On the S3 bucket, we enable event notifications for events such as addition, change, or deletion of the reports.
  3. These events are sent to Amazon Event Bridge, which trigger an AWS Lambda function.
  4. The Lambda function syncs the data source to an Amazon Bedrock knowledge base.
  5. Amazon Bedrock Knowledge Bases processes the documents and converts it into vector embeddings. For more information, see Amazon Bedrock Knowledge Bases supports advanced parsing, chunking, and query reformulation giving greater control of accuracy in RAG based applications
  6. Amazon Bedrock Knowledge Bases stores the vector embeddings in the vector database of your choice, such as in an Amazon OpenSearch Serverless collection.

Now the data is read, broken into chunks, converted to embeddings and stored in a vector store. You use a report generation flow to ask questions about the information in the knowledge base.

Report generation flow

To automate the report generation for sustainability teams, we created the report generation flow shown in the following figure.

report generation flow

The report generation flow includes the following steps:

  1. When user uploads an annual report, the data from the report is ingested into the knowledge base, as shown in the data ingestion flow.
  2. A Lambda function—Invoke Bedrock Agent—is triggered to invoke an Amazon Bedrock agent.
  3. The Amazon Bedrock agent determines NFRD or CSRD applicability based on various parameters such as employee numbers and annual revenues. This agent then passes on what kind of regulation to apply to a Lambda function.
  4. The Lambda function Retrieve Sustainability Metrics retrieves various parameters needed for NFRD or CSRD from the annual report.
    1. The function receives NFRD or CSRD applicability from the Amazon Bedrock agent.
    2. Based on NFRD or CSRD applicability, there are specific sustainability metrics that need to be retrieved. For NFRD, there are about 15 metrics that need to be retrieved, and for CSRD, there are about 30 metrics.
    3.  The function iteratively sends {variable} to the Amazon Bedrock flow. For example, if the metric to be retrieved is Scope 1 emission, then the Lambda function will send variable=‘Scope 1 emission’
    4. The function gets the metric value from the Amazon Bedrock flow and when the required metrics are retrieved, creates a CSV file with the details.
  5. Amazon Bedrock flow:
    1. Retrieve {variable} (for example, ‘Scope 1 emission’) from the annual report. For this, we create a prompt, as shown in the following diagram.
    2. Use the prompt to fetch the value from the knowledge base.
        • Prompt: 
          <query> You are an intelligent agent that helps retrieve information from a knowledgebase. Please find {{variable}}. Please return only a number and not any additional text. I only need the value so you will return one word</query>

    3. Return the value to the Lambda function in Step 4.

Breakdown of key components

Amazon S3 is used for storing annual statements and sustainability reports, providing highly durable and secure object storage that facilitate immediate access when needed for processing.

Amazon Bedrock Knowledge Bases enables using Retrieval-Augmented Generation (RAG) to optimize the output of a large language model by giving it the context of companies’ annual reports and regulatory requirements. It does so by creating chunks and vector embeddings from the annual reports to enable efficient information retrieval from a vector database of your choice.

Amazon Bedrock foundation models (FMs) extract information from an Amazon Bedrock knowledge base and generate standardized PDF reports for regulators, providing consistent formatting and alignment with CSRD requirements. We encourage you to choose the best foundational model for your use case through the flexibility and enterprise-grade controls of Amazon Bedrock. For this solution, we used Anthropic’s Claude Sonnet 3.5 as the model, but by using Amazon Bedrock, you can choose from over 50 different models to see which one best fits your use case.

Amazon Bedrock Flows orchestrates the document processing pipeline, coordinating between services to automatically extract required sustainability metrics and validate compliance requirements. This feature helps us manage the workflow from initial document ingestion through to final report generation.

Amazon Bedrock Prompt Management creates and helps manage precise prompts that help retrieve multiple sustainability metrics from reports for example: turnover, Scope 1, Scope 2, and Scope 3 emissions data. These structured prompts facilitate consistent data extraction across different document formats.

Amazon Bedrock Agents evaluates each uploaded document to determine NFRD or CSRD eligibility by analyzing company revenue, employee count, and incorporation details. The agents retrieve these parameters by using a Lambda function that’s part of the actions the agent can perform.

Lambda handles event-driven processing when new documents are uploaded. Lambda functions are also used by the agent to retrieve data from companies’ annual reports and trigger the appropriate workflows based on document type.

Amazon EventBridge is used to build event-driven applications at scale across AWS and manages workflow orchestration, automatically initiating document processing when new reports are uploaded through S3 event notifications.

This architecture enables banks to process thousands of sustainability reports efficiently. The solution scales automatically to handle increasing document volumes while keeping security a top priority.

Additional considerations

You can use the following additional AWS service to help further increase the accuracy of information retrieval from sustainability documents.

Amazon Bedrock Guardrails to make sure that the solution caters to responsible AI policies. Specifically, we have added contextual grounding checks to reduce hallucinations. This is important for the solution because we’re trying to find a few specific values in a large document, and these checks make sure that the metrics retrieved are based on the documents.

Automated reasoning checks which help to verify the metrics returned by the solution. Consider the metric Number of employees. There can be multiple places in the annual report where the number of employees is mentioned; for example, temporary workers, part-time employees, employees from various departments, employees from a company that was taken over last year, and so on. To arrive at the right number, automated reasoning checks help.

Benefits

This sustainability reporting solution cuts document processing time from 8—10 weeks to few hours. Banks get clear audit trails showing exactly how they extracted and validated sustainability data. When regulations are updated, the system adapts through its knowledge base without disrupting operations. Built-in security protects company data through the entire process. Access controls and encryption are in place to secure information. The output delivers standardized, accurate reports. This automation lets sustainability teams concentrate on environmental improvements rather than paperwork. Teams can instead analyze trends and develop initiatives instead of hunting through reports for data points.

Conclusion

As sustainability reporting requirements evolve, having a flexible and automated solution will become crucial. While we focused on NFRD reporting, the same pattern can be adapted for CSRD compliance reporting, SFDR reporting requirements, and Internal sustainability metrics, or EU Taxonomy alignment.

Customers looking to build their products in the Financial Services industry have access to industry and domain AWS specialists; contact us for help in your cloud journey.

You can also learn more about AWS services and solutions for financial services by visiting AWS for Financial Services and Generative AI on AWS.

About the authors

Amazon Bedrock baseline architecture in an AWS landing zone

Post Syndicated from Abdel-Rahman Awad original https://aws.amazon.com/blogs/architecture/amazon-bedrock-baseline-architecture-in-an-aws-landing-zone/

As organizations increasingly adopt Amazon Bedrock to build and deploy large-scale AI applications, it’s important that they understand and adopt critical network access controls to protect their data and workloads. These generative AI-enabled applications might have access to sensitive or confidential information within their knowledge bases, Retrieval Augmented Generation (RAG) data sources, or models themselves, which could pose a risk if exposed to unauthorized parties. Additionally, organizations might want to limit access to certain AI models to specific teams or services, making sure only authorized users can use the most powerful capabilities. Another important consideration is cost optimization, because organizations need to be able to monitor and control access to manage various aspects of their cloud spending.

In this post, we explore the Amazon Bedrock baseline architecture and how you can secure and control network access to your various Amazon Bedrock capabilities within AWS network services and tools. We discuss key design considerations, such as using Amazon VPC Lattice auth policies, Amazon Virtual Private Cloud (Amazon VPC) endpoints, and AWS Identity and Access Management (IAM) to restrict and monitor access to your Amazon Bedrock capabilities.

By the end of this post, you will have a better understanding of how to configure your AWS landing zone to establish secure and controlled network connectivity to Amazon Bedrock across your organization using VPC Lattice.

Solution overview

Addressing the aforementioned challenges requires a well-designed network architecture and security controls. For this, we use the standard AWS Landing Zone Accelerator networking configuration. It provides a good starting point for managing network communication across multiple accounts. On top of the AWS Landing Zone Accelerator network design, we add two shared accounts.

In this solution design, we create a centralized architecture for managing organization AI capabilities across different accounts. The architecture consists of three main parts that work together to provide secure and controlled access to AI services:

  • Service network account – This account serves as the central networking hub for the organization, managing network connectivity and access policies. Through this account, network administrators can centrally manage and control access to AI services across the organization. The account follows AWS Landing Zone Accelerator networking practices that scale with enterprise organizational needs.
  • Generative AI account – This account hosts the organization’s Amazon Bedrock capabilities and serves as the central point for AI/ML management. The organization’s AI/ML scientists and prompt engineers will centrally build and manage Amazon Bedrock capabilities. The account provides access to various large language models (LLMs) through Amazon Bedrock by using VPC interface endpoints, while also enabling centralized monitoring of cost consumption and access patterns.
  • Workload accounts (dev, test, prod) – These accounts represent different environments where teams develop and deploy applications that consume AI services. Through secure network connections established through the service network account, these workload accounts can access the AI capabilities hosted in the generative AI account. This separation enforces proper isolation between development, testing, and production workloads while maintaining secure access to AI services.
Amazon Bedrock baseline architecture in an AWS landing zone

Amazon Bedrock baseline architecture in an AWS landing zone

The following diagram illustrates the solution architecture.

The service network account has its own VPC Lattice service network—a centralized networking construct that enables service-to-service communication across your organization, which is shared with workload accounts using AWS Resource Access Manager (AWS RAM) to enable VPC Lattice Service network sharing.

Workload accounts (dev, test, prod) establish VPC associations with the shared VPC Lattice service network by creating a service network association in their VPC. When an application in these accounts makes a request, it first queries the VPC resolver for DNS resolution. The resolver routes the traffic to the VPC Lattice service network.

Access control is implemented through an VPC Lattice auth policy. The service network policies determine which accounts can access the VPC Lattice service network, and service-level policies control access to specific AI services and define what actions each account can perform.

In the central AI services account, we find the proxy layer, we create a VPC Lattice service that points to a proxy layer, which acts as a single entry point, providing workload accounts access to Amazon Bedrock. This proxy layer then connects to Amazon Bedrock through VPC endpoints. Through this setup, the AI team can configure which foundation models (FMs) are available and manage access permissions for different workload accounts. After the necessary policies and connections are in place, workload accounts can access Amazon Bedrock capabilities through the established secure pathway. This setup enables secure, cross-account access to AI services while maintaining centralized control and monitoring.

Network components

We use VPC Lattice, which is a fully managed application networking service that helps you simplify network connectivity, security, and monitoring for service-to-service communication needs. With VPC Lattice, organizations can achieve a centralized connectivity pattern to control and monitor access to the services required for building generative AI applications.

For details about VPC Lattice, refer to the Amazon VPC Lattice User Guide. The following is an overview of the constructs you can use in setting up the centralized pattern in this solution:

  • VPC Lattice service network – You can use the VPC Lattice service network to provide central connectivity and security to the central AI services account. The service network is a logical grouping mechanism that simplifies how you can enable connectivity across VPCs or accounts, and apply common security policies for application communication patterns. You can create a service network in an account and share it with other accounts within or outside AWS Organizations using AWS RAM.
  • VPC Lattice service – In a service network, you can associate a VPC Lattice service, which consists of a listener (protocol and port number), routing rules that allow for control of the application flow (for example, path, method, header-based, or weighted routing), and target group, which defines the application infrastructure. A service can have multiple listeners to meet various client capabilities. Supported protocols include HTTP, HTTPS, gRPC, and TLS. The path-based routing allows control to various high-performing FMs and other capabilities you would need to build a generative AI application.
  • Proxy layer – You use a proxy layer for the VPC Lattice service target group. The proxy layer can be built based on your organization’s preference of AWS services, such as AWS Lambda, AWS Fargate, or Amazon Elastic Kubernetes Service (Amazon EKS). The purpose of the proxy layer is to provide a single entry point to access LLMs, knowledge bases, and other capabilities that are tested and approved according to your organization’s compliance requirements.
  • VPC Lattice auth policies – For security, you use VPC Lattice auth policies. VPC Lattice auth policies are specified using the same syntax as IAM policies. You can apply an auth policy to VPC Lattice service network as well as to the VPC Lattice service.
  • Fully Qualified Domain Names –To facilitate service discovery, VPC Lattice supports custom domain names for your services and resources, and maintains a Fully Qualified Domain Name (FQDN) for each VPC Lattice service and resource you define. You can use these FQDNs in your Amazon Route 53 private hosted zone configurations, and empower business units or teams to discover and access services and resources.
  • Service network VPC – Business units or teams can access generative AI services in a service network using service network VPC associations or a service network VPC endpoint.
  • Monitoring – You can choose to enable monitoring at the VPC Lattice service network level and VPC Lattice service level. VPC Lattice generates metrics and logs for requests and responses, making it more efficient to monitor and troubleshoot applications

The preceding guidance takes a “secure by default” approach—you must be explicit about which features, models, and so on should be accessed by which business unit. The setup also enables you to implement a defense-in-depth strategy at multiple layers of the network:

  • The first level of defense is that business team needs to connect to the service network in order to get access to the generative AI service through the central AI service account.
  • The second level includes network-level security protections in the business team’s VPC for the service network, such as security groups and network access control lists (ACLs). By using these, you can allow access to specific workloads or teams in a VPC.
  • The third level is through the VPC Lattice auth policy, which you can apply at two layers: at the service network level to allow authenticated requests within the organization, and at the service level to allow access to specific models and features.

VPC Lattice auth policy

This solution makes it possible to centrally manage access to Amazon Bedrock resources across your organization. This approach uses an VPC Lattice auth policy to centrally control Amazon Bedrock resources and manage it from one location across all the organization accounts.

Typically, the auth policy on the service network is operated by the network or cloud administrator. For example, allowing only authenticated requests from specific workloads or teams in your AWS organization. In the following example, access is granted to invoke the generated AI service for authenticated requests and to principals that are part of the o-123456example organization:

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": "*",
         "Action": "vpc-lattice-svcs:Invoke",
         "Resource": "*",
         "Condition": {
            "StringEquals": {
               "aws:PrincipalOrgID": [ 
                  "o-123456example"
               ]
            }
         }
      }
   ]
}

The auth policy at the service level is managed by the central AI service team to set fine-grained controls, which can be more restrictive than the coarse-grained authorization applied at the service network level. For example, the following policy restricts access to claude-3-haiku for only business-team1:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect": "Allow",
         "Principal": {
            "AWS": [
               "arn:aws:iam::<account-number>:role/businss-team1"
            ]
         },
         "Action": "vpc-lattice-svcs:Invoke",
         "Resource": [
            "arn:aws:vpc-lattice:<aws-region>:<account-number>:service/svc-0123456789abcdef0/*"
         ],
         "Condition": {
            "StringEquals": {
               "vpc-lattice-svcs:RequestQueryString/modelid": "claude-3-haiku"            }
         }
      }
   ]
}

Monitoring and tracking

This design employs three monitoring approaches, using Amazon CloudWatch, AWS CloudTrail, and VPC Lattice access logs. This strategy provides a view of service usage, security, and performance.

CloudWatch metrics offer real-time monitoring of VPC Lattice service performance and usage. CloudWatch tracks metrics such as request counts and response times for Amazon Bedrock related endpoints, allowing for the setup of alarms for proactive management of service health and capacity. This enables monitoring of overall usage patterns of Amazon Bedrock models across different business units, facilitating capacity planning and resource allocation. CloudTrail provides detailed API-level auditing of Amazon Bedrock related actions. It logs cross-account access attempts and interactions with Amazon Bedrock services, providing a compliance and security audit trail. This tracking of who is accessing which Amazon Bedrock models, when, and from which accounts helps organizations adhere to their organizational policies.VPC Lattice access logs provide detailed insights into HTTP/HTTPS requests to Amazon Bedrock services, capturing specific usage patterns of AI models by different business teams. These logs contain client-specific information, which for example can be used to enable organizations to implement capabilities such as charge-back models. This allows for accurate attribution of AI service usage to specific teams or departments, facilitating fair cost allocation and responsible resource utilization across the organization. These services work together to enhance security, optimize performance, and provide valuable insights for managing cross-account Amazon Bedrock access.

Conclusion

In this post, we explored the importance of securing and controlling network access to Amazon Bedrock capabilities within an organization’s AWS landing zone. We discussed the key business challenges, such as the need to protect sensitive information in Amazon Bedrock knowledge bases, limit access to AI models, and optimize cloud costs by monitoring and controlling Amazon Bedrock capabilities. To address these challenges, we outlined a multi-layered network solution that uses AWS networking services, including a VPC Lattice auth policy to restrict and monitor access to Amazon Bedrock capabilities. Try out this solution for your own use case, and share your feedback in the comments.

About the authors

AWS Weekly Roundup: re:Inforce re:Cap, Valkey GLIDE 2.0, Avro and Protobuf or MCP Servers on Lambda, and more (June 23, 2025)

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-reinforce-recap-valkey-glide-2-0-avro-and-protobuf-or-mcp-servers-on-lambda-and-more-june-23-2025/

Last week’s hallmark event was the security-focused AWS re:Inforce conference.


AWS re:Inforce 2025
AWS re:Inforce 2025

Now a tradition, the blog team wrote a re:Cap post to summarize the announcements and link to some of the top blog posts.

To further summarize, several new security innovations were announced, including enhanced IAM Access Analyzer capabilities, MFA enforcement for root users, and threat intelligence integration with AWS Network Firewall. Other notable updates include exportable public SSL/TLS certificates from AWS Certificate Manager, a simplified AWS WAF console experience, and a new AWS Shield feature for proactive network security (in preview). Additionally, AWS Security Hub has been enhanced for risk prioritization (Preview), and Amazon GuardDuty now supports Amazon EKS clusters.

But my favorite announcement came from the Amazon Verified Permissions team. They released an open source package for Express.js, enabling developers to implement external fine-grained authorization for web application APIs. This simplifies authorization integration, reducing code complexity and improving application security.

The team also published a blog post that outlines how to create a Verified Permissions policy store, add Cedar and Verified Permissions authorisation middleware to your app, create and deploy a Cedar schema, and create and deploy Cedar policies. The Cedar schema is generated from an OpenAPI specification and formatted for use with the AWS Command Line Interface (CLI).

Let’s look at last week’s other new announcements.

Last week’s launches
Apart from re:Inforce, here are the launches that got my attention.

Kafka customers use Avro and Protobuf formats for efficient data storage, fast serialization and deserialization, schema evolution support, and interoperability between different programming languages. They utilize schema registries to manage, evolve, and validate schemas before data enters processing pipelines. Previously, you were required to write custom code within your Lambda function to validate, deserialize, and filter events when using these data formats. With this launch, Lambda natively supports Avro and Protobuf, as well as integration with GSR, CCSR, and SCSR. This enables you to process your Kafka events using these data formats without writing custom code. Additionally, you can optimize costs through event filtering to prevent unnecessary function invocations.

  • Amazon S3 Express One Zone now supports atomic renaming of objects with a single API call – The RenameObject API simplifies data management in S3 directory buckets by transforming a multi-step rename operation into a single API call. This means you can now rename objects in S3 Express One Zone by specifying an existing object’s name as the source and the new name as the destination within the same S3 directory bucket. With no data movement involved, this capability accelerates applications like log file management, media processing, and data analytics, while also lowering costs. For instance, renaming a 1-terabyte log file can now complete in milliseconds, instead of hours, significantly accelerating applications and reducing costs.
  • Valkey introduces GLIDE 2.0 with support for Go, OpenTelemetry, and pipeline batching – AWS, in partnership with Google and the Valkey community, announces the general availability of General Language Independent Driver for the Enterprise (GLIDE) 2.0. This is the latest release of one of AWS’s official open-source Valkey client libraries. Valkey, the most permissive open-source alternative to Redis, is stewarded by the Linux Foundation and will always remain open-source. Valkey GLIDE is a reliable, high-performance, multi-language client that supports all Valkey commands

GLIDE 2.0 introduces new capabilities that expand developer support, improve observability, and optimise performance for high-throughput workloads. Valkey GLIDE 2.0 extends its multi-language support to Go (contributed by Google), joining Java, Python, and Node.js to provide a consistent, fully compatible API experience across all four languages. More language support is on the way. With this release, Valkey GLIDE now supports OpenTelemetry, an open-source, vendor-neutral framework that enables developers to generate, collect, and export telemetry data and critical client-side performance insights. Additionally, GLIDE 2.0 introduces batching capabilities, reducing network overhead and latency for high-frequency use cases by allowing multiple commands to be grouped and executed as a single operation.

You can discover more about Valkey GLIDE in this recent episode of the AWS Developers Podcast: Inside Valkey GLIDE: building a next-gen Valkey client library with Rust.

 Podcast episode on Valkey Glide

For a full list of AWS announcements, be sure to keep an eye on the What’s New at AWS page.

Some other reading
My Belgian compatriot Alexis has written the first article of a two-part series explaining how to develop an MCP Tool server with a streamable HTTP transport and deploy it on Lambda and API Gateway. This is a must-read for anyone implementing MCP servers on AWS. I’m eagerly looking forward to the second part, where Alexis will discuss authentication and authorization for remote MCP servers.

Other AWS events
Check your calendar and sign up for upcoming AWS events.

AWS GenAI Lofts are collaborative spaces and immersive experiences that showcase AWS expertise in cloud computing and AI. They provide startups and developers with hands-on access to AI products and services, exclusive sessions with industry leaders, and valuable networking opportunities with investors and peers. Find a GenAI Loft location near you and don’t forget to register.

AWS Summits are free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Japan (this week June 25 – 26), Online in India (June 26), New-York City (July 16).

Save the date for these upcoming Summits in July and August: Taipei (July 29), Jakarta (August 7), Mexico (August 8), São Paulo (August 13), and Johannesburg (August 20) (and more to come in September and October).

Browse all upcoming AWS led in-person and virtual events here.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— seb

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Accelerate development with secure access to Amazon Q Developer using PingIdentity

Post Syndicated from Sid Vantair original https://aws.amazon.com/blogs/devops/accelerate-development-with-secure-access-to-amazon-q-developer-using-pingidentity/

 Overview

Customers adopting Amazon Q Developer, a generative AI-powered coding companion, often need authentication through existing identity providers like PingIdentity. By leveraging AWS IAM Identity Center, organizations can enable their developers to access Amazon Q Developer with their existing PingIdentity credentials, streamlining authentication and removing the need for separate login procedures. Amazon Q Developer can chat about code, provide inline code completions, and generate new code. It also scans your code for security vulnerabilities and makes code improvements, including language updates, debugging, and optimizations. Amazon Q Developer comes in two tiers. The Free Tier is available at no cost for individual use. The Pro Tier is a paid version offering enterprise access controls, an analytics dashboard, customization, and higher usage limits. Organizations that enable the Pro tier of Amazon Q Developer for their developers typically authenticate with AWS IAM Identity Center. This approach is popular due to its ability to federate with external identity providers. In this blog, we will show you how to set up PingIdentity as an external IdP for IAM Identity Center and allow developers to access Amazon Q Developer using their existing PingIdentity login credentials.

How it works

AWS authentication flow diagram: Developers interact with Amazon Q Developer and AWS IAM Identity Center, integrating with Ping Identity for SAML-based access.

Figure 1 – Solution Overview

The authentication workflow is as follows:

  1. The developer initiates an access request to Amazon Q Developer.
  2. IAM Identity center checks authentication status.
  3. If not authenticated, redirects to PingIdentity login.
  4. Developer provides PingIdentity Credentials.
  5. PingIdentity validates credentials and sends SAML response.
  6. IAM Identity Center verifies the SAML response.
  7. Upon successful verification, grants Amazon Q Developer access.
  8. Developer begins using Amazon Q Developer.

Prerequisites

  • AWS account
  • PingIdentity environment with users and groups already setup for Amazon Q Developer access
  • IAM identity center
  • Pro Tier subscription of Amazon Q Developer

Walkthrough

In this section, we demonstrate how to create a SAML-based connection between PingIdentity and IAM Identity Center, enabling you to access Amazon Q Developer seamlessly using your PingIdentity credentials.

Note: You will need to switch between PingIdentity portal and IAM Identity Center in your browser. We recommend opening a new browser tab for each console.

Step 1: Enable AWS Single Sign-On in PingIdentity

This step involves enabling AWS Single Sign-On application within PingIdentity.

    1. In the PingIdentity console, Navigate to the Applications Tab > Application Catalog
    2. Browse catalog for AWS Single Sign-On and select + to start the Quick Setup.
Screenshot of the PingIdentity Application Catalog interface. The search term "aws" is entered in the search bar, displaying three results: Amazon Web Services – AWS, AWS Gov-Cloud, and AWS Single Sign-On. The "AWS Single Sign-On" option is outlined with a red box and includes a plus button to add the application

Figure 2 – PingIdentity Application Catalog

Alt Text: Screenshot of the PingIdentity Application Catalog interface. The search term “aws” is entered in the search bar, displaying three results: Amazon Web Services – AWS, AWS Gov-Cloud, and AWS Single Sign-On. The “AWS Single Sign-On” option is outlined with a red box and includes a plus button to add the application

    1. Provide Name, SSO Region and SSO Tenant ID and choose Next
      • Name – Input an appropriate name for the connection
      • SSO Region – Input the appropriate region
      • Tenant ID – Identity Store ID
        You can run the following CLI command to retrieve the value. It’s a 10-digit alphanumeric prefixed by “d-“.
aws sso-admin list-instances –query ‘Instances[0].IdentityStoreId’Output: “d-XXXXXXXXXX”
    1. Navigate to PingOne Mappings and select Email Address from the drop down.
Screenshot of the AWS Single Sign-On configuration in PingIdentity. The screen shows Step 2 of the setup process where the SAML attribute SAML_SUBJECT is mapped to the PingOne attribute "Email Address". A red box highlights the mapping section under "PingOne Mappings".

Figure 3 – AWS Single Sign-On attribute mapping

Alt Text: Screenshot of the AWS Single Sign-On configuration in PingIdentity. The screen shows Step 2 of the setup process where the SAML attribute SAML_SUBJECT is mapped to the PingOne attribute “Email Address”. A red box highlights the mapping section under “PingOne Mappings”.

    1. Search and select the group that you have created earlier for enabling access to Amazon Q Developer and select + to add the group.
    2. Choose Save
Screenshot of Step 3 in the AWS Single Sign-On setup process in PingIdentity. The screen shows the group selection interface where the "Amazon Q" group is listed. A plus icon is shown next to the group to add it, and a blue "Save" button is highlighted in the bottom-right corner to confirm the configuration.

Figure 4 – Select PingIdentity directory Groups for Amazon Q Developer access

Alt Text: Screenshot of Step 3 in the AWS Single Sign-On setup process in PingIdentity. The screen shows the group selection interface where the “Amazon Q” group is listed. A plus icon is shown next to the group to add it, and a blue “Save” button is highlighted in the bottom-right corner to confirm the configuration.

Step 2: Connecting PingIdentity with IAM identity Center

This step involves configuring PingIdentity with the AWS IAM Identity Center sign-on details to complete the authentication setup.

  1. In the PingIdentity console, Navigate to the Applications Tab > Applications and select the application you created earlier in Step 1
  2. Select Enable Advanced Configuration and choose Enable.
Screenshot of the PingIdentity Applications dashboard showing the AWS Single Sign-On application selected. The overview panel displays key configuration sections including protocol (SAML), mapped attributes, selected policies, and access group (Amazon Q). The option "Enable Advanced Configuration" is highlighted near the bottom of the panel.

Figure 5 – Enable Advanced configuration for AWS single Sign-On application

Alt Text: Screenshot of the PingIdentity Applications dashboard showing the AWS Single Sign-On application selected. The overview panel displays key configuration sections including protocol (SAML), mapped attributes, selected policies, and access group (Amazon Q). The option “Enable Advanced Configuration” is highlighted near the bottom of the panel.

  1. Scroll down and select Download Metadata. This will save the Metadata file to your local computer, which you will use later during the configuration process.
  2. In another browser tab login to your AWS IAM Identity Center console and Select Choose your identity source.
  3. Under Identity source, select Change identity source from the Actions drop-down menu.
Screenshot of the IAM Identity Center settings page, focused on the "Identity source" tab. The page displays details such as identity source, authentication method, AWS access portal URL, issuer URL, and identity store ID. A dropdown menu labeled "Actions" is expanded in the top-right corner, showing options to "Customize AWS access portal URL" and "Change identity source," highlighted with a red box.

Figure 6 – Change identity source in IAM Identity Center Console

Alt Text: Screenshot of the IAM Identity Center settings page, focused on the “Identity source” tab. The page displays details such as identity source, authentication method, AWS access portal URL, issuer URL, and identity store ID. A dropdown menu labeled “Actions” is expanded in the top-right corner, showing options to “Customize AWS access portal URL” and “Change identity source,” highlighted with a red box.

  1. On the next page, select External identity provider and choose Next.
  2. Under Service provider metadata copy the IAM Identity Center Assertion Consumer Service (ACS) URL.

    Screenshot of the "Configure external identity provider" step in the AWS IAM Identity Center setup process. The screen displays service provider metadata including the AWS access portal sign-in URL, IAM Identity Center Assertion Consumer Service (ACS) URL (highlighted with a red box), and IAM Identity Center issuer URL. A button labeled "Download metadata file" is shown in the upper right.

    Figure 7 – Copy IAM Identity Center ACS URL

Alt Text: Screenshot of the “Configure external identity provider” step in the AWS IAM Identity Center setup process. The screen displays service provider metadata including the AWS access portal sign-in URL, IAM Identity Center Assertion Consumer Service (ACS) URL (highlighted with a red box), and IAM Identity Center issuer URL. A button labeled “Download metadata file” is shown in the upper right.

  1. Now go back to the PingIdentity browser tab and Navigate to the Configuration tab and select pencil icon to edit the details.
  2. Paste the ACS URL you copied from the IAM identity center console and choose Save.
Screenshots showing the configuration and editing of SAML settings for AWS Single Sign-On in PingIdentity. The first image displays the static configuration view, listing the ACS URL, signing key ("PingOne SSO Certificate for Administrators environment"), signing method ("Response"), and signing algorithm. The second image shows the editable configuration screen with the ACS URL input field highlighted in red, alongside dropdowns for selecting the signing key, options for signing method (Assertion, Response, or both), and the RSA_SHA256 signing algorithm. These screens guide users through setting up secure SAML integration with AWS SSO.

Figure 8 – Configuring AWS Single Sign-On SAML Settings in PingIdentity console

Alt Text: Two screenshots showing the configuration and editing of SAML settings for AWS Single Sign-On in PingIdentity. The first image displays the static configuration view, listing the ACS URL, signing key (“PingOne SSO Certificate for Administrators environment”), signing method (“Response”), and signing algorithm. The second image shows the editable configuration screen with the ACS URL input field highlighted in red, alongside dropdowns for selecting the signing key, options for signing method (Assertion, Response, or both), and the RSA_SHA256 signing algorithm. These screens guide users through setting up secure SAML integration with AWS SSO.

Step 3: Configure PingIdentity as external IdP in IAM identity Center

This step involves setting up PingIdentity as an external IdP in IAM Identity Center to enable federated access.

  1. Navigate back to the previous browser tab where you had IAM Identity Center console open.
  2. Upload the downloaded PingIdentity IdP SAML metadata file from step 3 of previous section and select Next.
Screenshot of the AWS Identity Center configuration screen where the user uploads the IdP SAML metadata XML file. The metadata file is shown as successfully selected. Below are empty fields for optional manual entry of IdP sign-in URL, IdP issuer URL, and IdP certificate. The "Next" button is highlighted in orange at the bottom right, indicating the next step in the setup process.

Figure 9 – AWS IAM Identity Center metadata

Alt Text: Screenshot of the AWS Identity Center configuration screen where the user uploads the IdP SAML metadata XML file. The metadata file is shown as successfully selected. Below are empty fields for optional manual entry of IdP sign-in URL, IdP issuer URL, and IdP certificate. The “Next” button is highlighted in orange at the bottom right, indicating the next step in the setup process.

  1. Review the list of changes. Once you are ready to proceed, type ACCEPT, then select Change identity source.

Step 4: Enable provisioning and identity-aware sessions in IAM identity Center

This step involves configuring user provisioning and enabling identity-aware sessions in AWS IAM Identity Center to support dynamic access control.

  1. In IAM Identity Center Console, Choose Settings in the left navigation pane.
  2. On the Settings page, locate and enable automatic provisioning. This immediately enabled automatic provisioning in IAM Identity Center and displays the necessary SCIM endpoint and access token information.
  3. In the Inbound automatic provisioning dialog box, copy each of the values for the following options. You will need to paste these later when you configure provisioning in PingIdentity.
    • SCIM endpoint
    • Access token
  4. Choose Close.
  5. Next enable identity-aware sessions and automatic provisioning.
Two options are displayed for further configuration: "Enable identity-aware sessions" and "Automatic provisioning." Both options have an "Enable" button on the right-hand side, highlighted in red.

Figure 10 – IAM Identity Center Settings for identity aware sessions and automatic provisioning

Alt Text: Two options are displayed for further configuration: “Enable identity-aware sessions” and “Automatic provisioning.” Both options have an “Enable” button on the right-hand side, highlighted in red.

Step 5: Configure connections provisioning in PingIdentity

This step involves setting up connection provisioning in PingIdentity to enable automatic user and group management.

  1. In the PingIdentity console, Navigate to the Integrations > Provisioning.
  2. Select plus icon > New Connection
  3. Under connection type Select Identity Store.
PingIdentity Provisioning configuration screen. The left sidebar highlights the "Provisioning" tab. The main panel shows the "Create a New Connection" dialog with two connection type options: "Identity Store" and "Gateway." The "Identity Store" option is selected using the "Select" button on the right. A plus (+) icon at the top indicates the option to add a new provisioning connection.

Figure 11 – PingIdentity connection provisioning

Alt Text: PingIdentity Provisioning configuration screen. The left sidebar highlights the “Provisioning” tab. The main panel shows the “Create a New Connection” dialog with two connection type options: “Identity Store” and “Gateway.” The “Identity Store” option is selected using the “Select” button on the right. A plus (+) icon at the top indicates the option to add a new provisioning connection.

  1. Select SCIM outbound from the list of options and select Next.
  2. Provide a name for the connection and select Next.
  3. Paste the SCIM endpoint URL into the SCIM BASE URL field.
  4. Navigate to Authentication Method and select OAuth 2 Bearer Token.
  5. Paste the Access token into the Oauth Access Token field.
  6. Select Test Connection to validate the connectivity and select Next.
PingIdentity interface showing the "Configure Authentication" step in the "Create a New Connection" wizard. Key fields include the SCIM Base URL, SCIM Version (2.0), Authentication Method (OAuth 2 Bearer Token), OAuth Access Token (obscured), and resource paths for Users and Groups. The "Test Connection" and "Next" buttons are visible at the bottom.

Figure 12 – Configure authentication details

Alt Text: PingIdentity interface showing the “Configure Authentication” step in the “Create a New Connection” wizard. Key fields include the SCIM Base URL, SCIM Version (2.0), Authentication Method (OAuth 2 Bearer Token), OAuth Access Token (obscured), and resource paths for Users and Groups. The “Test Connection” and “Next” buttons are visible at the bottom.

  1. Navigate to User Filter Expression and change to userName Eq “%s”.
  2. Choose Save. By default, the connection is created in a Disabled state.
Final step in the PingIdentity "Create a New Connection" wizard showing the "Configure Preferences" screen. The highlighted fields include "User Filter Expression" with the value userName Eq "%s", "User Identifier" set to userName, and group membership handling options ("Merge" and "Overwrite" with "Overwrite" selected). A "Save" button is highlighted at the bottom right.

Figure 13 – Edit UserFilter Expressions for the connection

Alt Text: Final step in the PingIdentity “Create a New Connection” wizard showing the “Configure Preferences” screen. The highlighted fields include “User Filter Expression” with the value userName Eq “%s”, “User Identifier” set to userName, and group membership handling options (“Merge” and “Overwrite” with “Overwrite” selected). A “Save” button is highlighted at the bottom right.

  1. Select the connection you created and select the toggle switch to enable the connection.
PingIdentity configuration screen showing the IAM Identity Store integration. The page displays the identity store name, and tabs for "Overview" and "Configuration." A toggle switch in the top-right corner is highlighted, indicating the integration is currently enabled.

Figure 14 – Enable the connection

Alt Text: PingIdentity configuration screen showing the IAM Identity Store integration. The page displays the identity store name, and tabs for “Overview” and “Configuration.” A toggle switch in the top-right corner is highlighted, indicating the integration is currently enabled.

Step 6: Configure rules provisioning in PingIdentity

This step involves setting up provisioning rules in PingIdentity to define how users and groups are synchronized.

  1. In the PingIdentity console, Navigate to the Integrations > Provisioning.
  2. Select plus icon > New Rule
  3. Provide a Name and Description for the rule.
  4. Choose Create.
  5. Select plus icon to select the Connection you created in the previous step.
  6. Choose Save.
Alt Text: Screenshots showing the final steps in connecting the IAM Identity Center to the IAM identity store using PingIdentity. The first image shows the IAM Identity Store connection listed under "Available Connections" with a plus (+) icon to initiate the link. The second image shows the selected connection from the PingOne Directory (P1) as the source and IAM identity store (SCIM) as the target, with the option to "Save" the configuration.

Figure 15 – Add the IAM identity center connection to the rule

Alt Text: Screenshots showing the final steps in connecting the IAM Identity Center to the IAM identity store using PingIdentity. The first image shows the IAM Identity Store connection listed under “Available Connections” with a plus (+) icon to initiate the link. The second image shows the selected connection from the PingOne Directory (P1) as the source and IAM identity store (SCIM) as the target, with the option to “Save” the configuration.

  1. If you want to sync users from your PingIdentity directory, create a user filter. To do so, navigate to User Filter and select pencil icon to edit the settings.
  2. Choose the appropriate filter from the drop down based on your use case and select Save. I have chosen Group Name which has been designated for Amazon Q Developer access.
Screenshot of the "Edit User Filter" interface in IAM Identity Center. The user filter is configured to provision users who belong to a group with names that contain "Amazon Q Developer." The condition logic is set to match if "Any" of the conditions are true.

Figure 16 – PingIdentity user filter

Alt Text: Screenshot of the “Edit User Filter” interface in IAM Identity Center. The user filter is configured to provision users who belong to a group with names that contain “Amazon Q Developer.” The condition logic is set to match if “Any” of the conditions are true.

  1. If you want to sync a group from your PingIdentity directory, create group provisioning. To do so, navigate to Group Provisioning and select pencil icon to edit the settings.
  2. Select the appropriate group which has been designated for Amazon Q Developer access and choose Save.
Screenshot of the "Edit Group Provisioning" screen in IAM Identity Center. The group "Amazon Q Developer" is selected for outbound provisioning. A "Save" button is highlighted in the bottom-left corner.

Figure 17 – PingIdentity Group Provisioning

Alt Text: Screenshot of the “Edit Group Provisioning” screen in IAM Identity Center. The group “Amazon Q Developer” is selected for outbound provisioning. A “Save” button is highlighted in the bottom-left corner.

  1. Navigate to Attribute Mapping and select the pencil icon to edit the settings.
  2. Delete the PingOne Directory attribute Primary Phone.
  3. Add a new attribute and select Username as PingOne Directory and displayName as IAM identity Store.
  4. Choose Save.
Two screenshots showing the editing of attribute mappings in IAM Identity Center. The first image displays default mappings such as 'Email Address' to 'workEmail' and 'Username' to 'userName', with an option to delete or update each field. The second image shows the addition of a new attribute mapping from 'Username' to 'displayName', along with highlighted 'Add' and 'Save' buttons.

Figure 18 – PingIdentity attribute mapping

Alt Text: Two screenshots showing the editing of attribute mappings in IAM Identity Center. The first image displays default mappings such as ‘Email Address’ to ‘workEmail’ and ‘Username’ to ‘userName’, with an option to delete or update each field. The second image shows the addition of a new attribute mapping from ‘Username’ to ‘displayName’, along with highlighted ‘Add’ and ‘Save’ buttons.

  1. Select the rule you created and select the toggle switch to enable the rule.
  2. This automatically provisions the users/groups from PingIdentity to IAM identity Center using SCIM.
IAM Identity Center sync summary showing successful user and group provisioning. The first image highlights two users impacted and successfully synced. The second image highlights one group impacted and successfully synced. Sync status is marked 'ACTIVE' in both views, confirming successful integration between PingOne and AWS IAM Identity Center.

Figure 19 – PingIdentity Users and Groups Sync status using SCIM

Alt Text: IAM Identity Center sync summary showing successful user and group provisioning. The first image highlights two users impacted and successfully synced. The second image highlights one group impacted and successfully synced. Sync status is marked ‘ACTIVE’ in both views, confirming successful integration between PingOne and AWS IAM Identity Center.

Step 7: Provide access to Amazon Q Developer

This step involves locating and subscribing the groups that need permission to use Amazon Q Developer.

  1. In the Amazon Q Developer console, under Subscriptions add the IAM identity center groups which require access to Amazon Q Developer.
  2. Select Subscribe and search for the group name.
  3. Select Assign.
Screenshot of the Amazon Q Developer Subscriptions page in the AWS Management Console. The "Groups" tab is selected, displaying “Amazon Q Developer,” with a subscription status of “Subscribed.” The “Amazon Q Developer” group is highlighted with a red box.

Figure 20 – Amazon Q Developer subscriptions page

Alt Text: Screenshot of the Amazon Q Developer Subscriptions page in the AWS Management Console. The “Groups” tab is selected, displaying “Amazon Q Developer,” with a subscription status of “Subscribed.” The “Amazon Q Developer” group is highlighted with a red box.

Setup Amazon Q Developer with IAM Identity Center

This section guides you through installing the Amazon Q Developer extension and setting up authentication with IAM Identity Center.

  1. To set up Amazon Q Developer extension in your integrated development environment (IDE), complete the steps in AWS documentation.
  2. Once extension is installed Choose Amazon Q icon in your IDE.
  3. Choose a sign-in option.
  4. Select Use with Pro license and choose
  5. Continue.
  6. Provide the Start URL. You can retrieve this AWS access portal URL from the IAM Identity Center Console.
Screenshot of the IAM Identity Center settings page in the AWS Console, displaying the identity source configuration. It shows that the identity source is set to "External identity provider" with SAML 2.0 authentication and SCIM provisioning. The highlighted section includes the AWS access portal URL and the Identity Store ID. The "Settings" tab is selected in the left navigation pane.

Figure 21 – IAM identity center access portal URL

Alt Text: Screenshot of the IAM Identity Center settings page in the AWS Console, displaying the identity source configuration. It shows that the identity source is set to “External identity provider” with SAML 2.0 authentication and SCIM provisioning. The highlighted section includes the AWS access portal URL and the Identity Store ID. The “Settings” tab is selected in the left navigation pane.

  1. Provide the region that hosts the identity directory and choose Continue
  2. Select Open on the resulting pop up which redirects to your browser.
  3. The browser redirects you to the Pingone URL where you enter your PingIdentity credentials and select Sign On.
  4. Upon successful authentication, select Allow access on the resulting pop up to login successfully.
A screen recording of Visual Studio Code where the user selects the Amazon Q icon from the sidebar. The screen transitions to a login prompt indicating that the user must authenticate using their PingIdentity credentials via IAM Identity Center before accessing Amazon Q Developer features. The message highlights that authentication is required to continue.

Figure 22 – Setup Visual Studio Code Amazon Q Developer extension

Alt Text: A screen recording of Visual Studio Code where the user selects the Amazon Q icon from the sidebar. The screen transitions to a login prompt indicating that the user must authenticate using their PingIdentity credentials via IAM Identity Center before accessing Amazon Q Developer features. The message highlights that authentication is required to continue.

Test Configuration

Upon successfully completing the previous step, you can now leverage the code suggestions by Amazon Q Developer.

A screen recording of Visual Studio Code where Amazon Q Developer generates a sample code inline.

Figure 23 – Amazon Q Developer example

Alt Text: A screen recording of Visual Studio Code where Amazon Q Developer generates a sample code inline.

Clean Up

To avoid ongoing charges after testing this solution, follow these steps to remove all provisioned resources:1. Remove PingIdentity Application Configuration

  • In the PingIdentity console, navigate to Applications.
  • Locate and delete the AWS Single Sign-On application that was configured for IAM Identity Center integration.

2. Reset IAM Identity Center Configuration

  • In the AWS IAM Identity Center console:
    • Navigate to Settings > Identity source.
    • Change the identity source back to the default IAM Identity Center directory if no longer using PingIdentity.
    • Remove any external metadata and configuration uploaded during the setup.

3. Revoke Subscriptions and Access

  • In the Amazon Q Developer console:
    • Go to Subscriptions and remove assigned groups such as Amazon Q Developer or code whisperer trial.
    • This will deactivate access and prevent any future charges tied to those subscriptions.

4. Remove Amazon Q Developer Extension

  • If desired, uninstall the Amazon Q Developer extension from Visual Studio Code to fully revert the development environment.

Conclusion

In this post, we demonstrated how to use existing PingIdentity credentials to access Amazon Q Developer through integration with IAM Identity Center. We provided a step-by-step guide for configuring PingIdentity as an external identity provider (IdP) with IAM Identity Center. Lastly, we demonstrated how to connect Amazon Q Developer extension within your IDE to AWS using your PingIdentity credentials, allowing seamless access to Amazon Q Developer.If you have any comments or questions, share them in the comments section.

To learn more about AWS Services

Amazon Q Developer

IAM Identity Center

AWS Toolkit for Visual Studio Code

About the author

Sid Vantair is a Solutions Architect with AWS covering Strategic accounts. He thrives on resolving complex technical issues to overcome customer hurdles. Outside of work, he cherishes spending time with his family and fostering inquisitiveness in his children.

AI security strategies from Amazon and the CIA: Insights from AWS Summit Washington, DC

Post Syndicated from Danielle Ruderman original https://aws.amazon.com/blogs/security/ai-security-strategies-from-amazon-and-the-cia-insights-from-aws-summit-washington-dc/

Speakers during AWS Summit Washington, DC 2025 on June 10, 2025.

At this year’s AWS Summit in Washington, DC, I had the privilege of moderating a fireside chat with Steve Schmidt, Amazon’s Chief Security Officer, and Lakshmi Raman, the CIA’s Chief Artificial Intelligence Officer. Our discussion explored how AI is transforming cybersecurity, threat response, and innovation across the public and private sectors. The conversation highlighted several key themes: how organizations can leverage AI to improve security outcomes, the rise of agentic AI and its impact on security, the importance of maintaining human oversight in AI systems, workforce development strategies, and practical approaches to implementing AI securely in enterprise environments. Below are a few excerpts from our conversation.

On leveraging AI to improve security outcomes

Steve Schmidt: “We’ve applied AI internally at Amazon in a couple of places that led to some significant benefits, including in the application security review process. By training our large language models internally on prior security reviews that we’ve done, it has allowed us to apply the knowledge and learning that our more senior staff have embodied in the documents that the LLM was trained on and expose that to our more junior staff. It really raises the bar on the absolute level of security that we can offer.”

Lakshmi Raman: “In the cybersecurity realm, we’re thinking about how AI helps us in our accreditation and authorization process, helping us ensure that the process to get systems accredited is going as quickly as possible, because the industry is moving so fast. Another area that we’re applying AI and machine learning is triaging data. We have vast amounts of data that comes in at an exponential rate, so we need to be able to go through it quickly so that we can surface insights. You can imagine a cybersecurity analyst who traditionally has gone through network data manually in order to think about blocking suspicious IP addresses or connections. Now there’s an opportunity to do all of that really efficiently and let the security analysts make the decision.”

On the rise of agentic AI and its implications for security

Steve Schmidt: “The biggest change we’re seeing right now in AI is the rise of agentic AI. The reason agentic AI is particularly interesting is that it brings with it a set of challenges about ensuring the software is taking actions within the context of the person who’s asking it…Think about that in the context of a government organization, where you have sets of information that are restricted to certain populations, there are classification decisions, access control limitations, and reasons that you can access certain data that have to be present before you can do so. Agentic AI brings opportunities—you can take actions using software automatically—but also challenges: how do we make sure that the software is doing exactly the right thing every single time, and more importantly, that we can prove what it did to stakeholders and regulators?”

Lakshmi Raman: “AI agents definitely have an opportunity to transform enterprise automation. Leveraging them to do complex multi-step workflows—to do tool calling across a variety of databases and other foundational tools—has tremendous potential, with a human as a crucial step to review what’s going on.”

On the importance of maintaining human oversight with AI

Lakshmi Raman: “In my world, I spend a lot of time thinking about how AI is impacting the workforce. One of the areas we’re looking at is the intersection between AI and our people. AI is able to speed up the processing and do automation, but at the end of the day, it’s really about who is taking on the risk, or deciding the intents and making the decisions. Whatever the machine output happens to be, really it’s about the human who’s deciding the level of oversight, the risk to take, and even whether to intervene.”

Steve Schmidt: “One thing that many people don’t realize about AI systems is that they’re nondeterministic. What nondeterminism means is you can ask an AI model the same question 100 times, and you will not get the same answer every time. So, having a human who can make a judgment about what the AI comes up with is critically important. We look at it this way: if you’re just asking a question and getting an answer, that may be one set of scrutiny that you have to get assistance. But if you’re going to take an action, you’ve got to be really sure the AI is correct. There has to be that skilled person that Lakshmi spoke about, at the end of the AI use process saying, ‘Yes, this is the right thing to do at this point in time with this context.’”

On building an AI-savvy security workforce

Steve Schmidt: “There’s a real problem in our industry: we don’t have enough security people. We simply can’t hire enough people with the right skills to do this job. What we’ve we found is that AI allows us to do a lot of the heavy lifting for the security staff, using tooling that used to have to be done by humans. Our staff is actually materially happier with their jobs if we remove a lot of that grunt work from them, which is super important. You want to keep the employees you have, so you give them tooling that helps them get the job done more efficiently, and they enjoy their job.”

Lakshmi Raman: “We’re looking for people who can live between the intersection of technology and social intelligence, people who can understand how those two areas can potentially interact around human behavior and how to think about future activities. When we’re thinking about analysts, for example, we’re thinking about people who have critical thinking skills, who can demonstrate analytic rigor, who can think multiple steps ahead with incomplete information. We’re also looking for people who have digital acumen with an understanding of cloud and cyber and AI, so that we have those technical skills in house. And finally, people who are interested in lifelong learning and curiosity, because threats change over the years. We need people who understand and are willing to learn about that.”

On advice for security leaders as AI accelerates

Steve Schmidt: “When you’re looking at making a decision, ask the person who’s bringing the information to you: ‘Why can’t AI do this?’ And if they don’t have an answer, ask ‘When will it be able to and under what condition?’ Move it into the now, the probable, the possible, and make it real for all of your staff all the time. If they’re not intentionally making that decision, they’re missing an opportunity.”

Lakshmi Raman: “You’ve got to get training out there for your users. We think of it at three different levels. First is our general workforce—which might be the most important user base—people who are sitting side by side with our AI practitioners and can help describe the workflows that need automation. Then we think about it for our practitioners, so they are keeping up with the latest. And then finally, our senior executives, who can think about how they can transform their organization with AI and generate that buy-in from the top level.”

AI is not just changing what we can do, but how we work. As Steve and Lakshmi emphasized, the most successful AI implementations will be those that thoughtfully balance automation with human oversight, focusing on use cases that deliver tangible value while managing risks appropriately. For security professionals, understanding both the technical and human dimensions of AI will be critical as we navigate this changing space.

Danielle Ruderman

Danielle Ruderman

Danielle is a Senior Manager for the AWS Worldwide Security Specialist Organization, where she leads a team that enables global CISOs and security leaders to better secure their cloud environments. Danielle is passionate about improving security by building company security culture that starts with employee engagement.

Use Model Context Protocol with Amazon Q Developer for context-aware IDE workflows

Post Syndicated from Ritik Khatwani original https://aws.amazon.com/blogs/devops/use-model-context-protocol-with-amazon-q-developer-for-context-aware-ide-workflows/

Earlier today, Amazon Q Developer announced Model Context Protocol (MCP) support in their Integrated Development Environment (IDE) plugins for Visual Studio Code and JetBrains. This allows developers to connect external tools or MCP servers to Q Developer, enabling more context-aware responses and complex workflows. MCP support has already been available in Amazon Q Developer for Command Line since April 29, 2025.

Introduction

Q Developer already had the ability to use tools within the IDE such as executing shell commands, reading local files, and generating code with the addition of the agentic coding experience. Now, developers have the ability to add additional tools that support MCP to their toolkit. MCP is an open protocol that standardizes how Large Language Models (LLMs) integrate with applications. It provides a way to share context, access data sources, and interact with APIs. You can read more about MCP in this introduction.

This ability to add additional context and tools allows Q Developer to write more accurate code, integrate with your planning tools, create UI components from designs, generate database documentation by examining your actual schema, and execute complex multi-tool tasks – all without the need for custom integration code. I’m excited to see this functionality coming to Q Developer IDE plugins, enhancing the development process right where developers spend most of their time.

In this post, I’ll walk you through a common scenario where I, as a developer, am tasked with working on an issue defined in a project management tool like Jira. The issue contains a user story, acceptance criteria, a link to a Figma design of the user interface, and additional technical implementation notes. To accomplish this efficiently, I’ll demonstrate how Q Developer can streamline the entire process by using two separate MCP servers to interact with Jira and Figma independently. Rather than manually switching between browser tabs, copying information, and trying to keep track of requirements across multiple tools, I’ll show how Q Developer can automatically fetch details using MCP and help me implement the feature while maintaining context across both platforms as shown in the figure below.

Q Developer extension in Visual Studio Code interacting with external tools using MCP servers

Figure 1: Q Developer extension in Visual Studio Code interacting with external tools using MCP servers

Configuring MCP Servers

To begin setup, click on the Configure MCP servers button at the top of the Chat tab bar as shown in the image below. This will bring up the list of MCP servers currently configured. Click the + (Add new MCP) button to add a new server.

Add MCP server configuration in Visual Studio Code’s Q Developer extension

Figure 2: Add MCP server configuration in Visual Studio Code’s Q Developer extension

You will set the scope of your MCP servers during configuration. A Global scope allows you to use the MCP server across all your projects, whereas a Workspace scope sets it up for only the current IDE workspace. Here’s an example configuration for the Atlassian and Figma MCPs I’ll be using:

Atlassian
Scope: This workspace
Name: Atlassian
Transport: stdio
Command: npx
Arguments:
-y
mcp-remote
https://mcp.atlassian.com/v1/sse
Figma
Scope: This workspace
Name: Figma
Transport: stdio
Command: npx
Arguments:
-y
mcp-remote
http://127.0.0.1:3845/sse

Note: The first time you set up the Atlassian MCP server, you’ll be asked to complete the OAuth authentication flow in your browser and provide access permissions to your Jira projects. Similarly, to connect to the Figma Dev Mode MCP server, you’ll need to enable it via the Figma desktop app.

Q Developer’s MCP management window showing configured Figma and Atlassian MCPs servers

Figure 3: Q Developer’s MCP management window showing configured Figma and Atlassian MCPs servers

To understand an MCP server’s individual tools, click on the expand icon next to its name as shown in the image below. Tools are executable functions exposed by the MCP server. They enable Q Developer’s agentic chat to perform actions and interact with external systems on your behalf. You can also configure permissions for individual tools. Each tool presents the option to Ask, Always allow, or Deny it such that Q Developer can’t invoke it. In my example, I’ll set all tools that only read data to Always allow for my workspace and set the rest of the tools to Ask.

MCP tool descriptions and configuration dropdown with options to Ask, Always allow or Deny

Figure 4: MCP tool descriptions and configuration dropdown with options to Ask, Always allow or Deny

With the MCP servers configured, let’s see how I can integrate them into my workflow.

Walkthrough

Q Developer is now enriched with additional information and tools available via the configured MCP servers. To demonstrate how this accelerates my developer productivity, I’ll be working with the Q Words game.

Scenario

Q-Words is an interactive word guessing game used in our customers’ workshops to demonstrate Q Developer’s capabilities. I’ve been tasked by the Product Manager to add a dark mode to the game. The User Story is logged in Jira and links to a Figma design that our designers have prepared.

Jira ticket showing user story and acceptance criteria for adding dark mode to a Q-Words game application

Figure 5: Jira ticket showing user story and acceptance criteria for adding dark mode to a Q-Words game application

Figma design showing dark and light mode interfaces for a QWords game application

Figure 6: Figma design showing dark and light mode interfaces for a QWords game application

Integrating MCPs into your development workflow

Let’s begin by asking Q Developer to check on tasks assigned to me in Jira by typing the following prompt in the agentic chat:

List issues that I need to work on

Q Developer will understand your intent and interact with your Atlassian MCP server to filter and show Jira issues that are assigned to you and in the To Do state. You can optionally prompt Q Developer to use a particular MCP server. Just as with any prompt, providing clear instructions will yield better results. In the image below, Q Developer retrieves details for the issue I’m assigned to work on.

Q Developer retrieves and describes issues assigned to me in Jira using the Atlassian MCP server

Figure 7: Q Developer retrieves and describes issues assigned to me in Jira using the Atlassian MCP server

Let’s begin work on the issue with the following prompt:

Move issue CRM-9 to In Progress and checkout a new git branch named after the issue id to begin working on it

Prompt Q Developer to begin working on an assigned issue

Figure 8: Prompt Q Developer to begin working on an assigned issue

Next, I’d like to understand the impact of the design changes on the current application. I can use the following prompt to accomplish this:

Analyze the Jira User Story and linked Figma design. Give me a technical implementation plan explaining the UI components that will need to be modified in the existing code.

Prompt Q Developer to help you analyze changes in existing code to implement the new UI

Figure 9: Prompt Q Developer to help you analyze changes in existing code to implement the new UI

Q Developer automatically pulls in issue details from Jira, along with the design specifics like colors from Figma. Before MCP, I would have had to add those details directly into the prompt or provided them as context from a local file. Now, my prompt only includes the description of the task whereas the context is enriched with details from the MCP servers. Review the proposed plan and suggest edits if needed. Once satisfied, prompt Q Developer to begin working on the changes:

Implement the plan

The diff view of changes by Q Developer to implement a dark mode feature in HTML and CSS

Figure 10: The diff view of changes by Q Developer to implement a dark mode feature in HTML and CSS

After reviewing the diff of the files changed by Q Developer, I can verify that the new Dark Mode feature has been implemented as desired. Let’s test the changes and ensure all acceptance criteria is met. To run the application, I use the following prompt:

Run the application locally

Q Developer will ask permission and run commands to spin up the local web server. I can then test the changes in my browser.

Updated application with dark mode toggle button implemented by Q Developer using MCP

Figure 11: Updated application with dark mode toggle button implemented by Q Developer using MCP

After a bit of testing, I can confirm that we’ve met all the acceptance criteria for the story. Let’s update the rest of the team on what we’ve accomplished with the following prompt:

Update the Jira issue status to Done and add a comment summarizing the changes made.

This convenient integration between Q Developer and Jira via MCP, saves me the back and forth between different tools to document the work accomplished.

A Jira ticket comment detailing the completed implementation of dark mode features, including theme toggle, CSS variables, and UI components

Figure 12: A Jira ticket comment detailing the completed implementation of dark mode features, including theme toggle, CSS variables, and UI components

Conclusion

The addition of MCP support in Amazon Q Developer for the IDE provides a standardized way to share context and interact with additional tools. In this post, I’ve demonstrated how I can use Q Developer in the IDE to interact with Atlassian Jira for task management and Figma for UI updates. I was able to do this without explicitly including user story details in my prompts or separately downloading design assets from UI mockups. Instead, Q Developer could automatically access user story context and easily integrate design assets using tools exposed by MCP servers. I encourage you to explore the new MCP capabilities and also check out the AWS MCP Servers repository on GitHub. Refer MCP configuration for Q Developer in the IDE to learn more.

To learn more about Amazon Q Developer’s features and pricing details, visit the Amazon Q Developer product page.

About the Author

Ritik Khatwani

Ritik Khatwani

Ritik is a Generative AI Specialist Solutions Architect at AWS based in New York City. He has deep expertise in building products as an engineer, architect, and founder. At AWS, he previously advised startups on how to build and grow in the cloud and now works with developers to reimagine their software development lifecycle using Amazon Q Developer.

Access Claude Sonnet 4 in Amazon Q Developer CLI

Post Syndicated from Kirankumar Chandrashekar original https://aws.amazon.com/blogs/devops/access-claude-sonnet-4-in-amazon-q-developer-cli/

Amazon Q Developer now supports Claude Sonnet 4 within the CLI, bringing advanced coding and reasoning capabilities to your development workflows at no additional cost. This latest model excels in coding with a state-of-the-art 72.7% for agentic coding on the SWE-bench (see Claude 4 announcement for more information). With enhanced coding and reasoning capabilities, it helps you analyze complex code, optimize everyday development tasks, implementing bug fixes, running bash commands, and developing new features with immediate feedback loops and more precise responses.

To help you leverage Claude Sonnet 4, Amazon Q Developer lets you easily select specific Claude Sonnet models, giving you increased flexibility the CLI.

  • Claude Sonnet 4: High-performance model with balanced intelligence
  • Claude Sonnet 3.7: High-performance model with extended thinking capability
  • Claude Sonnet 3.5: High-performance intelligent model

For detailed information about Claude model capabilities and comparison, refer to the Anthropic models overview.

In this blog, I will show you how to select Claude Sonnet 4 as your model within the Q Developer CLI and then walk you through a quick demo.

How to Choose Claude Sonnet 4

Make sure to update to the latest version (v1.11.0 onwards) of Amazon Q Developer CLI. Refer installing Amazon Q for command line for installation instructions. You can access Claude Sonnet 4 through these options:

  • During an active chat, use the /model command and select claude-4-sonnet
  • Start a new chat with q chat --model claude-4-sonnet
  • Set it as your default model using q settings chat.defaultModel claude-4-sonnet.

The supported model names for the --model parameter and settings are:

  • claude-3.5-sonnet
  • claude-3.7-sonnet (default)
  • claude-4-sonnet

Model Selection Priority Order

Q Developer CLI selects models in the following order:

  1. Current session model selections (via /model or --model)
  2. User-configured preferences in settings
  3. System default (Claude 3.7 Sonnet)

Key Behaviors

The Q Developer CLI agent defaults to Claude 3.7 Sonnet when no specific model is selected. During active chat sessions, you can seamlessly switch between models using the /model command. Chat continuity is maintained across sessions, with the system retaining the previously selected model when conversations are resumed. If you prefer Claude Sonnet 4, setting it as the default model in user settings will automatically apply to all new chat sessions, though this can be overridden with specific model selections as needed.

qcli-model-selection

Figure 1: Q Developer CLI showing the model loaded for the session

Claude Sonnet 4 with Q Developer CLI in Action

After switching to Claude Sonnet 4 in Q Developer CLI, let’s explore its capabilities with a practical coding example. Here’s the prompt I’ll use for this demonstration:

Create a Python command-line to-do list app with these features:
- Add tasks with descriptions and priorities (low/medium/high)
- Mark tasks as complete by index
- Display tasks sorted by priority, then insertion order
- Show completion status ([x] done, [ ] pending)
- Handle errors for empty tasks and invalid indices
- Store tasks in memory only
Please provide the code to implement this application.

qcli-model-selection-claude-sonnet-in-action

Figure 2: Q Developer CLI interface showing Claude Sonnet 4 in action

In the above demonstration, Q Developer CLI with Claude Sonnet 4 went beyond what was asked in the provided requirements in the prompt by implementing sophisticated command parsing with quoted descriptions, comprehensive error handling, and clean object-oriented design enhanced by type hints. The interface features a helpful guidance system with clear error messages, elegant enum-based priority management, and formatted output for clear task representation.

Additionally, Q Developer CLI with Claude Sonnet 4 also generated documentation in the README for the to-do application, including practical error handling examples and clear usage instructions – transforming the prompt requirements into a well-structured, user-friendly application.

Conclusion

The availability of Claude Sonnet 4 represents a significant advancement in Amazon Q Developer’s capabilities. From intricate code refactoring to streamlined documentation creation, Claude Sonnet 4 helps you accomplish both complex and routine development tasks efficiently.

Whether selecting Claude Sonnet 4 for complex tasks or using other models for specific needs, Amazon Q Developer adapts to your preferences, optimizing AI assistance while maintaining efficiency in your workflow.

The latest version(v1.11.0) of Amazon Q Developer awaits in the CLI, ready to support your development journey with enhanced model capabilities and selection options. Refer Installing Amazon Q for Command line for installation instructions.

To learn more about Amazon Q Developer’s features and pricing details, visit the Amazon Q Developer product page.

About the Author

kirankumar.jpeg

Kirankumar Chandrashekar is a Generative AI Specialist Solutions Architect at AWS, focusing on Amazon Q Developer. Bringing deep expertise in AWS cloud services, DevOps, modernization, and infrastructure as code, he helps customers enhance their development workflows using Amazon Q Developer. Kirankumar is passionate about solving complex customer challenges and enjoys music, cooking, and traveling.

Analyze media content using AWS AI services

Post Syndicated from Jack Bradham original https://aws.amazon.com/blogs/architecture/analyze-media-content-using-aws-ai-services/

Organizations managing large audio and video archives face significant challenges in extracting value from their media content. Consider a radio network with thousands of broadcast hours across multiple stations and the challenges they face to efficiently verify ad placements, identify interview segments, and analyze programming patterns.

In this post, we demonstrate how you can automatically transform unstructured media files into searchable, analyzable content. By combining Amazon Transcribe, Amazon Bedrock, Amazon QuickSight, and Amazon Q, organizations can achieve the following:

  • Process and transcribe media files upon upload
  • Identify commercials, interviews, and program segments
  • Extract insights using foundation models (FMs)
  • Create a searchable knowledge base
  • Generate rich visualizations for decision-making
  • Enable natural language queries across their media archive
  • Visualize complex information with intuitive graphics

In the following sections, we explore how these AWS services work together to help organizations unlock the full potential of their media content, whether for advertising compliance, content analysis, or discovering specific segments within thousands of hours of recordings.

Solution overview

This solution provides an event-driven media analysis pipeline that transforms how you manage and extract value from your content:

  • Streamline content management – Automatically process media files the moment they’re uploaded, saving time and reducing manual work
  • Unlock deeper insights – Generate accurate transcriptions that capture not just words, but the full context of your content—including speakers, timing, and key moments
  • Harness AI – Automatically extract meaningful insights and uncover hidden patterns in your media without extensive manual review
  • Build a searchable knowledge base – Turn scattered media files into a discoverable catalog that your entire team can use
  • Build a customizable interface – Create a customizable UI to search the catalog
  • Create powerful visualizations – Bring your insights to life with intuitive visualizations that make complex information immediately understandable

The following diagram illustrates our architecture.

Media Analysis Architecture

This event-driven architecture automatically processes and analyzes multimedia content using AWS services. The workflow consists of the following steps:

  1. A user uploads media files to an Amazon Simple Storage Service (Amazon S3) bucket. A “New Media” event triggers the first AWS Step Functions workflow. This workflow handles the initial cataloging based on values in the file name and launches the transcription process.
  2. Amazon Transcribe converts the audio into accurate, readable text. The transcribed content is securely saved to an S3 bucket for further analysis. A “Transcription Complete” event triggers the next step.
  3. A second Step Functions workflow processes the transcription. Using predefined prompts, Amazon Bedrock analyzes the transcripts to extract meaningful information. Key insights extracted from the transcript are stored in an S3 data lake.
  4. The processed results are organized systematically, structured by date (year/month/day) and tagged with relevant attributes. This organized data enables natural language queries through Amazon Q when used as a knowledge base, interactive visualizations using QuickSight, and straightforward content discovery and analysis.
  5. Amazon Athena serves as the data exploration tool to query the data lake. Athena is used as the data source in QuickSight, which turns complex data into clear, compelling visuals.

This architecture automatically transforms raw media content into searchable, analyzable data while maintaining an organized hierarchy for efficient access and analysis. The event-driven design provides automatic processing of new content as it arrives, and the combination of AWS AI services enables deep content understanding and insight extraction. Each AWS service plays a crucial role in transforming your media content:

  • Amazon Bedrock – Reviews content after transcription for insights and entity extraction:
    • Uses advanced FMs to analyze transcripts
    • Identifies commercials, interviews, and program segments
    • Extracts meaningful insights from content
  • Amazon EventBridge – Triggers actions in the cataloging workflow:
    • Monitors for new media files and completed transcriptions
    • Automatically triggers Step Functions workflows
  • AWS Lambda – Handles custom code actions needed in the workflow:
    • Facilitates interaction with Amazon Bedrock
    • Executes custom prompts on transcripts
    • Enables flexible, scalable processing
  • Amazon Q – Serves as the frontend and Retrieval Augmented Generation (RAG) engine:
    • Addresses enterprise generative AI needs by providing a turnkey solution with built-in security features like single sign-on (SSO) integration and responsible AI governance policies
    • Allows businesses to quickly deploy AI assistance while maintaining compliance, data privacy, and security standards
    • Enables natural language queries across the media archive
    • Links results to the source media files
    • Provides conversational access to content
  • Amazon QuickSight – Turns insights in beautiful visualization for better consumption:
    • Creates interactive dashboards and visualizations
    • Displays comprehensive media analytics
    • Helps track advertising, programming, and content patterns
  • Amazon S3 – Stores assets and the catalog:
    • Securely stores raw media files, transcripts, and processed data
    • Automatically triggers processing when new content is uploaded
  • AWS Step Functions – Orchestrates the entire content processing workflow:
    • Manages transcription and AI analysis steps
    • Provides robust error handling and automatic retries
  • Amazon Transcribe – Converts speech to accurate, readable text:
    • Identifies speakers and timestamps
    • Provides accurate transcriptions of audio content

Security considerations

Although this post focuses on the technical implementation of media content analysis, it’s important to acknowledge that production deployments should include comprehensive security measures:

  • Data storage security (Amazon S3):
    • Enable server-side encryption using AWS Key Management Service (AWS KMS) keys
    • Apply bucket policies restricting access to authorized principals only
    • Enable Amazon S3 Block Public Access at account and bucket levels
    • Enable versioning for data recovery
    • Implement lifecycle policies for data retention
    • Enable S3 access logging
    • Use presigned URLs for temporary access
  • Identity and Access Management (IAM):
    • Create dedicated service roles with minimum required permissions for:
      • Step Functions execution
      • Amazon Transcribe jobs
      • Amazon Bedrock API calls
      • Athena queries
    • Implement role-based access control
    • Regularly rotate credentials
    • Enable multi-factor authentication (MFA) for all users
    • Use AWS Organizations for multi-account management
  • Network security:
    • Deploy virtual private cloud (VPC) endpoints for:
      • Amazon S3
      • Athena
      • QuickSight
    • Implement network access control lists (ACLs) and security groups
    • Enable VPC Flow Logs
    • Use AWS PrivateLink where applicable
    • Configure route tables to control traffic flow
  • Data encryption:
    • Implement AWS KMS encryption for S3 objects
    • Use TLS 1.2+ for all API communications
    • Enable automatic key rotation in AWS KMS
    • Implement envelope encryption for sensitive data
  • Monitoring and detection:
    • Enable AWS CloudTrail for API activity logging
    • Configure Amazon GuardDuty for threat detection
    • Set up Amazon CloudWatch:
      • Metrics for service health
      • Alarms for security events
      • Log groups for application logs
    • Enable S3 server access logging
    • Configure VPC Flow Logs
  • Access controls:
    • Implement fine-grained access controls for:
      • Amazon Bedrock model access
      • Athena query permissions
      • QuickSight dashboard sharing
    • Conduct regular access reviews

Additionally, compliance requirements and data governance policies might impact how you implement this solution in your environment.

These security considerations are crucial but beyond the scope of this post. We recommend consulting AWS security best practices and working with your security team to implement appropriate measures for your specific use case. For more information on AWS security best practices, refer to Best Practices for Security, Identity, & Compliance.

The following sections walk you through setting up each component of the architecture to help you transform raw media into actionable insights.

Prerequisites

The following are the prerequisites to follow along this post:

Create S3 buckets

For this solution, we create three distinct buckets to support the media analytics workflow:

  • Raw media bucket for incoming files
  • Transcription outputs bucket
  • Processed insights bucket

For instructions on creating buckets, refer to Creating a general purpose bucket.

Configure EventBridge

You can enable event notifications on the raw media bucket to trigger your automated workflow through EventBridge. Establish your automation backbone by monitoring S3 bucket activities. When new media arrives or transcription completes, EventBridge will trigger the appropriate workflow, providing continuous processing. For further instructions, refer to Creating rules that react to events in Amazon EventBridge.

The following are two example triggers that can be used to filter events and trigger Step Functions workflows. The following is an example filter for new media files:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["rawinputbucket"]
    }
  }
}

The following is an example filter for new transcripts added to the data lake:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {
      "name": ["business-data-lake"]
    },
    "object": {
      "key": [{
        "suffix": ".transcription"
      }]
    }
  }
}

Create Step Functions workflows

We design the orchestration layer with two key workflows. The first handles media intake and transcription, and the second manages AI analysis. Each workflow includes safeguards for potential failures and retry mechanisms. For further instructions, refer to Learn how to get started with Step Functions.

The following diagram shows an example of processing new media uploads for indexing and transcription.

Media Analysis - Step Function Example

The following diagram shows an example of the Step Functions workflow that analyzes the transcription.

Set up Amazon Transcribe

To create an Amazon Transcribe job, you need permissions to do so. You can implement a speech-to-text conversion with powerful features like language detection, speaker identification, and custom vocabulary support to provide accurate transcription of your media content. For further instructions, refer to How Amazon Transcribe works.

Configure Amazon Bedrock

You can power your AI analysis engine by setting up precise prompts that extract meaningful insights. Amazon Bedrock processes transcripts to identify key segments, speakers, and topics, transforming raw text into structured data. For instructions, refer to Design a prompt.

The following is a sample prompt:

You will be reviewing a radio transcription to identify advertisements and extract relevant details. Your task is to analyze the provided transcript and output the results in a specific JSON format based on a given schema.
Please follow these steps to complete the task:
1. Carefully read through the entire transcript.
2. Identify all advertisements within the transcript. Look for clear indicators such as product mentions, promotional language, or transitions from regular content to commercial content.
3. For each advertisement you identify, determine the following information:
    - Company: The name of the company being advertised
    - Start time: The timestamp in the transcript where the ad begins
    - End time: The timestamp in the transcript where the ad ends
    - Product: The specific product or service being advertised
4. Format your findings into a JSON object that follows the provided schema. Each advertisement should be a separate object within an array.
5. Ensure these fields in your response are provided for each advertisement.. All are required fields: company, starttime, endtime, product. 
6. Use precise timestamps for start and end times. If exact times are not available, make a best estimate based on the transcript's context.
7. If a particular field is unclear or not explicitly mentioned in the transcript, you may use "Unknown" as the value.
8. Only respond with json and nothing else. Do not provide comments or explain your answer. 
9. Surround the JSON response with standard ```json markers
Here's an example of how your output should be formatted:
{
"advertisements": [
        {
            "company": "TechGadgets Inc.",
            "starttime": "00:05:30",
            "endtime": "00:06:15",
            "product": "SmartHome Hub"
        },
        {
            "company": "FreshFoods Market",
            "starttime": "00:15:45",
            "endtime": "00:16:30",
            "product": "Organic Produce Delivery Service"
        }
    ]
}
Do not add any fields that are not specified in the schema, and ensure all required fields are present for each advertisement.

Create a structured data lake

We create a hierarchical data organization strategy that enables efficient access and analysis. You can use AWS Glue crawlers to automatically discover and catalog your media metadata. For instructions, refer to Using crawlers to populate the Data Catalog. Configure Athena tables to enable SQL-based querying of your media insights:

CREATE OR REPLACE VIEW "commercials_view" AS 
SELECT
  metadata.market market
, metadata.station_call station_call
, metadata.format_type format_type
, CAST(metadata.timestamp AS timestamp) timestamp
, ads.company adCompany
, ads.product adProduct
, ads.starttime
, ads.endtime
FROM
  (commercials
CROSS JOIN UNNEST(advertisements) t (ads))

Set Up Amazon Q

You can enable natural language interaction with your media archive using Amazon Q Business. Configure the knowledge base and metadata to make your content searchable and accessible through conversational queries. Use the processed insights S3 buckets to configure the knowledge base. For instructions, refer to Getting started with Amazon Q Business.

The following screenshot shows example conversations with an AI assistant.

Build QuickSight dashboards

With QuickSight, you can create visual analytics that bring your data to life. Connect to Athena views to display advertising patterns, content analysis, and performance metrics in interactive dashboards. For more information, refer to Tutorial: Create an Amazon QuickSight dashboard.

The following screenshots are a few examples of dashboards created for a fictitious radio station as part of our use case.

Validate and optimize your media analytics solution

After you implement your media analytics architecture, follow these critical steps to achieve robust performance and alignment with your organization’s needs. First, configure a comprehensive testing approach. Imagine you’re preparing to launch your media analysis solution. Your testing journey begins with accuracy validation:

  • Compare transcription outputs against original media
  • Verify AI-generated insights for precision
  • Use representative sample sets from your content library

You start by taking a recently processed radio show and comparing its transcription against the original broadcast. Your team meticulously reviews the AI-generated insights, checking if key moments like ad transitions or interview segments are correctly identified. To make sure your system works across all content types, you select diverse samples from your library—perhaps a morning talk show, an evening news segment, and a weekend sports broadcast. Next, you delve into performance benchmarking:

  • Measure processing time for different media types
  • Evaluate resource utilization across AWS services
  • Identify potential bottlenecks in the workflow

Time how long it takes to process different types of media files, from short commercial segments to lengthy program recordings. As you watch how your AWS services respond under various loads, you can monitor resource consumption patterns. This helps you identify processing bottlenecks—for instance, you might discover that certain file types take longer to transcribe or that concurrent processing needs optimization. Finally, you put yourself in your users’ shoes for a thorough user experience assessment:

  • Test natural language queries with Amazon Q
  • Validate search result relevance
  • Gather feedback from potential end-users

Team members can interact with Amazon Q, asking questions they would naturally pose when searching for specific content. For example, you can test whether searching for “interviews about climate change last week” returns relevant results. Gathering feedback from potential users—perhaps a content manager with different needs than a compliance officer—provides invaluable insights. Their real-world experiences guide your refinements and make sure the system serves its intended purpose. This comprehensive testing approach, combining structured evaluation with real-world scenarios, sets the stage for a robust and user-friendly media analysis solution. As your media analysis solution moves from initial deployment to production, optimizing its performance becomes crucial for both cost-efficiency and user satisfaction. A radio network processing thousands of hours of content weekly might find that even small improvements in transcription accuracy or processing speed can lead to significant cost savings and better content discoverability. Similarly, a marketing team analyzing ad placements across multiple stations needs precise insights to make data-driven decisions about advertising effectiveness. With these business imperatives in mind, consider the following configuration optimization strategies:

  • Transcription refinement:
    • Adjust language models for domain-specific terminology
    • Fine-tune speaker identification settings
    • Implement custom vocabularies for improved accuracy
  • AI insight generation:
    • Refine prompts for more targeted analysis
    • Experiment with different AI models
    • Align extraction parameters with business objectives
  • Scalability considerations:
    • Test workflow performance with increasing media volumes
    • Implement appropriate auto scaling configurations
    • Monitor cost-effectiveness of your architecture
  • Continuous improvement:
    • Establish regular review cycles
    • Track key performance metrics
    • Iterate on your solution based on real-world usage

We recommend starting with a pilot implementation and gradually expanding your media analytics capabilities.

Clean up

To avoid incurring ongoing charges, clean up the resources you created as part of this solution:

  1. Delete QuickSight resources:
    1. Delete dashboards created for media analytics.
    2. Delete the datasets connected to Athena.
    3. If no longer needed, delete the QuickSight Enterprise subscription.
  2. Delete S3 buckets:
    1. Empty and delete the raw media bucket, transcription outputs bucket, and processed insights bucket.
  3. Remove EventBridge rules:
    1. Delete the rules created for monitoring S3 bucket activities.
    2. Remove targets associated with these rules.
  4. Delete Step Functions workflows:
    1. Delete the media intake and transcription workflow.
    2. Delete the AI analysis workflow.
  5. Remove Lambda functions:
    1. Delete Lambda functions created for interaction with Amazon Bedrock.
    2. Remove associated IAM roles and policies.
  6. Clean up data lake components:
    1. Delete Athena views and tables.
    2. Remove AWS Glue crawlers and databases.
    3. Delete stored query results.
  7. Remove Amazon Q configurations:
    1. Delete knowledge bases created.
    2. Remove custom configurations.
  8. Remove Amazon Bedrock settings:
    1. Remove custom prompts.
    2. Disable access to FMs if no longer needed.
  9. Delete Amazon Transcribe settings:
    1. Remove custom vocabularies.
    2. Delete stored transcription jobs.
  10. Remove IAM resources:
    1. Delete custom IAM roles created for this solution.
    2. Remove associated IAM policies.
  11. Complete additional cleanup:
    1. Delete CloudWatch Logs groups associated with Lambda functions.
    2. Remove CloudWatch alarms or metrics created for monitoring.
    3. Delete saved queries in Athena.

Common use cases

Organizations in different sectors can use this architecture to unlock value from their audio and video content. You can adapt this solution to meet your specific needs, such as managing broadcast media, corporate communications, educational materials, and more. Let’s explore how different industries might apply this technology:

  • Media and broadcasting:
    • Track advertising compliance
    • Verify media placement accuracy
    • Analyze broadcast content at scale
  • Corporate and enterprise:
    • Convert meeting recordings into searchable knowledge bases
    • Identify key decisions and action items
    • Enhance organizational knowledge management
  • Education and training:
    • Create comprehensive, topic-based course catalogs
    • Index training materials for quick retrieval
    • Support continuous learning initiatives
  • Legal services:
    • Generate precise, timestamped transcripts
    • Develop searchable legal proceeding archives
    • Improve document review efficiency
  • Healthcare:
    • Extract critical medical insights from consultations
    • Categorize patient interaction data
    • Support clinical documentation processes
  • Government and public sector:
    • Build comprehensive public meeting archives
    • Implement automated topic categorization
    • Enhance transparency and accessibility
  • Customer service:
    • Analyze call recordings for quality improvement
    • Identify service trends and customer pain points
    • Drive continuous customer experience enhancement

This media analytics architecture demonstrates notable versatility. By using AI, organizations can transform raw audio and video content into structured, meaningful insights that drive decision-making across industries.

Conclusion

In this post, we demonstrated how to use AWS services to convert unstructured media content into actionable intelligence. By combining Amazon Transcribe, Amazon Bedrock, QuickSight, and Amazon Q, you can create a scalable, automated solution for media analysis that adapts to your organizational needs.

This solution offers the following key architectural advantages:

  • Automated media file processing at scale
  • AI-powered insight generation
  • Natural language search capabilities
  • Interactive decision-making visualizations
  • Flexible, maintainable infrastructure

Organizations can now convert content into searchable knowledge, extract insights automatically, develop data-driven content strategies, and enhance operational efficiency through automation.

As audio and video content generation continues to accelerate, the ability to efficiently process and extract value becomes increasingly critical. This architecture provides a robust foundation for current needs while remaining adaptable to future technological innovations.

We invite you to explore how this media analytics solution can address your organization’s unique challenges. Consider your specific use cases and unlock the insights waiting to be discovered in your media archives.

About the authors

AWS Weekly Roundup: Amazon Aurora DSQL, MCP Servers, Amazon FSx, AI on EKS, and more (June 2, 2025)

Post Syndicated from Prasad Rao original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-aurora-dsql-mcp-servers-amazon-fsx-ai-on-eks-and-more-june-2-2025/

It’s AWS Summit Season! AWS Summits are free in-person events that take place across the globe in major cities, bringing cloud expertise to local communities. Each AWS Summit features keynote presentations highlighting the latest innovations, technical sessions, live demos, and interactive workshops led by Amazon Web Services (AWS) experts. Last week, events took place at AWS Summit Tel Aviv and AWS Summit Singapore.

The following photo shows the packed keynote at AWS Summit Tel Aviv.

AWS Summit Tel Aviv Keynote

Find an AWS Summit near you and join thousands of AWS customers and cloud professionals taking the next step in their cloud journey.

Last week, the announcement that piqued my interest most was the general availability of Amazon Aurora DSQL, which was introduced in preview at re:Invent 2024. Aurora DSQL is the fastest serverless distributed SQL database that enables you to build always available applications with virtually unlimited scalability, the highest availability, and zero infrastructure management.

Aurora DSQL active-active distributed architecture is designed for 99.99% single-Region and 99.999% multi-Region availability with no single point of failure and automated failure recovery. This means your applications can continue to read and write with strong consistency, even in the rare case an application is unable to connect to a Region cluster endpoint.

Single and multi region deployment of Amazon Aurora DSQL

What’s more fascinating is the journey behind building Aurora DSQL, a story that goes beyond the technology in the pursuit of engineering efficiency. Read the full story in Dr. Werner Vogels’ blog post, Just make it scale: An Aurora DSQL story.

Last week’s launches
Here are the other launches that got my attention:

  • Announcing new Model Context Protocol (MCP) servers for AWS Serverless and Containers – MCP servers are now available for AWS Lambda, Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and Finch. With MCP servers, you can get from idea to production faster by giving your AI assistants access to an up-to-date framework on how to correctly interact with your AWS service of choice. To download and try out the open source MCP servers, visit the aws-labs GitHub repository.
  • Announcing the general availability of Amazon FSx for Lustre Intelligent-Tiering – FSx for Lustre Intelligent-Tiering, a new storage class, automatically optimizes costs by tiering cold data to the applicable lower-cost storage tier based on access patterns and includes an optional SSD read cache to improve performance for your most latency-sensitive workloads.
  • Amazon FSx for NetApp ONTAP now supports write-back mode for ONTAP FlexCache volumes – Write-back mode is a new ONTAP capability that helps you achieve faster performance for your write-intensive workloads that are distributed across multiple AWS Regions and on-premises file systems.
  • AWS Network Firewall Adds Support for Multiple VPC Endpoints – AWS Network Firewall now supports configuring up to 50 Amazon Virtual Private Cloud (Amazon VPC) endpoints per Availability Zone for a single firewall. This new capability gives you more options to scale your Network Firewall deployment across multiple VPCs, using a centralized security policy.
  • Cost Optimization Hub now supports Savings Plans and reservations preferences – You can now use Cost Optimization Hub, a feature within the Billing and Cost Management Console, to configure preferred Savings Plans and reservation term and payment options preferences, so you can see your resulting recommendations and savings potential based on your preferred commitments.
  • AWS Neuron introduces NxD Inference GA, new features, and improved tools – With the release of Neuron 2.23, the NxD Inference library (NxDI) moves from beta to general availability and is now recommended for all multi-chip inference use cases. Neuron 2.23 also introduces new training capabilities, including context parallelism and Odds Ratio Preference Optimization (ORPO), and adds support for PyTorch 2.6 and JAX 0.5.3.
  • AWS Pricing Calculator, now generally available, supports discounts and purchase commitment – We announced the general availability of the AWS Pricing Calculator in the AWS console. You can now create more accurate and comprehensive cost estimates by providing two types of cost estimates: cost estimation for a workload, and estimation of a full AWS bill. You can also import your historical usage or create net new usage when creating a cost estimate. Additionally, with the new rate configuration inclusive of both pricing discounts and purchase commitments, you can gain a clearer picture of potential savings and cost optimizations for your cost scenarios.
  • AWS CDK Toolkit Library is now generally available – AWS CDK Toolkit Library provides programmatic access to core AWS CDK functionalities such as synthesis, deployment, and destruction of stacks. You can use this library to integrate CDK operations directly into your applications, custom CLIs, and automation workflows, offering greater flexibility and control over infrastructure management.
  • Announcing Red Hat Enterprise Linux for AWS – Red Hat Enterprise Linux (RHEL) for AWS, starting with RHEL 10, is now generally available, combining Red Hat’s enterprise-grade Linux software with native AWS integration. RHEL for AWS is built to achieve optimum performance of RHEL running on AWS.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS? page.

Additional updates
Here are some additional projects, blog posts, and news items that you might find interesting:

  • Introducing AI on EKS: powering scalable AI workloads with Amazon EKS – AI on EKS is a new open source initiative from AWS designed to help you deploy, scale, and optimize AI/ML workloads on Amazon EKS. AI on EKS repository includes deployment-ready blueprints for distributed training, LLM inference, generative AI pipelines, multi-model serving, agentic AI, GPU and Neuron-specific benchmarks, and MLOps best practices.
  • Revolutionizing earth observation with geospatial foundation models on AWS – Emerging transformer-based vision models for geospatial data—also called geospatial foundation models (GeoFMs)—offer a new and powerful technology for mapping the earth’s surface at a continental scale. This post explores how Clay Foundation’s Clay foundation model can be deployed for large-scale inference and fine-tuning on Amazon SageMaker. You can use the ready-to-deploy code samples to get started quickly with deploying GeoFMs in your own applications on AWS.

High level solution flow for inference and fine tuning using Geospatial Foundation Models

  • Going beyond AI assistants: Examples from Amazon.com reinventing industries with generative AI – Non-conversational applications offer unique advantages, such as higher latency tolerance, batch processing, and caching, but their autonomous nature requires stronger guardrails and exhaustive quality assurance compared to conversational applications, which benefit from real-time user feedback and supervision. This post examines four diverse Amazon.com examples of non-conversational generative AI applications.

Upcoming AWS events
Check your calendars and sign up for these upcoming AWS events:

  • AWS Summits – Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Stockholm (June 4), Sydney (June 4–5), Hamburg (June 5), Washington (June 10–11), Madrid (June 11), Milan (June 18), Shanghai (June 19–20), and Mumbai (June 19).
  • AWS re:Inforce – Mark your calendars for AWS re:Inforce (June 16–18) in Philadelphia, PA. AWS re:Inforce is a learning conference focused on AWS security solutions, cloud security, compliance, and identity.
  • AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Milwaukee, USA (June 5), Mexico (June 14), Nairobi, Kenya (June 14), and Colombia (June 28).

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Prasad

New and improved Amazon Q Developer experience in the AWS Management Console

Post Syndicated from Brendan Jenkins original https://aws.amazon.com/blogs/devops/new-and-improved-amazon-q-developer-experience-in-the-aws-management-console/

Amazon Q Developer just launched a new agentic experience within the AWS Management Console, that enables builders to get deeper insights about their AWS resources and improve their operational troubleshooting efficiency. This expands the agentic capabilities of Amazon Q Developer from both the integrated development environment (IDE) and command line interface (CLI) to the AWS console. Amazon Q Developer now functions as a resource analysis and operational troubleshooting assistant, able to consult multiple information sources and resolve complex queries, to get deeper insight into AWS environments faster and more easily than before. These capabilities are also available in chat applications such as Microsoft Teams and Slack. Now users can ask any question about AWS services and their resources, leaving Amazon Q Developer to automatically identify appropriate tools for the task, selecting from any AWS API across all services. It breaks queries into executable steps, asks for clarification when needed and combines information from multiple services to solve the task at hand. It can help analyze relationships between resources across multiple AWS services, examine configurations spanning different parts of infrastructure, synthesize information from various data sources to provide comprehensive insights, and respond to complex queries with detailed, actionable information.

For example, while troubleshooting an AWS Lambda function, a builder can simply ask, “How is this Lambda function getting invoked?” or “What are the IAM roles and permissions of my Lambda function?” and Amazon Q Developer will provide insights about the dependencies and interdependencies, evaluating their integration with other AWS services – all from a single natural language prompt. This enhancement allows builders to quickly obtain nuanced, contextual information about their AWS environment, significantly reducing the time and effort required for complex infrastructure analysis.

In this blog post, I’ll showcase several examples of complex prompts to demonstrate how Amazon Q Developer now delivers relevant and insightful responses based on the builder’s specific resources. Specifically, we’ll deep-dive into two main use cases: deeper resource introspection analysis and increased operational troubleshooting efficiency.

Deeper resource introspection and analysis

Amazon Q Developer now offers enhanced capabilities that make it even easier for builders to understand their AWS resources. With a single prompt, builders can now get comprehensive insights about their AWS services that previously required multiple steps. For example, when analyzing Amazon Simple Notification Service (SNS) topics and their subscribers, builders can simply ask “Show me all my SNS topics and their subscribers” to get a complete view of their configurations. This streamlined approach saves valuable time and effort, allowing developers to focus on building rather than navigating through multiple queries.

These new enhanced capabilities enable builders to simply ask for the insight needed, and Amazon Q Developer will perform the necessary multi-step reasoning based on a builder’s prompt. When the request is made, Amazon Q Developer determines the analytical steps required, retrieves information about the resources from multiple data sources, analyzes the relationships and configurations, and provides a comprehensive answer that addresses the need. Rather than builders having to think about which APIs to call or which services to check, Amazon Q Developer handles the complexity of the analysis, allowing builders to focus on understanding infrastructure rather than querying it.

To illustrate Amazon Q Developer’s capability in handling complex queries, let’s consider an example. Suppose a builder has a three-tier web application in an AWS account and they need to identify which Amazon Elastic Compute Cloud (Amazon EC2) instances, based on their Amazon Machine Images (AMIs) in the application layer, are actively communicating with Amazon Relational Database (RDS) in the backend. With this new update, a builder could open a new Amazon Q Developer chat in the AWS Management Console, and enter a prompt such as “List the AMIs used by my running EC2 instances in us-west-2 that can communicate with my RDS cluster”.

User prompts Amazon Q Developer about which Amazon EC2 AMIs are being used that communicate with Amazon RDS in the backend

Figure 1: Prompt to Amazon Q Developer and Amazon RDS database

Based on Amazon Q Developer’s response shown in figure 1 above, Amazon Q Developer was able to list the steps it took to gather the information, pulled applicable information from each service API, and gave one comprehensive and detailed insight about which AMIs were being used to communicate with the Amazon RDS cluster. This shows how Amazon Q Developer can take a single prompt, pull in information from multiple resources and give a comprehensive insight.

Let’s move to another example around AWS Lambda. Suppose a builder wants to know which AWS CloudFormation stacks are managing Lambda function resources. To do this, a builder could enter a prompt such as “List my AWS Lambda functions and the CloudFormation stacks that manage those resources”.

User prompts Amazon Q Developer to see what AWS CloudFormation Stacks are managing their AWS Lambda resources.

Figure 2: Prompt to Amazon Q Developer about Lambda and AWS CloudFormation

As shown above in figure 2, Amazon Q Developer was able to pull AWS CloudFormation information related to the AWS Lambda resources, and list each stack that was associated with the Lambda functions in the account. This, for example, can help many development and IT professionals better understand and manage their account resources by leveraging the complex reasoning of Amazon Q Developer.

Proceeding with one more example around AWS Lambda, let’s now suppose a builder wants to use Amazon Q Developer to see if there are any Amazon Simple Storage Service (Amazon S3) buckets invoking an AWS Lambda function in their AWS account. To identify this, a builder could enter a prompt such as “What AWS Lambda functions do I have in us-east-1 and are any of them invoked by an Amazon S3 bucket in the same region?”.

User prompts Amazon Q Developer to see if they have any AWS Lambda functions with Amazon S3 buckets as a trigger in their AWS account.User prompts Amazon Q Developer to see if they have any AWS Lambda functions with Amazon S3 buckets as a trigger in their AWS account.

Figure 3: Prompt and response from Amazon Q Developer about Amazon S3 and AWS Lambda

As shown in figure 3 above, Amazon Q Developer again called applicable service APIs to analyze Amazon S3 and AWS Lambda resources and was able to find that there was one AWS Lambda function with S3 as an event trigger.

Furthermore, building on our previous example, builders can try prompts around costs as well. For example, a builder can now prompt Amazon Q Developer “How much did I spend on Lambda functions that are invoked by my S3 bucket?” and Amazon Q will use its deeper resource introspection to tie costs to the resources that are connected.

These examples demonstrate Amazon Q Developer’s enhanced capability to process complex prompts involving multiple resource relationships. This improvement allows builders to obtain comprehensive answers with fewer steps, streamlining the overall process of asking questions about resources in accounts and making it easier to understand and manage AWS resources.

Improved Operational Troubleshooting

Amazon Q Developer can not only discover resources, their configurations, and their relationships, but also correlate that information with logs, metrics, and events to identify, analyze, and determine the root cause while troubleshooting operational issues in the AWS console. This helps streamline the process of resolving issues to enable quick troubleshooting.

To illustrate Amazon Q Developer’s capability in improved operational troubleshooting, let’s consider an example. Suppose a builder has a simple payment processing application consisting of Amazon API Gateway, AWS Lambda, and Amazon RDS in the backend. Furthermore, the application is returning 500 internal server errors causing downstream issues. Now, a builder can prompt Amazon Q Developer “Why is my user-profile-service-prod Lambda function throwing a 500 Internal server error?”.

User prompts Amazon Q Developer to see why their AWS Lambda functions are facing 500 internal server errors.

Figure 4: Prompt to Amazon Q Developer about internal server error

As shown above in figure 4, Amazon Q Developer automatically begins to gather relevant Amazon CloudWatch metrics, examines the function’s configuration and permissions, checks connected services like API Gateway and Amazon RDS, and analyzes recent changes

Response from Amazon Q after its analysis of various data sources.

Figure 5: Response from Q Developer for database timeouts

As shown above in figure 5, after querying applicable resources, Amazon Q Developer identified the root cause of the 500 internal server error. It shared information it pulled from the database and Lambda function logs and referenced a custom CloudWatch metric dashboard for evidence that the issue is due to database connection timeouts. Lastly, Amazon Q Developer also provided a list of ways to resolve the issue it identified. This example showcases how this new capability streamlines the process of analyzing operational issues, enabling quick troubleshooting.

Conclusion

The examples we’ve shown demonstrate how Amazon Q Developer handles the heavy lifting for users even better than before – from breaking down requests into analytical steps, to gathering data from multiple sources, to delivering meaningful insights about infrastructure, costs, and providing troubleshooting assistance.

As we continue to enhance Amazon Q Developer’s multi-step reasoning capabilities, builders will see it tackle even more complex analysis scenarios, helping them better understand and optimize AWS environments. Whether analyzing security configurations, examining resource relationships, or troubleshooting infrastructure issues, Amazon Q Developer can help save time and provide deeper insights into AWS resources.

To learn more and get started, visit Amazon Q Developer and Chatting with Amazon Q Developer in AWS Console Documentation.

About the authors

Brendan Jenkins

Brendan Jenkins is a Tech Lead Solutions Architect at Amazon Web Services (AWS) working with Enterprise AWS customers providing them with technical guidance and helping achieve their business goals. He has an area of specialization in DevOps and Machine Learning technology.

Optimizing fleet operations using Amazon SageMaker AI and Amazon Bedrock

Post Syndicated from Manny Sidhu original https://aws.amazon.com/blogs/architecture/optimizing-fleet-operations-using-amazon-sagemaker-ai-and-amazon-bedrock/

Every year in the United States, distracted driving claims thousands of lives and causes immense financial damage. More than 1.6 million accidents annually are caused by cell phone use while driving, and another 1.5 million result from drowsy drivers falling asleep at the wheel. These devastating—and preventable—accidents have sparked a major push for enhanced driver safety.

This initiative is particularly crucial in the commercial fleet industry, as accidents involving a large truck are often more dangerous and can cost hundreds of thousands of dollars. This post explores an innovative solution that leverages Amazon SageMaker AI and Amazon Bedrock to revolutionize driver coaching and enhance fleet efficiency. By harnessing the power of machine learning and artificial intelligence, we demonstrate how fleet operators can transform raw dashcam footage into actionable insights, empowering real-time driver monitoring and proactive safety measures – reducing costly accidents. Our approach combines AWS Artificial Intelligence (AI) and Internet of Things (IoT) services to create a comprehensive solution that not only detects distracted driving but also continuously improves its performance over time. Through this solution, we aim to show how fleet managers can significantly reduce distracted driving incidents, improve operational efficiency, and ultimately drive down costs in their commercial vehicle operations.

The Challenge: Effectively managing multiple dashcam feeds from commercial vehicle fleet

Today’s commercial vehicles are equipped with multi-camera systems that provide comprehensive coverage: inward-facing cameras monitor driver behavior, outward-facing cameras track oncoming traffic, and side/rear cameras detect cross-traffic and potential rear-end collisions. The sheer volume of video data generated by thousands of vehicles daily creates significant management and analysis challenges. While fleet operators traditionally use this dashcam footage for reactive purposes – such as law enforcement reporting, insurance claims, and driver exoneration – many organizations are missing a significant opportunity to leverage this data. As commercial fleets accumulate more miles, they generate rich datasets that can be used to train AI models capable of facilitating proactive safety improvements.

In this post, we’ll explore how to maximize the value of dashcam footage through best practices for implementing and managing Computer Vision systems in commercial fleet operations. We’ll demonstrate how to build and deploy edge-based machine learning models that provide real-time alerts for distracted driving behaviors, while effectively collecting, processing, and analyzing footage to train these AI models. This approach transforms fleet operations from reactive incident management to proactive safety enhancement, helping organizations convert raw video data into actionable insights that reduce safety incidents and improve overall fleet operational efficiency and cost-effectiveness.

Solution overview

A Distracted Driving Incident can occur when drivers engage in unsafe behaviors such as speeding, rolling stops, harsh braking, and aggressive acceleration. Fleet managers need to understand not just what happened during these incidents, but also the driver’s state of attention – whether they were focused on the road or distracted by activities like using a cellphone, eating, drinking, or experiencing fatigue common in long-haul driving.

Our solution leverages AWS services to create an end-to-end workflow capable of detecting and mitigating distracted driving. The steps involved include:

  1. Incident capture, ingestion, and labeling
  2. Model training, optimization, and deployment
  3. Continuous testing and improvement

Solution deep dive

This solution relies on a mix of AWS IoT, AI and generative AI services to build a scalable and cost-effective solution. Let’s start by looking at high level solution architecture and build the solution step-by-step.

Incident capture, ingestion, and labeling

To start the process of ingesting videos from a driver’s dashboard camera into the cloud, we capture the dashcam’s feed using the IoT Greengrass Kinesis Video Streamer Component. The video is streamed into the AWS Cloud using Kinesis Video Streams and stored in Amazon S3 by leveraging Kinesis Firehose. The videos are then converted into individual frames, analyzed by the Amazon Bedrock Nova Pro model to determine driver distraction, and sorted by an AWS Lambda function into an S3 bucket based on the analysis results. The sorted frames will next be used to train an AI model for edge deployment to detect distracted driving.

From a security perspective, it’s good practice to encrypt data in Amazon S3 buckets using AWS Key Management Service (KMS). You can enforce this by setting up SSE-KMS as the default encryption method to automatically encrypt uploaded objects. We also recommend implementing fine-grained AWS Identity & Access Management (IAM) roles to grant scoped access to images and videos. For data in transit between the edge and the cloud, you can use AWS IoT Greengrass certificates to encrypt your data and enforce identity verification. These measures can help protect against unauthorized access.

Edge-to-cloud architecture for real-time driver monitoring using AWS IoT, Kinesis, and ML services

With this process in place, we are continually collecting data from our fleet of commercial vehicles (while keeping security in mind). This data is automatically categorized and labeled based on the analysis from our Nova Pro model, and conveniently stored in S3, enabling us to seamlessly train an AI model – a process which we will describe next.

Model training, optimization, and deployment

The following diagram illustrates the process of training and deploying a distracted driver detection model. The process runs inside of an Amazon SageMaker Pipelines Workflow, which allows for seamless orchestration of other Amazon SageMaker AI services. This workflow begins with labeled driver images stored in Amazon S3, generated from the previously described workflow. This labeled dataset – consisting of driver images labeled as “distracted” or “not distracted’ – is used to train a ResNet50 model using Amazon SageMaker Training Jobs running on a Trn1 instance for price performance. As we train, the model learns how to identify distracted drivers. Once complete, the trained model is then quantized to INT8 using SageMaker Processing Jobs, and optimized for our specific type of edge hardware using SageMaker Neo. The optimized model is then stored in the SageMaker Model Registry for version control and governance (this will be helpful later when we iterate on our model with new training data). Finally, the model is pushed to S3 where AWS IoT Greengrass can initiate a deployment to the fleet of edge devices.

Running on the edge, the model performs inference multiple times a second on frames from the inward facing dashcam. (Inference speed calculated assuming edge compute has specs comparable to a Raspberry-Pi class of device.) If the driver is found to be distracted, the system alerts the driver by means of a noise. (ex. driver was falling asleep, and alert awakens them).

End-to-end AWS architecture for distracted driver detection: from model training to edge deployment

With this process in place, we have successfully leveraged the dataset we generated in the first diagram to train, optimize, and deploy our custom model to the ‘edge’ – in this case, to each vehicle in our fleet. Our model is now alerting drivers of dangerous behavior and helping to proactively prevent collisions. But our model likely isn’t perfect – perhaps it misses a dangerous behavior that wasn’t in the training dataset, or alerts unnecessarily. To validate our model is working well and further improve it to reduce errors, we implement continuous testing and improvement procedures.

Continuous testing and improvement

We need to continue to ingest driver dashcam data and compare our edge model’s predictions with our original source of ‘ground truth’ – Nova Pro.

The system collects frames for model validation in two scenarios: when vehicle telemetry detects incidents (hard braking, crashes) or when the edge model identifies distracted driving. These frames are sent to Amazon Bedrock for a ‘fact check’ to see if the edge model performed optimally. The comparative results between Amazon Bedrock and the edge model are stored in a dedicated S3 bucket for model evaluation. When sufficient new validated data is collected, or when the model’s agreement with Amazon Bedrock falls below a threshold, Amazon EventBridge triggers the previously described SageMaker Pipelines Workflow to fine tune, optimize, and re-deploy the improved model to the edge, now powered by our newly collected ‘disagreement data’.

Edge-to-cloud feedback loop for ML model validation using AWS IoT, Bedrock, and SageMaker services

We should also perform comparative analysis of our new model against our historical models stored in the Amazon SageMaker Model Registry to validate that our latest model actually performs better than historical models, verifying we don’t see a regression. If our latest model doesn’t outperform historical models, we should not deploy it, and instead investigate if we are suffering from overfitting or bad training data. In summary, we now have a model running inside fleet vehicles capable of alerting drivers to unsafe behavior. This could effectively reduce drowsy driving accidents by keeping drivers awake and alert, while also warning drivers about unsafe decisions like eating or using a cell phone while driving. This system is also self-training and self-improving, so it will continue to get better over time. Additionally, fleet management companies could aggregate safety data and reward top drivers to further incentivize safe driving habits.

Conclusion

In this post, we’ve explored an innovative solution that leverages AWS services to revolutionize driver coaching and fleet operations. By combining the power of Amazon SageMaker and Amazon Bedrock with AWS IoT and edge computing capabilities, we’ve demonstrated how to create a comprehensive, scalable solution for monitoring and improving driver behavior in real-time. This solution addresses the challenges of managing vast amounts of dashcam footage from commercial vehicle fleets, transforming raw video data into actionable insights. By implementing an end-to-end workflow that includes incident capture, categorization, model training, deployment, and continuous improvement, fleet operators can shift from reactive incident management to proactive safety enhancement. The benefits of this approach include:

  1. Enhanced safety: Real-time detection of distracted driving behaviors allows for immediate intervention and coaching.
  2. Improved efficiency: Automated analysis of dashcam footage reduces manual review time and costs.
  3. Scalability: The solution can handle large fleets and growing datasets with ease.
  4. Continuous improvement: The system learns and adapts over time, becoming more accurate and effective.
  5. Cost-effectiveness: By leveraging edge computing and optimized models, the solution minimizes compute costs.

As the transportation industry continues to evolve, solutions like this will play a crucial role in improving road safety, reducing operational costs, and enhancing overall fleet performance. By harnessing the power of AI and cloud computing, fleet operators can create safer, more efficient driving environments that benefit not only their businesses but also society as a whole. The future of fleet operations is here, and it’s driven by intelligent, data-driven systems that turn every mile driven into an opportunity for improvement and innovation.

Learn more by exploring AWS code samples to build hands-on SageMaker expertise. See the service in action through practical examples that demonstrate how to optimize model training and deployment across various use cases. Understand the financial advantages by conducting a cloud economics TCO analysis comparing traditional infrastructure against SageMaker’s managed services. This exercise reveals how SageMaker alleviates hidden costs while accelerating your ML development cycle.

Ready to take the next step? Connect with your AWS Solutions Architect to arrange a SageMaker AI Immersion Day tailored to your team’s specific challenges. These expert-led sessions provide personalized guidance that will help you implement SageMaker effectively within your organization’s unique context. For deeper dive into other relevant services Amazon Kinesis Video Streams, AWS IoT Greengrass, Amazon Bedrock

About the authors

AWS Weekly Roundup: Claude 4 in Amazon Bedrock, EKS Dashboard, community events, and more (May 26, 2025)

Post Syndicated from Veliswa Boya original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-4-in-amazon-bedrock-eks-dashboard-community-events-and-more-may-26-2025/

As the tech community we continue to have many opportunities to learn and network with other like-minded folks. This past week AWS customers attended the AWS Summit Dubai for an action-packed day featuring live demos, hands-on experiences with cutting-edge AI/ML tools, and more. Right here in South Africa I attended the Data & AI Community in Durban for a day of inspiration and learning from the community. In India, the AWS Community Day Bengaluru brought together hundreds of passionate tech enthusiasts for a day of learning and networking.

Last week’s launches
Here are the launches that got my attention:

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS? page.

Additional updates
Here are some additional projects, blog posts, and news items that you might find interesting:

Upcoming AWS events
Check your calendars and sign up for these upcoming AWS events:

  • AWS Summits – Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Tel Aviv (May 28), Singapore (May 29), Stockholm (June 4), Sydney (June 4–5), Washington (June 10-11), and Madrid (June 11).
  • AWS re:Inforce – Mark your calendars for AWS re:Inforce (June 16–18) in Philadelphia, PA. AWS re:Inforce is a learning conference focused on AWS security solutions, cloud security, compliance, and identity.
  • AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Milwaukee, USA (June 5), and Nairobi, Kenya (June 14).

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Veliswa.

Empower financial analytics by creating structured knowledge bases using Amazon Bedrock and Amazon Redshift

Post Syndicated from Nita Shah original https://aws.amazon.com/blogs/big-data/empower-financial-analytics-by-creating-structured-knowledge-bases-using-amazon-bedrock-and-amazon-redshift/

Traditionally, financial data analysis could require deep SQL expertise and database knowledge. Now with Amazon Bedrock Knowledge Bases integration with structured data, you can use simple, natural language prompts to query complex financial datasets. By combining the AI capabilities of Amazon Bedrock with an Amazon Redshift data warehouse, individuals with varied levels of technical expertise can quickly generate valuable insights, making sure that data-driven decision-making is no longer limited to those with specialized programming skills.

With the support for structured data retrieval using Amazon Bedrock Knowledge Bases, you can now use natural language querying to retrieve structured data from your data sources, such as Amazon Redshift. This enables applications to seamlessly integrate natural language processing capabilities on structured data through simple API calls. Developers can rapidly implement sophisticated data querying features without complex coding—just connect to the API endpoints and let users explore financial data using plain English. From customer portals to internal dashboards and mobile apps, this API-driven approach makes enterprise-grade data analysis accessible to everyone in your organization. Using structured data from a Redshift data warehouse, you can efficiently and quickly build generative AI applications for tasks such as text generation, sentiment analysis, or data translation.

In this post, we showcase how financial planners, advisors, or bankers can now ask questions in natural language, such as, “Give me the name of the customer with the highest number of accounts?” or “Give me details of all accounts for a specific customer.” These prompts will receive precise data from the customer databases for accounts, investments, loans, and transactions. Amazon Bedrock Knowledge Bases automatically translates these natural language queries into optimized SQL statements, thereby accelerating time to insight, enabling faster discoveries and efficient decision-making.

Solution overview

To illustrate the new Amazon Bedrock Knowledge Bases integration with structured data in Amazon Redshift, we will build a conversational AI-powered assistant for financial assistance that is designed to help answer financial inquiries, like “Who has the most accounts?” or “Give details of the customer with the highest loan amount.”

We will build a solution using sample financial datasets and set up Amazon Redshift as the knowledge base. Users and applications will be able to access this information using natural language prompts.

The following diagram provides an overview of the solution.

For building and running this solution, the steps include:

  1. Load sample financial datasets.
  2. Enable Amazon Bedrock large language model (LLM) access for Amazon Nova Pro.
  3. Create an Amazon Bedrock knowledge base referencing structured data in Amazon Redshift.
  4. Ask queries and get responses in natural language.

To implement the solution, we use a sample financial dataset that is for demonstration purposes only. The same implementation approach can be adapted to your specific datasets and use cases.

Download the SQL script to run the implementation steps in Amazon Redshift Query Editor V2. If you’re using another SQL editor, you can copy and paste the SQL queries either from this post or from the downloaded notebook.

Prerequisites

Make sure your meet the following prerequisites:

  1. Have an AWS account.
  2. Create an Amazon Redshift Serverless workgroup or provisioned cluster. For setup instructions, see Creating a workgroup with a namespace or Create a sample Amazon Redshift database, respectively. The Amazon Bedrock integration feature is supported in both Amazon Redshift provisioned and serverless.
  3. Create an AWS Identity and Access Management (IAM) role. For instructions, see Creating or updating an IAM role for Amazon Redshift ML integration with Amazon Bedrock.
  4. Associate the IAM role to a Redshift instance.
  5. Set up the required permissions for Amazon Bedrock Knowledge Bases to connect with Amazon Redshift.

Load sample financial data

To load the finance datasets to Amazon Redshift, complete the following steps:

  1. Open the Amazon Redshift Query Editor V2 or another SQL editor of your choice and connect to the Redshift database.
  2. Run the following SQL to create the finance data tables and load sample data: 
    -- Create table
CREATE TABLE accounts (
    id integer ,
    account_id integer PRIMARY KEY,
    customer_id integer,
    account_type character varying(256),
    opening_date date,
    balance bigint,
    currency character varying(256)
);

CREATE TABLE customer (
    id integer,
    customer_id integer PRIMARY KEY ,
    name character varying(256) ,
    age integer,
    gender character varying(256) ,
    address character varying(256) ,
    phone character varying(256) ,
    email character varying(256)
);

CREATE TABLE investments (
    id integer ,
    investment_id integer PRIMARY KEY,
    customer_id integer ,
    investment_type character varying(256) ,
    investment_name character varying(256) ,
    purchase_date date ,
    purchase_price bigint ,
    quantity integer 
);


CREATE TABLE loans (
    id integer ,
    loan_id integer PRIMARY KEY,
    customer_id integer ,
    loan_type character varying(256) ,
    loan_amount bigint ,
    interest_rate integer ,
    start_date date ,
    end_date date 
);

CREATE TABLE orders (
    id integer ,
    order_id integer PRIMARY KEY,
    customer_id integer ,
    order_type character varying(256) ,
    order_date date ,
    investment_id integer ,
    quantity integer ,
    price integer 
);

CREATE TABLE transactions (
    id integer ,
    transaction_id integer PRIMARY KEY ,
    account_id integer REFERENCES accounts(account_id),
    transaction_type character varying(256) ,
    transaction_date date ,
    amount integer ,
    description character varying(256) 
);

  3. Download the sample financial dataset to your local storage and unzip the zipped folder.
  4. Create an Amazon Simple Storage Service (Amazon S3) bucket with a unique name. For instructions, refer to Creating a general purpose bucket.
  5. Upload the downloaded files into your newly created S3 bucket.
  6. Using the following COPY command statements, load the datasets from Amazon S3 into the new tables you created in Amazon Redshift. Replace <<your_s3_bucket>> with the name of your S3 bucket and <<your_region>> with your AWS Region. 
    -- Load sample data
COPY accounts FROM 's3://<<your_s3_bucket>>/accounts.csv' IAM_ROLE DEFAULT FORMAT AS CSV DELIMITER ',' QUOTE '"' IGNOREHEADER 1 REGION AS '<<your_region>>';

COPY customer FROM 's3://<<your_s3_bucket>>/customer.csv' IAM_ROLE DEFAULT FORMAT AS CSV DELIMITER ',' QUOTE '"' IGNOREHEADER 1 REGION AS '<<your_region>>';
COPY investments FROM 's3://<<your_s3_bucket>>/investments.csv' IAM_ROLE DEFAULT FORMAT AS CSV DELIMITER ',' QUOTE '"' IGNOREHEADER 1 REGION AS '<<your_region>>';
COPY loans FROM 's3://<<your_s3_bucket>>/loans.csv' IAM_ROLE DEFAULT FORMAT AS CSV DELIMITER ',' QUOTE '"' IGNOREHEADER 1 REGION AS '<<your_region>>';
COPY orders FROM 's3://<<your_s3_bucket>>/orders.csv' IAM_ROLE DEFAULT FORMAT AS CSV DELIMITER ',' QUOTE '"' IGNOREHEADER 1 REGION AS '<<your_region>>';
COPY transactions FROM 's3://<<your_s3_bucket>>/transactions.csv' IAM_ROLE DEFAULT FORMAT AS CSV DELIMITER ',' QUOTE '"' IGNOREHEADER 1 REGION AS '<<your_region>>';

Enable LLM access

With Amazon Bedrock, you can access state-of-the-art AI models from providers like Anthropic, AI21 Labs, Stability AI, and Amazon’s own foundation models (FMs). These include Anthropic’s Claude 2, which excels at complex reasoning and content generation; Jurassic-2 from AI21 Labs, known for its multilingual capabilities; Stable Diffusion from Stability AI for image generation; and Amazon Titan models for various text and embedding tasks. For this demo, we use Amazon Bedrock to access the Amazon Nova FMs. Specifically, we use the Amazon Nova Pro model, which is a highly capable multimodal model designed for a wide range of tasks like video summarization, Q&A, mathematical reasoning, software development, and AI agents, including high speed and accuracy for text summarization tasks.

Make sure you have the required IAM permissions to enable access to available Amazon Bedrock Nova FMs. Then complete the following steps to enable model access in Amazon Bedrock:

  1. On the Amazon Bedrock console, in the navigation pane, choose Model access.
  2. Choose Enable specific models.
  3. Search for Amazon Nova models, select Nova Pro, and choose Next.
  4. Review the selection and choose Submit.

Create an Amazon Bedrock knowledge base referencing structured data in Amazon Redshift

Amazon Bedrock Knowledge Bases uses Amazon Redshift as the query engine to query your data. It reads metadata from your structured data store to generate SQL queries. There are different supported authentication methods to create the Amazon Bedrock knowledge base using Amazon Redshift. For more information, refer to the Set up query engine for your structured data store in Amazon Bedrock Knowledge Bases.

For this post, we create an Amazon Bedrock knowledge base for the Redshift database and sync the data using IAM authentication.

If you’re creating an Amazon Bedrock knowledge base through the AWS Management Console, you can skip the service role setup mentioned in the previous section. It automatically creates one with the necessary permissions for Amazon Bedrock Knowledge Bases to retrieve data from your new knowledge base and generate SQL queries for structured data stores.

When creating an Amazon Bedrock knowledge base using an API, you must attach IAM policies that grant permissions to create and manage knowledge bases with connected data stores. Refer to Prerequisites for creating an Amazon Bedrock Knowledge Base with a structured data store for instructions.

Complete the following steps to create an Amazon Bedrock knowledge base using structured data:

  1. On the Amazon Bedrock console, choose Knowledge Bases in the navigation pane.
  2. Choose Create and choose Knowledge Base with structure data store from the dropdown menu.
  3. Provide the following details for your knowledge base:
    1. Enter a name and optional description.
    2. Select Amazon Redshift as the query engine.
    3. Select Create and use a new service role for resource management.
    4. Make note of this newly created IAM role.
    5. Choose Next to proceed to the next part of the setup process.
    6. Configure the query engine:
      • Select Redshift Serverless (Amazon Redshift provisioned is also supported).
      • Choose your Redshift workgroup.
      • Use the IAM role created earlier.
      • Under Default storage metadata, select Amazon Redshift databases and for Database, choose dev.
      • You can customize settings by adding specific contexts to enhance the accuracy of the results.
      • Choose Next.
    7. Complete creating your knowledge base.
    8. Record the generated service role details.
    9. Next, grant appropriate access to the service role for Amazon Bedrock Knowledge Bases through the Amazon Redshift Query Editor V2. Update <your Service Role name> in the following statements with your service role, and update the value for <your schema>. 
      CREATE USER "IAMR:<your Service Role name>" WITH PASSWORD DISABLE;
SELECT * FROM PG_USER; -- To verify that the user is created.
GRANT SELECT ON ALL TABLES IN SCHEMA <your schema> TO "IAMR:<your Service Role name>";
--You can also Restricting access to certain tables for finer-grained control on the tables that can be accessed as shown below
GRANT SELECT ON TABLE customer to "IAMR:<your Service Role name>";
GRANT SELECT ON TABLE loan to "IAMR:<your Service Role name>";

Now you can update the knowledge base with the Redshift database.

  1. On the Amazon Bedrock console, choose Knowledge Bases in the navigation pane.
  2. Open the knowledge base you created.
  3. Select the dev Redshift database and choose Sync.

It may take a few minutes for the status to display as COMPLETE.

Ask queries and get responses in natural language

You can set up your application to query the knowledge base or attach the knowledge base to an agent by deploying your knowledge base for your AI application. For this demo, we use a native testing interface on the Amazon Bedrock Knowledge Bases console.

To ask questions in natural language on the knowledge base for Redshift data, complete the following steps:

  1. On the Amazon Bedrock console, open the details page for your knowledge base.
  2. Choose Test.
  3. Choose your category (Amazon), model (Nova Pro), and inference settings (On demand), and choose Apply.
  4. In the right pane of the console, test the knowledge base setup with Amazon Redshift by asking a few simple questions in natural language, such as “How many tables do I have in the database?” or “Give me list of all tables in the database.”

The following screenshot shows our results.

  1. To view the generated query from your Amazon Redshift based knowledge base, choose Show details next to the response.
  2. Next, ask questions related to the financial datasets loaded in Amazon Redshift using natural language prompts, such as, “Give me the name of the customer with the highest number of accounts” or “Give the details of all accounts for customer Deanna McCoy.”

The following screenshot shows the responses in natural language.

Using natural language queries in Amazon Bedrock, you were able to retrieve responses from the structured financial data stored in Amazon Redshift.

Considerations

In this section, we discuss some important considerations when using this solution.

Security and compliance

When integrating Amazon Bedrock with Amazon Redshift, implementing robust security measures is crucial. To protect your systems and data, implement essential safeguards including restricted database roles, read-only database instances, and proper input validation. These measures help prevent unauthorized access and potential system vulnerabilities. For more information, see Allow your Amazon Bedrock Knowledge Bases service role to access your data store.

Cost

You incur a cost for converting natural language to text based on SQL. To learn more, refer to Amazon Bedrock pricing.

Use custom contexts

To improve query accuracy, you can enhance SQL generation by providing custom context in two key ways. First, specify which tables to include or exclude, focusing the model on relevant data structures. Second, supply curated queries as examples, demonstrating the types of SQL queries you expect. These curated queries serve as valuable reference points, guiding the model to generate more accurate and relevant SQL outputs tailored to your specific needs. For more information, refer to Create a knowledge base by connecting to a structured data store.

For different workgroups, you can create separate knowledge bases for each group, with access only to their specific tables. Control data access by setting up role-based permissions in Amazon Redshift, verifying each role can only view and query authorized tables.

Clean up

To avoid incurring future charges, delete the Redshift Serverless instance or provisioned data warehouse created as part of the prerequisite steps.

Conclusion

Generative AI applications provide significant advantages in structured data management and analysis. The key benefits include:

  • Using natural language processing – This makes data warehouses more accessible and user-friendly
  • Enhancing customer experience – By providing more intuitive data interactions, it boosts overall customer satisfaction and engagement
  • Simplifying data warehouse navigation – Users can understand and explore data warehouse content through natural language interactions, improving ease of use
  • Improving operational efficiency – By automating routine tasks, it allows human resources to focus on more complex and strategic activities

In this post, we showed how the natural language querying capabilities of Amazon Bedrock Knowledge Bases when integrated with Amazon Redshift enables rapid solution development. This is particularly valuable for the finance industry, where financial planners, advisors, or bankers face challenges in accessing and analyzing large volumes of financial data in a secured and performant manner.

By enabling natural language interactions, you can bypass the traditional barriers of understanding database structures and SQL queries, and quickly access insights and provide real-time support. This streamlined approach accelerates decision-making and drives innovation by making complex data analysis accessible to non-technical users.

For additional details on Amazon Bedrock and Amazon Redshift integration, refer to Amazon Redshift ML integration with Amazon Bedrock.

About the authors

Nita Shah is an Analytics Specialist Solutions Architect at AWS based out of New York. She has been building data warehouse solutions for over 20 years and specializes in Amazon Redshift. She is focused on helping customers design and build enterprise-scale well-architected analytics and decision support platforms.

Sushmita Barthakur is a Senior Data Solutions Architect at Amazon Web Services (AWS), supporting Strategic customers architect their data workloads on AWS. With a background in data analytics, she has extensive experience helping customers architect and build enterprise data lakes, ETL workloads, data warehouses and data analytics solutions, both on-premises and the cloud. Sushmita is based in Florida and enjoys traveling, reading and playing tennis.

Jonathan Katz is a Principal Product Manager – Technical on the Amazon Redshift team and is based in New York. He is a Core Team member of the open source PostgreSQL project and an active open source contributor, including PostgreSQL and the pgvector project.

AWS Weekly Roundup: Strands Agents, AWS Transform, Amazon Bedrock Guardrails, AWS CodeBuild, and more (May 19, 2025)

Post Syndicated from Danilo Poccia original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-strands-agents-aws-transform-amazon-bedrock-guardrails-aws-codebuild-and-more-may-19-2025/

Many events are taking place in this period! Last week I was at the AI Week in Italy. This week I’ll be in Zurich for the AWS Community Day – Switzerland. On May 22, you can join us remotely for AWS Cloud Infrastructure Day to learn about cutting-edge advances across compute, AI/ML, storage, networking, serverless technologies, and global infrastructure. Look for events near you for an opportunity to share your knowledge and learn from others.

What got me particularly excited last Friday was the introduction of Strands Agents, an open source SDK that you can use to build and run AI agents in just a few lines of code. It can scale from simple to complex use cases, including local development and production deployment. By default, it uses Amazon Bedrock as model provider, but many others are supported, including Ollama (to run models locally), Anthropic, Llama API, and LiteLLM (to provide a unified interface for other providers such as Mistral). With Strands, you can use any Python function as a tool for your agent with the @tool decorator. Strands provides many example tools for manipulating files, making API requests, and interacting with AWS APIs. You can also choose from thousands of published Model Context Protocol (MCP) servers, including this suite of specialized MCP servers that help you get the most out of AWS. Multiple teams at AWS already use Strands for their AI agents in production, including Amazon Q Developer, AWS Glue, and VPC Reachability Analyzer. Read it all in Clare’s post.

Strands Agents SDK agentic loop

Last week’s launches
Here are the other launches that got my attention:

Additional updates
Here are some additional projects, blog posts, and news items that you might find interesting:

  • Securing Amazon S3 presigned URLs for serverless applications – Focusing on the security ramifications of using Amazon S3 presigned URLs, explaining mitigation steps that developers can take to improve the security of their systems using S3 presigned URLs, and walking through an AWS Lambda function that adheres to the provided recommendations.
    Architectural diagram.
  • Running GenAI Inference with AWS Graviton and Arcee AI Models – While large language models (LLMs) are capable of a wide variety of tasks, they require compute resources to support hundreds of billions and sometimes trillions of parameters. Small language models (SLMs) in contrast typically have a range of 3 to 15 billion parameters and can provide responses more efficiently. In this post, we share how to optimize SLM inference workloads using AWS Graviton based instances.
    AWS Graviton processors.

Upcoming AWS events
Check your calendars and sign up for these upcoming AWS events:

  • AWS Summits – Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Dubai (May 21), Tel Aviv (May 28), Singapore (May 29), Stockholm (June 4), Sydney (June 4–5), Washington (June 10-11), and Madrid (June 11)
  • AWS Cloud Infrastructure Day – On May 22, discover the latest innovations in AWS Cloud infrastructure technologies at this exclusive technical event.
  • AWS re:Inforce – Mark your calendars for AWS re:Inforce (June 16–18) in Philadelphia, PA. AWS re:Inforce is a learning conference focused on AWS security solutions, cloud security, compliance, and identity.
  • AWS Partners Events – You’ll find a variety of AWS Partner events that will inspire and educate you, whether you’re just getting started on your cloud journey or you’re looking to solve new business challenges.
  • AWS Community Days – Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Zurich, Switzerland (May 22), Bengaluru, India (May 23), Yerevan, Armenia (May 24), Milwaukee, USA (June 5), and Nairobi, Kenya (June 14)

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Danilo

How Smartsheet boosts developer productivity with Amazon Bedrock and Roo Code

Post Syndicated from Rony Blum original https://aws.amazon.com/blogs/architecture/how-smartsheet-boosts-developer-productivity-with-amazon-bedrock-and-roo-code/

This post was co-written with JB Brown from Smartsheet.

Organizations are often seeking ways to enhance developer productivity while maintaining cost-efficiency. AI-powered coding assistants are effective solutions to accelerate development cycles, but implementing these solutions at scale while managing costs presents unique challenges. Roo Code, an AI-powered autonomous coding agent located directly in developers’ editors, represents the latest evolution in AI-assisted development. It helps developers code faster and smarter by offering capabilities ranging from code generation and refactoring to documentation writing and code base analysis through natural language interaction. This post explores how Smartsheet successfully deployed Roo Code with Amazon Bedrock and Anthropic’s Claude, achieving significant improvements in developer efficiency while optimizing costs through innovative caching strategies.

The Amazon Bedrock prompt caching capability, which became generally available in April 2025, represents a significant advancement in optimizing AI model interactions. With this feature, organizations can cache frequently used prompts across multiple API calls, reducing costs by up to 90% and lowing latency by up to 85%. The technology is particularly valuable for applications that repeatedly use similar context, such as coding assistants that need to maintain context about code files. Prompt caching supports multiple foundation models (FMs), including Anthropic’s Claude 3.5 Haiku and Anthropic’s Claude 3.7 Sonnet, as well as the Amazon Nova family of models. The cached context remains available for up to 5 minutes after each access, with each cache hit resetting this countdown. For organizations implementing AI-assisted development workflows, they can use this capability to balance performance with cost-efficiency.

Benefits of Amazon Bedrock prompt caching

Smartsheet is a leading cloud-based enterprise work management service, enabling millions of users worldwide to plan, manage, track, automate, and report on work at scale. Their engineering team identified an opportunity to enhance developer productivity by implementing AI-assisted coding capabilities across their organization. The solution needed to scale efficiently while maintaining reasonable costs, leading them to choose Roo Code with Amazon Bedrock with Anthropic’s Claude as their FM provider.

“Our engineering team has achieved a 60% reduction in operational costs and 20% improvement in response latency when using Roo Code with Amazon Bedrock prompt caching. These improvements directly translate to enhanced developer productivity across our organization,” says JB Brown, VP of Engineering at Smartsheet.

This fact is underscored by the team’s ability to rapidly generate comprehensive code documentation and architecture diagrams in a mere four hours, a task that previously consumed 2 weeks of manual effort. This newfound efficiency extends to critical areas like AWS cost management, where a vital analysis tool was built in just 30 minutes, and is projected to save Smartsheet $450,000 annually. Furthermore, the speed and accessibility of information have been significantly amplified, as evidenced by the 6–8 hours of senior engineer time saved by quickly explaining complex Terraform/Terragrunt infrastructure. These examples highlight how Amazon Bedrock prompt caching is not just about incremental gains, but about unlocking substantial time and resource savings, empowering our developers to focus on innovation rather than time-consuming manual processes.

Solution overview

When implementing AI-assisted coding at scale, Smartsheet faced several key challenges around managing costs associated with large language model (LLM) API calls, reducing latency for developer interactions and handling repetitive elements in prompts efficiently. The development team recognized that a significant portion of the prompts sent to Amazon Bedrock were repeated elements: the system prompt and the continuously building message history, presenting an opportunity for optimization.

To address these challenges, Smartsheet implemented a comprehensive solution centered around Roo Code integration with Amazon Bedrock and prompt caching. Smartsheet deployed Roo Code, powered by Amazon Bedrock and Anthropic’s Claude 3.7, across the organization and integrated seamlessly into existing development workflows. This provided developers with immediate access to AI assistance for code writing, reviewing, and debugging tasks.

The team then contributed a sophisticated Amazon Bedrock prompt caching system to Roo Code that identifies and stores common prompt elements, significantly reducing redundant data transmission to Bedrock. This caching layer proved crucial in optimizing both cost and performance metrics. The caching decision flow incorporates two key checks: first, it verifies if the selected model supports prompt caching, then it makes sure the system prompt meets a minimum token threshold to make caching worthwhile.

The solution integrates Roo Code directly into developers’ integrated development environments (IDE), using the Amazon Bedrock prompt caching capabilities. The architecture separates content into static elements (like system instructions and code context) and dynamic elements (like user queries). When a developer makes a request, static content is automatically marked with cache checkpoints and stored for 5 minutes, with each access refreshing this window.

For subsequent queries, the system checks for matching cached content before processing. When matches are found, it bypasses recomputation of these sections, significantly reducing both response time and costs.

The following diagram shows the integration points between Roo Code and Amazon Bedrock.

Technical workflow diagram showing data flow between user query, cache decision logic, and Amazon Bedrock with response handling

The following screenshot highlights the model provider settings and connection parameters required for implementation.

Amazon Bedrock configuration interface showing AWS authentication, region settings, and Claude model with caching enabled

Optimizing performance with prompt caching

When analyzing code files, much of the context—such as file contents and environment details—remains consistent across multiple queries. By caching this content, subsequent requests can reuse it from the cache, significantly reducing cost and latency.

In our example, we first asked Roo Code to analyze a Python file. We then sent multiple follow-up queries about different aspects of the code. The first query explained the overall structure, followed by questions about specific classes, unused modules, and potential improvements.

The usage section of the responses showed significant improvements through caching. During the first query, the model processed the input and wrote the cached content. For subsequent queries, 90% of the input was served from the cache. Because cache reads are 90% cheaper than processing new input tokens, this resulted in an 83% cost reduction for follow-up queries.

Here’s how the caching worked across queries:

  • First query – The model processed the entire file content and environment details, writing them to cache
  • Subsequent queries – The model loaded the cached content, only processing the new question text

To illustrate this, let’s look at the usage statistics from a series of queries:

Query 1: "Explain my @/src/models/product.py code"
{
    "inputTokens": 1051,
    "outputTokens": 902,
    "totalTokens": 19374,
    "cacheReadInputTokenCount": 0,
    "cacheWriteInputTokenCount": 12421
}

Query 2: "Describe what is ProductUpdate class used for"
{
    "inputTokens": 1078,
    "outputTokens": 694,
    "totalTokens": 21664,
    "cacheReadInputTokenCount": 19892,
    "cacheWriteInputTokenCount": 0
}

Query 3: "Which module is not in use?"
{
    "inputTokens": 4,
    "outputTokens": 774,
    "totalTokens": 22753,
    "cacheReadInputTokenCount": 19892,
    "cacheWriteInputTokenCount": 2083
}

Query 4: "Any improvement to implement to my code making it more efficient?"
{
    "inputTokens": 1097,
    "outputTokens": 1270,
    "totalTokens": 24342,
    "cacheReadInputTokenCount": 21975,
    "cacheWriteInputTokenCount": 0
}

We can see from the results that the first query didn’t return cacheReadInputTokenCount but wrote to the cache the whole input. For subsequent queries, most of the input was read from the cache, except for the specific question.

The results clearly demonstrate the benefits of prompt caching:

  • 70% of total input was served from cache across queries
  • 90% of input for follow-up queries came from cache
  • Overall input costs reduced by 60%
  • Follow-up query costs reduced by 83%

This approach is particularly effective for development workflows where developers frequently query the same code base with different questions. The cached content provides necessary context for each query while significantly reducing processing overhead.

The following screenshot displays Roo Code’s interface, showing detailed metrics including token usage, cache hit rates, and associated API costs. The interface presents the cost savings achieved through caching and provides comprehensive usage analytics.

IDE showing Roo Code analysis with metrics, API costs, and detailed Python code explanation

Results and future innovation

The following line graph demonstrates the cost reduction trend over multiple queries runs, having the first query uncached, and the subsequent input using significant query responses from the cache. The y-axis shows costs in dollars, and the x-axis shows the cost per query. The graph shows a clear downward trend, with costs decreasing by 60% from the baseline.

Dual-axis graph displaying per-query and cumulative costs, demonstrating cost savings through caching over 4 queries

The implementation delivered remarkable improvements across key metrics. The prompt caching system achieved a 60% reduction in operational costs while simultaneously improving response latency by 20%.

The choice of Amazon Bedrock prompt caching proved particularly effective for Smartsheet’s coding assistance workflows. The 5-minute cache Time to Live (TTL) aligns perfectly with the natural rhythm of developer interactions with Roo Code, where multiple related queries typically occur within short timeframes. During active coding sessions, developers engage in rapid back-and-forth exchanges with the AI assistant—reviewing code, asking follow-up questions, and requesting modifications—while maintaining context through the cache. This agentic workflow, where each interaction builds upon previous context, takes full advantage of the prompt caching mechanism—each query refreshes the cache timer while preserving valuable context about the code being discussed.

Best practices and lessons learned

Through their implementation journey, Smartsheet developed several key best practices for organizations looking to implement similar solutions. Early implementation of prompt caching proved crucial, allowing the team to analyze prompt patterns and design efficient caching strategies from the start. The team found that focusing on developer experience through response time optimization and seamless tool integration was essential for successful adoption.

Continuous monitoring and analysis of usage patterns became a cornerstone of their optimization strategy. By tracking key metrics like response times and costs, the team could identify opportunities for further optimization and regularly adjust their caching strategies to maintain optimal performance.

Looking ahead

Smartsheet continues to innovate with Amazon Bedrock and Roo Code, exploring new ways to enhance developer productivity. Their engineering team is investigating advanced caching strategies and evaluating new FMs as they become available on Amazon Bedrock. The success of this implementation has established a strong foundation for future AI-assisted development initiatives.

Conclusion

Smartsheet’s implementation of Roo Code with Amazon Bedrock and prompt caching demonstrates how organizations can successfully deploy AI-assisted coding solutions while maintaining cost-efficiency. Their approach provides a blueprint for others looking to enhance developer productivity through AI while optimizing operational costs. The combination of strategic implementation, innovative caching solutions, and continuous optimization has enabled Smartsheet to achieve significant improvements in performance and cost metrics.

To learn more about implementing AI solutions at scale, refer to the Amazon Bedrock documentation and explore the AWS Machine Learning Blog. The frameworks and strategies outlined in this post can help organizations of varying size implement efficient, cost-effective AI-assisted development workflows.

About the Authors