Tag Archives: Amazon OpenSearch Service

The next generation of Amazon OpenSearch Serverless: Built from the ground up for agents

Post Syndicated from Sohaib Katariwala original https://aws.amazon.com/blogs/big-data/the-next-generation-of-amazon-opensearch-serverless-built-from-the-ground-up-for-agents/

Audience note: This is the deep-dive technical launch post. For a shorter overview of what changed and why, see the related post on the AWS News Blog.

Today, we are announcing a ground-up re-architecture of Amazon OpenSearch Serverless that delivers up to 20 times faster autoscaling, scale to zero, and up to 60% lower cost than provisioning clusters for peak load. Amazon OpenSearch Service is a fully managed, open source retrieval engine that unifies vector, lexical, hybrid, and agentic search, delivering low-latency, accurate and relevant results. Amazon OpenSearch Serverless is an automatically scaled deployment option.

Modern workloads are increasingly dynamic and unpredictable. An ecommerce platform sees a 10x traffic spike during a flash sale. An artificial intelligence (AI) agent triggers hundreds of concurrent vector queries while reasoning through a multi-step task, then goes idle. A multi-tenant SaaS application serves dozens of tenants with wildly different activity patterns. These workloads need infrastructure that scales up to meet demand and releases resources when demand drops.

That is why we rebuilt the Amazon OpenSearch Serverless architecture from the ground up. The new architecture decouples compute from storage. The service provisions infrastructure in seconds instead of minutes, and scales compute all the way to zero when your application is idle. In this post, we walk through the new architecture, what it means for your applications, and how to get started with a hands-on tutorial.

With this launch, Amazon OpenSearch Serverless introduces two named architectures. Existing collections are now referred to as
Classic collections. The new architecture is called
NextGen and is now the default when you create a new collection via the AWS Console. You can use NextGen architecture in the API by specifying
--generation NEXTGEN in the CLI. To continue using the Classic architecture, specify
--generation CLASSIC in the CLI or omit the optional
--generation parameter.

What this means for your applications

The new architecture delivers improvements across three pillars: performance, cost, and a simplified user experience.

Performance: Autoscaling in seconds

An OpenSearch Compute Unit (OCU) is the unit of compute capacity that powers your indexing and search workloads. Amazon OpenSearch Serverless now provisions additional OCUs in seconds. When traffic arrives, the service adds resources in line with demand instead of reacting after a worker is already under pressure. The same mechanism scales the infrastructure back down quickly when traffic drops. The new architecture scales capacity up to 20 times faster than the previous architecture, so your users experience consistent performance during traffic surges, and you stop paying for capacity when you no longer need it.

Cost efficiency: Pay only for what you use

Indexing, search, storage, and Vector Index GPU-Acceleration are metered and billed independently, so you can see and optimize each dimension of your workload separately.

Decoupled compute and storage: OpenSearch Serverless now has full decoupling between compute and storage, allowing OCUs to scale up and down irrespective of the amount of data stored in a collection. This is powered by a new storage layer that is accessible to both indexing and search OCUs. You can now have multiple indices with data indexed in them but not pay any compute costs if you are not actively indexing or searching data. For workloads with significant idle time, the new architecture can reduce infrastructure costs by up to 60% compared to the cost of provisioning OpenSearch Service domains for peak capacity.

Scale to zero: When no requests arrive within the idle timeout window (10 minutes), the service releases compute resources and your OCU usage scales to 0. When traffic resumes, capacity is back in approximately 10 seconds. During this window, the service queues incoming requests and serves them once capacity is available; it does not drop them. If you anticipate a burst of traffic, for example before a scheduled batch job or a marketing campaign, you can send a lightweight query (such as a match_all with size=1) to warm the collection before your application starts sending production traffic. This reduces the latency your users experience on the first real request. Indexing and search scale independently. If you have no search requests, search OCUs scale to zero, even while OpenSearch Serverless maintains indexing OCUs for indexing requests, and vice versa.

GPU acceleration for vector workloads: For vector collections created in the new architecture, OpenSearch Serverless automatically uses GPU-backed compute to accelerate Hierarchical Navigable Small World (HNSW) vector index construction, significantly reducing indexing time compared to CPU-only builds. GPU acceleration kicks in automatically whenever there is an opportunity to leverage GPUs to reduce overall indexing time and cost. In the Classic architecture, you had to opt in or out of GPU acceleration at the collection level through the API. If you want to disable GPU acceleration for NextGen collections for a specific index, you can
turn off the remote index build setting at the index level. GPU usage appears as a separate line item on your bill, so you have full visibility into when acceleration was active and what it cost. For more details on how GPU acceleration works and performance benchmarks, refer to
Build billion-scale vector databases in under an hour with GPU acceleration on Amazon OpenSearch Service.

Simplified experience: Fewer steps to production

We also simplified the day-to-day experience of running OpenSearch Serverless:

With the new architecture, you can provision a collection and start sending requests in seconds. There is no need for capacity planning, no sizing decisions, and no waiting for infrastructure to warm up. This makes Amazon OpenSearch Serverless a natural fit for agentic workloads, where an AI agent can spin up a vector search or retrieval step on demand and expect a response without delay.

To make getting started even faster, we have introduced Express Create on the console. You supply a collection name and a collection type, choose Express Create, and your collection is active in seconds with no upfront network, encryption, or access policies to configure. You can add those later if your workload requires them.

Collection groups and collections can also be created programmatically using the AWS Command Line Interface (AWS CLI) and AWS SDKs. AWS CloudFormation support is coming soon.

The new architecture introduces two endpoint formats on the on.aws domain. The per-collection endpoint (<collectionId>.aoss.<region>.on.aws) works the same way as before with one endpoint per collection. The per-account Regional endpoint (<accountId>.aoss.<region>.on.aws) is new: it serves all of your collections through a single hostname, with the target collection identified in each request using the x-amz-aoss-collection-name or x-amz-aoss-collection-id header. This means one connection pool, one Transport Layer Security (TLS) session, and one endpoint to manage regardless of how many collections you have — a significant improvement for multi-tenant workloads where each tenant maps to its own collection. Both endpoints use standard AWS PrivateLink, so you create virtual private cloud (VPC) endpoints from the VPC console or the EC2 API just like any other AWS service. Private Domain Name System (DNS) is configured automatically, eliminating the Amazon Route 53 Private Hosted Zones, forwarding rules, and custom DNS infrastructure that were required with the original architecture. Cross-VPC, cross-account, and on-premises access all work using standard vpce-* DNS names with no additional setup.

Collection groups are the new unit of organization for your collections. You can share compute capacity across multiple collections with Collection Groups, which reduces cost for smaller collections that have complementary traffic patterns. You can also assign different AWS Key Management Service (AWS KMS) keys to collections within the same group, so you get both cost efficiency and per-collection encryption isolation. Collection groups are required when creating collections with the new architecture.

You also get the benefits of OpenSearch open-source releases without needing to manage versions and upgrades. The service tracks upstream releases automatically.

Amazon OpenSearch Serverless is also available on the Vercel Marketplace, making it straightforward for developers to add search infrastructure directly from their Vercel projects. You can link an existing AWS account through delegated access, or get started through a Limited Scope Account with USD $100 in AWS credit if you are new to AWS.

The integration creates a collection with sensible defaults, scale-to-zero billing, public endpoints, and AWS-managed encryption, and automatically sets connection details as environment variables in your Vercel project. You can choose from Search or Vector Search collection types depending on your use case, whether that is full-text search or semantic and AI-powered search.

How the architecture works

The new Amazon OpenSearch Serverless architecture separates compute from storage entirely. OCUs are stateless and read from and write to a distributed shared storage layer that is accessible to both indexing and search OCUs. The storage layer is designed for high durability, keeping your data available independently of the compute nodes that process it.

Architecture diagram showing OpenSearch Serverless NextGen with stateless indexing and search OCUs reading from and writing to a shared distributed storage layer

This design has two practical consequences:

  1. Fast provisioning. New OCUs start serving requests in seconds because there is no local disk to bootstrap. The OCU mounts the shared storage layer and begins processing immediately.
  2. Efficient scale down. Idle capacity can be released with no impact to your stored data, because the data never lived on the OCU. When traffic subsides, compute resources are released and your cost drops accordingly.

Architecture comparison

The following table summarizes the key differences between the original and new architectures:

Capability Classic Architecture NextGen Architecture
Minimum capacity 2 OCUs (always on) 0 OCUs (scale to zero)
Scaling speed Minutes Seconds
Storage Local storage per compute node Distributed shared storage (decoupled)
Collection organization

Individual collections (Default)

Collection groups (Optional)

Collection groups (required)
Cold start from zero N/A (always on) ~10 seconds
Endpoint Per-collection endpoint Regional endpoint (static per account)
Cost vs. OpenSearch Service domain Baseline Up to 60% lower cost
Scaling speed (vs. Classic) Baseline Up to 20 times faster than baseline

Walkthrough: Create a vector collection and observe scale to zero

In this walkthrough, you create a vector search collection with Express Create, index a few sample documents with embeddings, run a k-nearest neighbor (k-NN) query, and watch the collection scale to zero in Amazon CloudWatch. The entire process takes about 10 minutes.

Prerequisites

  • An AWS account with permissions to create Amazon OpenSearch Serverless collections.
  • AWS Command Line Interface (AWS CLI) configured with appropriate credentials.
  • curl 7.75 or later (for built-in --aws-sigv4 support).

Step 1: Configure security policies

Create encryption, network, and data access policies. These must exist before the collection can be created.

# Create an encryption policy
aws opensearchserverless create-security-policy \
    --name product-vectors-encryption \
    --type encryption \
    --policy '{"Rules":[{"ResourceType":"collection","Resource":["collection/product-vectors"]}],"AWSOwnedKey":true}' \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

# Create a network policy (public access for this tutorial)
aws opensearchserverless create-security-policy \
    --name product-vectors-network \
    --type network \
    --policy '[{"Rules":[{"ResourceType":"collection","Resource":["collection/product-vectors"]},{"ResourceType":"dashboard","Resource":["collection/product-vectors"]}],"AllowFromPublic":true}]' \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

# Get your principal ARN
PRINCIPAL_ARN=$(aws sts get-caller-identity --query 'Arn' --output text)

# Create a data access policy
aws opensearchserverless create-access-policy \
    --name product-vectors-data \
    --type data \
    --policy "[{\"Rules\":[{\"ResourceType\":\"index\",\"Resource\":[\"index/product-vectors/*\"],\"Permission\":[\"aoss:CreateIndex\",\"aoss:DescribeIndex\",\"aoss:UpdateIndex\",\"aoss:DeleteIndex\",\"aoss:ReadDocument\",\"aoss:WriteDocument\"]}],\"Principal\":[\"\${PRINCIPAL_ARN}\"]}]" \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

Note: If you use the AWS console’s Express Create workflow, these policies are created automatically.

Important: After creating the data access policy, wait approximately 30 to 60 seconds for the policy to propagate before making API calls to the collection. If you receive a 403 Forbidden error, wait and retry.

Step 2: Create a collection group and collection

Create a collection group with scale-to-zero capacity limits, then create a vector search collection within it.

# Create a collection group with scale-to-zero enabled (min OCU = 0)
aws opensearchserverless create-collection-group \
    --name product-search-cg \
    --generation NEXTGEN \
    --standby-replicas ENABLED \
    --capacity-limits "minIndexingCapacityInOCU=0,maxIndexingCapacityInOCU=4,minSearchCapacityInOCU=0,maxSearchCapacityInOCU=4" \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

# Create a vector search collection in the group
aws opensearchserverless create-collection \
    --name product-vectors \
    --type VECTORSEARCH \
    --collection-group-name product-search-cg \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

The collection status transitions to ACTIVE within seconds.

Step 3: Create a vector index

Retrieve the collection endpoint and create a k-NN index using 3-dimensional vectors:

ENDPOINT=$(aws opensearchserverless batch-get-collection \
    --names product-vectors \
    --query 'collectionDetails[0].collectionEndpoint' \
    --output text \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2")

awscurl --service aoss --region us-east-2 \
    -XPUT "${ENDPOINT}/items" \
    -H "Content-Type: application/json" \
    -d '{
      "settings": {"index.knn": true},
      "mappings": {
        "properties": {
          "description": {"type": "text"},
          "embedding": {"type": "knn_vector", "dimension": 3,
            "method": {"name": "hnsw", "space_type": "cosinesimil", "engine": "faiss"}}
        }
      }
    }'

Note: If the collection has scaled to zero, the first request might take a few seconds while capacity scales up. If the request times out, wait 10 to 15 seconds and retry.

Step 4: Index sample documents with embeddings

awscurl --service aoss --region us-east-2 \
    -XPOST "${ENDPOINT}/items/_bulk" \
    -H "Content-Type: application/json" \
    -d '
{ "index": { "_id": "1" } }
{ "description": "Wireless noise-cancelling headphones", "embedding": [0.8, 0.2, 0.1] }
{ "index": { "_id": "2" } }
{ "description": "Portable Bluetooth speaker", "embedding": [0.7, 0.3, 0.2] }
{ "index": { "_id": "3" } }
{ "description": "Over-ear studio monitor headphones", "embedding": [0.9, 0.1, 0.05] }
'

Step 5: Run a k-NN query

Search for the two nearest neighbors to a query vector. Wait 30 seconds after indexing to allow the vector index to build before running this query:

awscurl --service aoss --region us-east-2 \
    -XGET "${ENDPOINT}/items/_search" \
    -H "Content-Type: application/json" \
    -d '{
      "query": {
        "knn": {
          "embedding": {
            "vector": [0.85, 0.15, 0.08],
            "k": 2
          }
        }
      }
    }'

The response returns the two most similar items, in this case, the headphone documents whose embeddings are closest to your query vector.

You can also run this query in OpenSearch UI by navigating to your collection in the Amazon OpenSearch Service console and choosing the OpenSearch UI Application URL. Then follow the steps outlined in this blog to create a workspace. Then navigate to Dev Tools and paste and run the following query.

GET items/_search
{
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.85, 0.15, 0.08],
        "k": 2
      }
    }
  }
}

Step 6: Observe scale to zero

After a period of inactivity (no indexing or search traffic), the collection group scales down to 0 OCU. Verify with:

aws opensearchserverless batch-get-collection-group \
    --names product-search-cg \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

In the response, currentCapacity.search.capacityInOcu and currentCapacity.indexing.capacityInOcu will show 0 after the collection has scaled down.

You can also navigate to the Collection groups page in the Amazon OpenSearch Service console. Choose your collection group, then scroll down to the Monitoring section. Here you can see two charts: Indexing capacity (OCUs) and Search capacity (OCUs). After 10 minutes of idle time (no indexing or search requests), both metrics drop to zero, confirming that the service has released all compute resources for your collection.

CloudWatch monitoring charts in the Amazon OpenSearch Service console showing indexing and search capacity dropping to zero OCUs after 10 minutes of idle time

Clean up

To avoid ongoing charges, delete the resources you created in this walkthrough when you are done. Delete the collection first so the collection group becomes empty, then delete the group, then remove the security and access policies.

# Look up the collection ID, then delete the collection
COLLECTION_ID=$(aws opensearchserverless batch-get-collection \
    --names product-vectors \
    --query 'collectionDetails[0].id' \
    --output text \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2")

aws opensearchserverless delete-collection \
    --id "${COLLECTION_ID}" \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

# Look up the collection group ID, then delete the collection group
GROUP_ID=$(aws opensearchserverless batch-get-collection-group \
    --names product-search-cg \
    --query 'collectionGroupDetails[0].id' \
    --output text \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2")

aws opensearchserverless delete-collection-group \
    --id "${GROUP_ID}" \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

# Delete the security and access policies
aws opensearchserverless delete-security-policy \
    --name product-vectors-encryption \
    --type encryption \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

aws opensearchserverless delete-security-policy \
    --name product-vectors-network \
    --type network \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

aws opensearchserverless delete-access-policy \
    --name product-vectors-data \
    --type data \
    --endpoint-url "https://aoss.us-east-2.amazonaws.com" \
    --region "us-east-2"

Upgrading existing collections

To move to the new architecture, create a new collection group and collection, then reindex your data into it. For a step-by-step walkthrough of the reindexing process, refer to Perform reindexing in Amazon OpenSearch Serverless using Amazon OpenSearch Ingestion. Your queries and index mappings remain the same. Only the collection endpoint changes. With the new static Regional endpoint, that is a one-time update.

The new architecture supports SEARCH and VECTORSEARCH collection types. TIMESERIES is not supported at launch.

Conclusion

The new Amazon OpenSearch Serverless architecture is available today. You can create your first OpenSearch Serverless collection in seconds with Express Create, scale it to handle production traffic, and your OpenSearch Serverless compute costs drop to zero when it sits idle.

To learn more:

  1. Amazon OpenSearch Service documentation.
  2. Amazon OpenSearch Service console.
  3. Amazon OpenSearch Service pricing page.

If you have questions or feedback, open a support case or reach out through your AWS account team. We look forward to seeing what you build.


About the authors

Sohaib Katariwala

Sohaib Katariwala

Sohaib is a Senior Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service based out of Chicago, IL. His interests are in all things data and analytics. More specifically he loves to help customers use AI in their data strategy to solve modern day challenges.

Raj Ramasubbu

Raj Ramasubbu

Raj is a Senior Analytics and AI Specialist Solutions Architect at AWS, focused on big data, analytics, and AI/ML. He partners with customers to architect and build highly scalable, performant, and secure cloud-based solutions.

Arjun Nambiar

Arjun Nambiar

Arjun is a Product Manager with Amazon OpenSearch Service. He focuses on ingestion technologies that enable ingesting data from a wide variety of sources into Amazon OpenSearch Service at scale. Arjun is interested in large-scale distributed systems and cloud-centered technologies, and is based out of Seattle, Washington.

Automate root cause analysis across Datadog and Elasticsearch with AWS DevOps Agent

Post Syndicated from Bhuvan Jain original https://aws.amazon.com/blogs/devops/automate-root-cause-analysis-across-datadog-and-elasticsearch-with-aws-devops-agent/

Modern distributed systems route business transactions through dozens of microservices, message queues, and event streams. When a message fails to process or processing exceeds SLA thresholds, troubleshooting requires correlating logs from tools like Elasticsearch, metrics from Datadog, and infrastructure change events in AWS CloudTrail. Correlating these signals manually across heterogeneous backends, each with different query languages, schemas, and time granularities, can take hours per incident and demands deep institutional knowledge of the system topology.

This post shows how AWS DevOps Agent, combined with a custom Model Context Protocol (MCP) server for Elasticsearch and native Datadog integration, automates end-to-end root cause analysis. When a Datadog alert fires, AWS DevOps Agent automatically initiates an investigation, correlates signals across all observability backends, and delivers root cause findings in minutes, without manual intervention.

In this post, we walk through the architecture, configuration steps, and a real-world scenario demonstrating how AWS DevOps Agent dramatically reduces mean time to identify (MTTI) for distributed system failures. DevOps engineers, site reliability engineers (SREs), and operations leaders managing containerized workloads will learn how to implement alert-triggered automated investigations that eliminate manual correlation and accelerate root cause identification in their own environments.

Challenges in correlating telemetry signals at scale

At scale, correlating telemetry signals across distributed systems is a key challenge. A platform processing billions of communications for regulated industries must track every message through its full lifecycle — ingestion, transformation, policy evaluation, archival, and retrieval — across dozens of production clusters, thousands of worker nodes, and terabytes of daily telemetry spread across multiple observability backends. A single message ID can generate log entries across multiple indices, correlated metrics in monitoring systems, and change events in audit trails. When a message goes missing or processing stalls, the operations team must pinpoint which cluster processed it, which log store holds the evidence, whether a recent deployment preceded the failure, and whether the issue is isolated or systemic — all while context-switching across tools with different query languages and data schemas. Before AWS DevOps Agent, this process routinely took hours per incident, and longer for complex multi-service failures.

The core difficulty is not the volume of data. It is the correlation of signals across heterogeneous systems that use different identifiers, different time granularities, and different data schemas. A message ID in Elasticsearch logs must be correlated to:

  • A trace ID in application performance monitoring (APM) systems
  • A pod name and namespace in Kubernetes event logs
  • A container image tag in Amazon Elastic Container Registry (ECR) push events
  • Metric anomalies (error rate spikes, pod restarts, CPU/memory deviations) in Datadog
  • Deployment events captured in AWS CloudTrail logs

Manual correlation requires engineers to maintain mental models of these relationships while executing queries across multiple systems. It is error-prone, non-repeatable, and heavily dependent on institutional knowledge. When the engineer with the deepest system familiarity is unavailable, resolution times increases.

Prerequisites

Complete the following prerequisites before configuring the integrations:

  1. The AWS Command Line Interface (AWS CLI) version 2. For installation instructions, see installing or updating to the latest version of the AWS CLI.
  2. Helm – the Kubernetes package manager used to deploy the sample application.
  3. Kubectl – the Kubernetes command-line tool used to deploy Filebeat and manage cluster resources.
  4. An EKS cluster with Control plane logs enabled.
  5. AWS DevOps Agent Agentspace. For installation instructions, refer to Creating an Agent Space.
  6. Elasticsearch cluster deployed and accessible (EC2-hosted, Amazon OpenSearch Service, or self-managed). Filebeat configured as a DaemonSet to collect pod logs and forward to Elasticsearch.
  7. Datadog account with API key and application key. To create, see API & Applications key.

Solution Architecture

The solution presented in this post combines three integrated components to deliver automated end-to-end message ID traceability:

  • AWS DevOps Agent as the intelligent investigation orchestrator
  • A custom ELK MCP Server providing structured access to Elasticsearch log data
  • Native Datadog integration for metrics, events, and alert-triggered investigations

Together, these components form an autonomous investigation pipeline that activates when an alert fires, correlates signals across all observability sources, builds a topological understanding of the affected services, and delivers a structured root cause analysis, without manual intervention.

In our implementation, application pods are instrumented to emit custom metrics to Datadog, including per-message-ID processing status, trace ID labels, and endpoint-level error counters. This instrumentation provides AWS DevOps Agent with the ability to correlate a specific message ID from an alert payload to its corresponding trace ID in application performance data, a correlation that previously required manual cross-referencing.

Webhook-Based Alert Triggering

A critical aspect of the architecture is the automated triggering of investigations when alerts fire. Rather than requiring manual investigation initiation, the solution configures Datadog alerting webhooks to invoke AWS DevOps Agent directly.

When a Datadog monitor enters an alert state, it fires a webhook to the AWS DevOps Agent endpoint, including:

  • The message ID associated with the processing failure
  • The trace ID for APM correlation
  • The alert timestamp and alert condition
  • The triggering monitor name and severity

AWS DevOps Agent authenticates the webhook using a bearer token and immediately initiates an investigation in the configured Agent Space. The Datadog webhook payload serves as the investigation’s initial context, seeding the agent with the specific identifiers it needs to perform targeted queries rather than broad searches across the full data volume.

Architecture Diagram

Architecture diagram showing the automated root cause analysis pipeline. A Datadog alert fires a webhook to AWS DevOps Agent, which queries three sources: an ELK MCP Server connected to Elasticsearch for log data, native Datadog integration for metrics, and AWS CloudTrail for deployment events. Results flow back to the Agent Space for correlated root cause analysis.
Figure 1:
Automated root cause analysis pipeline — from Datadog alert to AWS DevOps Agent investigation across Elasticsearch, Datadog, and AWS CloudTrail

Implementation Walkthrough

The following walkthrough describes the end-to-end setup required to replicate the message ID traceability solution in your own environment.

Step 1: Configure EKS Cluster Access for AWS DevOps Agent

AWS DevOps Agent requires an access entry in each EKS cluster it will investigate. This enables the agent to describe Kubernetes objects, retrieve pod logs, and access cluster events.

  1. In the AWS DevOps Agent console, navigate to your Agent Space and select the Capabilities tab.
  2. Under the Cloud section, select the primary source and choose Edit. Note the Role Name shown in the Role Name field, this is the IAM role that requires EKS access.
  3. In the Amazon EKS console, select each cluster and open the Access tab.
  4. Under IAM Access Entries, choose Create to add a new access entry.
  5. Set the IAM Principal ARN to the Agent Space role noted in step 2.
  6. Under Access Policies, select AmazonAIOpsAssistantPolicy with Cluster scope. Choose Add Policy, then Next.
  7. Review and create the access entry.

At scale: For environments with 50+ clusters, use the AWS CLI, Terraform, or a GitOps pipeline to automate access entry creation across all clusters. The following CLI command creates an access entry for a single cluster:

aws eks create-access-entry --cluster-name <CLUSTER_NAME> --principal-arn <AGENTSPACE_ROLE_ARN> --region <REGION>
aws eks associate-access-policy --cluster-name <CLUSTER_NAME> --principal-arn <AGENTSPACE_ROLE_ARN> --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonAIOpsAssistantPolicy --access-scope type=cluster --region <REGION>

Step 2: Configure Datadog Integration in AWS DevOps Agent

AWS DevOps Agent includes native Datadog integration. Configuration requires your Datadog API credentials and designates the integration as a data source in the Agent Space. In the AWS DevOps Agent console, navigate to Integrations and choose Add Integration and follow the steps.

After configuration, AWS DevOps Agent can query Datadog metrics, monitors, and events during investigations. Custom application metrics (message throughput, error rates, processing status per message ID) are automatically accessible once the integration is active.

Step 3: Deploy and Configure the Custom ELK MCP Server

The Elasticsearch MCP server bridges AWS DevOps Agent to your self-managed Elasticsearch deployment. The MCP server is deployed as a publicly accessible endpoint with TLS authentication, enabling AWS DevOps Agent to call Elasticsearch APIs securely without requiring direct network access to the actual Elasticsearch/Kibana instances.

Note: For basic Elasticsearch integration, the official Elasticsearch MCP server provides a ready-to-use option. For this use case, we built a custom Python MCP server using FastMCP to expose investigation-specific tools — trace ID correlation, time-window log retrieval, and latency analysis — tailored to our message traceability workflow.

The custom server implementation provides the thirteen tools required for log search, index discovery, and aggregation.

Table: ELK MCP Server tools access by AWS DevOps Agent during investigations
MCP Tool Description
search_logs Search by Lucene query string, time range, and log level
get_error_summary Top recurring errors within a time window
get_recent_logs Fetch most recent log entries from an index
get_logs_by_service Filter logs by service or application name
count_logs_by_level Breakdown of log counts by level (ERROR, WARN, INFO, DEBUG)
get_slow_requests Find requests exceeding a latency threshold
get_logs_around_time Fetch logs within ±N minutes of a specific timestamp
search_by_trace_id Find all logs for a specific trace, request, or correlation ID
get_unique_errors Get distinct error messages in a time window
list_indices List all available Elasticsearch indices with doc counts and health
get_index_stats Get size, document count, and health of a specific index
get_logs_by_host Filter logs by the hostname that sent them via Filebeat
get_logs_by_file Filter logs by the source log file path
  1. Launch an Ubuntu instance (t4g.medium or larger) with a security group allowing inbound TCP 443 from AWS DevOps Agent service endpoints.Note: The single-instance deployment described here is intended for demonstration purposes and does not provide high availability. For production workloads requiring managed infrastructure, automatic scaling, and built-in resilience for your MCP servers, consider using Amazon Bedrock AgentCore Runtime.
  2. Install dependencies and obtain a TLS certificate:
    sudo apt update &&sudo apt install -y python3 python3-venv certbot
    python3 -m venv ~/elk-mcp-venv
    source ~/elk-mcp-venv/bin/activate
    pip install mcp elasticsearch uvicorn starlette
    sudo certbot certonly --standalone -d elk-mcp.yourcompany.com
  3. Create the MCP server. The key architectural decisions are: FastMCP for the Streamable HTTP transport, Starlette middleware for API key authentication, and environment variables for configuration:
    import json, os
    from mcp.server.fastmcp import FastMCP
    from elasticsearch import Elasticsearch
    from starlette.middleware.base import BaseHTTPMiddleware
    from starlette.responses import JSONResponse
    
    ES_HOST = os.environ.get("ES_HOST", "http://localhost:9200")
    API_KEY = os.environ.get("MCP_API_KEY", "YOUR_API_KEY")
    es = Elasticsearch(hosts=[ES_HOST])
    mcp = FastMCP("elk-logs", host="elk-mcp.yourcompany.com")
    
    # API key authentication middleware
    class APIKeyMiddleware(BaseHTTPMiddleware):
        async def dispatch(self, request, call_next):
            if request.headers.get("x-api-key") != API_KEY:
                return JSONResponse({"error": "Unauthorized"}, status_code=401)
            return await call_next(request)
    
    # Example: trace ID correlation tool
    @mcp.tool
    def search_by_trace_id(trace_id: str, index: str, size: int = 100) -> str:
        """Find all logs for a specific request or trace ID."""
        result = es.search(index=index, body={
            "query": {"multi_match": {
                "query": trace_id,
                "fields": ["trace_id", "request_id", "correlation_id", "traceId"]
            }},
            "sort": [{"@timestamp": "asc"}],
            "size": size
        })
        return json.dumps([h["_source"] for h in result["hits"]["hits"]], indent=2)
    
    # ... additional tools follow the same pattern
    
    app = mcp.streamable_http_app()
    app.add_middleware(APIKeyMiddleware)
    
    if __name__ == "__main__":
        import uvicorn
        uvicorn.run(app, host="0.0.0.0", port=443,
            ssl_keyfile="/etc/letsencrypt/live/elk-mcp.yourcompany.com/privkey.pem",
            ssl_certfile="/etc/letsencrypt/live/elk-mcp.yourcompany.com/fullchain.pem")
  4. Start the server:
    sudo -E ES_HOST=http://<ELASTICSEARCH_IP>:9200 MCP_API_KEY=<YOUR_API_KEY> nohup ~/elk-mcp-venv/bin/python ~/elk_mcp_server.py > ~/mcp.log 2>&1 &
  5. Register in AWS DevOps Agent:
    1. In the AWS DevOps Agent console, navigate to Integrations -> Add MCP Integration.
    2. Enter the endpoint URL: https://elk-mcp.yourcompany.com:443/mcp
    3. Enter the API key for authentication.
    4. Verify the integration shows all available tools in the integration detail view.
    5. Add the MCP integration to your Agent Space under the Integrations tab.

Note: The MCP server must be publicly accessible over HTTPS. If your Elasticsearch cluster is in a private VPC, deploy the MCP server with network access to the cluster (e.g., in the same VPC or a peered VPC) while exposing only the MCP server endpoint publicly. Use security group rules to restrict access to AWS DevOps Agent’s known egress IP ranges where possible.

Step 4: Deploy the Application and Configure Filebeat on EKS

Filebeat runs as a Kubernetes DaemonSet on each EKS cluster, collecting pod logs and enriching them with Kubernetes metadata before forwarding to Elasticsearch. The following pipeline configuration ensures that message IDs and trace IDs are preserved as indexed fields, enabling efficient targeted queries during investigations.

  1. Clone the Repository and Build the Container Image:
    # Clone the DevOps agent sample repository
    git clone https://github.com/aws-samples/Amazon-prometheus-bedrock-agent-example.git
    
    # Navigate to the smart demo directory
    cd devops-agent/smart-demo-main/
    
    # Authenticate Docker to your ECR registry
    aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <your-account-id>.dkr.ecr.us-west-2.amazonaws.com
    
    # Build the container image
    docker build -t sample-app .
    
    # Tag the image for ECR
    docker tag sample-app:latest <your-account-id>.dkr.ecr.us-west-2.amazonaws.com/smart-demp:latest
    
    # Push the image to ECR
    docker push <your-account-id>.dkr.ecr.us-west-2.amazonaws.com/smart-demp:latest
  2. Deploy the Application to EKS:
    # Deploy the sample application to EKS using Helm
    helm install sample-app ./helm --set image.repository=<your-account-id>.dkr.ecr.us-west-2.amazonaws.com/smart-demp --set image.tag=latest
  3. Deploy Filebeat as DaemonSet:
    # Navigate to the Filebeat directory
    cd filebeat
    
    # Apply the Filebeat ConfigMap (autodiscovery, JSON parsing, K8s metadata enrichment, Logstash output)
    kubectl apply -f filebeat-configmap.yaml
    
    # Deploy Filebeat as a DaemonSet on every node in the cluster
    kubectl apply -f filebeat-ds.yaml

Step 5: Configure Datadog Webhook for Automatic Investigation Triggering

We will automatically trigger AWS DevOps Agent investigations when Datadog alerts fire. This eliminates the human latency between alert detection and investigation initiation.

Retrieve the AWS DevOps Agent webhook URL and secret:

  1. In the AWS DevOps Agent console, navigate to your Agent Space and open the Capabilities tab.
  2. Under the Webhook section, choose Configure, then Generate webhook.
  3. Save the webhook URL and HMAC secret. These credentials are used to authenticate webhook requests from Datadog.

In Datadog, configure a webhook integration:

  1. Navigate to Integrations -> Webhooks and create a new webhook.
  2. Set the URL to the AWS DevOps Agent webhook endpoint.
  3. Add the Authorization header with the bearer token from step 3.
  4. Configure the payload to include the message ID, trace ID, and alert context:
    {
    "title": "Message Processing Failure - $EVENT_TITLE",
    "description": "$EVENT_MSG",
    "alert_id": "$ALERT_ID",
    "alert_status": "$ALERT_STATUS",
    "timestamp": "$TIMESTAMP",
    "message_id": "$tags.message_id",
    "trace_id": "$tags.trace_id",
    "service": "$tags.service",
    "cluster": "$tags.cluster_name",
    "severity": "$ALERT_PRIORITY"
    }
  5. Associate the webhook with the Datadog monitors that detect message processing failures by adding @webhook-webhook-name to the monitor notification message.

Step 6: Configure Agent Space Skills (Optional but Recommended)

AWS DevOps Agent Skills provide a Retrieval-Augmented Generation (RAG) knowledge base that gives the agent organization-specific context during investigations. Even a brief skills document (2-3 paragraphs) that identifies your application’s purpose, its key components, and its observability backends can reduce AWS DevOps Agent investigation time by helping the agent understand context before executing its first queries.

In our implementation, the skills document described the sample message-processing application, identified Elasticsearch as the logging backend, and identified Datadog as the metrics backend.

Real-World Investigation: Message Processing Failure Diagnosed in 6 Minutes

The following walkthrough and the scenario represent a common class of incident in distributed systems: a silent functional regression introduced through a new container image deployment that causes specific message types to fail processing without immediately obvious symptoms.

The Scenario

The production EKS cluster runs a message-processing application (sample-app) with four HTTP endpoints:

  • /health – Application health check
  • /metrics – Prometheus metrics endpoint
  • /process – Core message processing endpoint
  • /alert – Alert notification endpoint (newly introduced in recent deployment)

A new container image was pushed to Amazon ECR with a new /alert endpoint. However, the endpoint implementation was incomplete when called; it returned an HTTP 404 response and silently dropped the associated message. The Filebeat DaemonSet collected pod logs and sent them into Elasticsearch. Datadog captured application metrics with message ID and trace ID labels. A Datadog monitor detected the elevated error rate and fired.

Phase 1: Alert Fires and Investigation Initiates (T+0:00)

At 14:52:57 UTC, a Datadog monitor detected an elevated rate of failed message processing requests on the /alert endpoint. The monitor fired a webhook to the AWS DevOps Agent endpoint with the alert payload context.

Within 10 seconds, the Agent read its investigation skills (sample-app-incident and triaging-3p-monitoring-alerts) and began planning the investigation approach.

Screenshot of AWS DevOps Agent console showing an investigation automatically initiated via Datadog webhook. The alert payload context includes message ID, trace ID, and alert timestamp. Investigation skills sample-app-incident and triaging-3p-monitoring-alerts are loaded
Figure 2: AWS DevOps Agent investigation initiated automatically via Datadog webhook, showing alert payload context and investigation skills loaded

Phase 2: Cross-Source Signal Correlation (T+0:10 – T+2:30)

AWS DevOps Agent began its investigation by using the message ID from the alert payload as its primary search key. It decided its investigation strategy: extract the trace_id from Elasticsearch using the message_id, get Datadog monitor details, and search for errors around the alert time.

Elasticsearch Log Search
The agent invoked the ELK MCP server to list available indices and identify the relevant log store for the message-processor application. It then executed a targeted search and identified the relevant Elasticsearch index (logs-2026.03.25):

Screenshot of AWS DevOps Agent querying the ELK MCP Server to correlate a message ID to a trace ID across Elasticsearch indices. The agent identifies the relevant log index logs and retrieves matching log entries
Figure 3: AWS DevOps Agent querying the ELK MCP Server to correlate message ID to trace ID across Elasticsearch indices

AWS DevOps Agent surfaced the Message ID to Trace ID correlation without any explicit cross-referencing instruction. It recognized the relationship from the log structure. At T+1:41, the DevOps Agent launched three parallel tasks simultaneously, rather than investigating sequentially.

Screenshot showing the trace ID successfully resolved from the Elasticsearch log structure, triggering three parallel investigation tasks: Datadog metrics correlation, EKS topology analysis, and CloudTrail event correlation.Figure 4: Trace ID successfully resolved from log structure, triggering three parallel investigation tasks

Datadog Metrics Correlation
Using the trace ID extracted from the Elasticsearch logs, the agent queried Datadog for correlated metrics. It retrieved CPU and memory utilization, pod count, restart metrics and enhanced metrics for request latency and errors.

Screenshot of Datadog metrics retrieved by AWS DevOps Agent using the extracted trace ID, showing CPU utilization, memory usage, pod restart counts, and request latency metrics for the affected service.
Figure 5: Datadog metrics correlation showing CPU, memory, pod restarts, and request latency retrieved using the extracted trace ID

AWS EKS Topology & CloudTrail Event Correlation
AWS DevOps Agent queried AWS CloudTrail for deployment and configuration change events in the time window preceding the alert.

Screenshot of AWS CloudTrail event correlation identifying deployment and configuration change events in the time window preceding the alert, including ECR image push and EKS rolling deployment events
Figure 6: AWS CloudTrail event correlation identifying deployment and configuration changes in the alert time window

With the initial timeline established, AWS DevOps Agent examined all endpoints and their response patterns. This revealed that the application was fundamentally healthy, with three of four endpoints returned consistent 200 responses. However, it also revealed the anomaly was isolated to the /alert endpoint, which had never successfully served a request in its observable history.

Screenshot of AWS DevOps Agent endpoint analysis showing three healthy endpoints returning HTTP 200 responses (health, metrics, process) and the alert endpoint returning consistent HTTP 404 failures, isolating the anomaly to the newly deployed alert endpoint.
Figure 7: AWS DevOps Agent endpoint analysis revealing isolated 404 failures on the /alert endpoint while other endpoints remain healthy

Phase 3: Observation Streaming from Parallel Tasks (T+2:30 – T+3:54)

As the three tasks ran simultaneously, observations streamed in chronologically:

T+2:58 – Observation: Anomalous CPU Behavior Signals Pod Disruption

The first sign of trouble came from pod tr8pl. Its CPU usage dropped sharply. Around the same time, two unfamiliar pods (85bx4, 7h4bd) briefly appeared with minimal CPU. Shortly after, three new pods (hclf9, b7bp2, g6gkc) spun up. This pattern of old pods dying, short-lived intermediaries, and fresh containers starting up pointed strongly toward a rolling deployment in progress.

T+3:04 – Observation: ECR Image Push Traced as the Trigger

With the deployment pattern established, the next question was: what initiated it? CloudTrail provided the answer. At 14:57:19 UTC, a user vik**** (via role nht-admin) pushed a new container image to ECR repository sm***demp:latest (ECR image tag). The timeline now made sense:

  1. ECR image push at 14:57:19 UTC
  2. Rolling deployment (pods replaced)
  3. /alert endpoint 404 errors begin
  4. Datadog alert fires

Screenshot showing parallel task observations streaming into AWS DevOps Agent. Anomalous CPU behavior indicates pod disruption from a rolling deployment, and CloudTrail evidence links an ECR image push by user vik at 14:57:19 UTC to the deployment that introduced the failing endpoint.
Figure 8: Parallel task observations streaming into AWS DevOps Agent – anomalous CPU behavior indicating pod disruption and CloudTrail evidence linking the ECR image push to the rolling deployment

Phase 4: Findings Documented – Causal Chain Established (T+3:55 – T+5:08)

At T+3:55, the Agent documented its first formal Finding (elevated from Observation):

Screenshot of AWS DevOps Agent documenting its formal finding, establishing the causal chain: ECR image push triggered rolling deployment, new container image contained incomplete alert endpoint implementation, endpoint returns 404 and drops messages.
Figure 9: AWS DevOps Agent formal finding documenting the causal chain from ECR image push to alert endpoint failure

Phase 5: Root Cause Confirmation and Infrastructure Validation (T+5:10 – T+5:30)

AWS DevOps Agent validated infrastructure health. The complete absence of infrastructure issues, combined with the timeline evidence from CloudTrail and the historical 404 pattern on the /alert endpoint, allowed AWS DevOps Agent to deliver a high-confidence root cause identification.

At 14:58:12 UTC, exactly 5 minutes and 14 seconds (under 6 minutes) after the investigation began, AWS DevOps Agent delivered its root cause analysis:

Screenshot of the final root cause analysis delivered by AWS DevOps Agent at 14:58:12 UTC, 5 minutes and 14 seconds after investigation began. Root cause identified as an incomplete alert endpoint in the newly deployed container image that returns HTTP 404 and silently drops associated messages.
Figure 10: Final root cause analysis delivered by AWS DevOps Agent, identifying the incomplete alert endpoint in the newly deployed container image

Clean-up

Step 1: Delete the AWS DevOps Agent AgentSpace

  1. In the AWS DevOps Agent console, navigate to Agent Spaces.
  2. Select the Agent Space you created for this walkthrough.
  3. Remove all integrations (Datadog, ELK MCP Server) from the Agent Space by navigating to the Capabilities tab and choosing Remove for each.
  4. Choose Delete Agent Space from Actions dropdown and confirm the deletion.

Step 2: Terminate the Microservices and Delete the EKS Cluster

First, remove the application workloads and Filebeat DaemonSet deployed on the cluster:

# Uninstall the sample application Helm release
helm uninstall sample-app

# Delete the Filebeat DaemonSet and ConfigMap
kubectl delete -f filebeat/filebeat-ds.yaml
kubectl delete -f filebeat/filebeat-configmap.yaml

Once the workloads are removed, delete the EKS cluster.

Step 3: Terminate EC2 Instances

Terminate the EC2 instances hosting both the MCP server and the Elasticsearch cluster. You can do this from the AWS Management Console or the AWS CLI:

# Terminate the MCP server EC2 instance
aws ec2 terminate-instances --instance-ids <MCP_SERVER_INSTANCE_ID> --region <REGION>

# Terminate the Elasticsearch EC2 instances
aws ec2 terminate-instances --instance-ids <ELASTICSEARCH_INSTANCE_ID> --region <REGION>

Conclusion

Distributed systems have created a correlation problem that scales faster than the human capacity to solve it. As microservices architectures grow to span dozens of clusters, hundreds of services, and terabytes of daily telemetry, the manual investigation practices that worked at smaller scale become the primary obstacle in maintaining operational quality.

AWS DevOps Agent addresses this challenge at its root by automating the multi-source correlation that previously required experienced engineers working across multiple systems. The combination of native Datadog integration, custom ELK MCP server connectivity, and AWS CloudTrail access enables AWS DevOps Agent to build the complete picture of an incident: from the first metric anomaly, through the log evidence, to the deployment event that caused it. The scenario described in this post demonstrates that a message processing incident that previously consumed hours can be diagnosed to root cause in under six minutes automatically, without manual intervention, and with the full investigation documented for audit and learning purposes.

About the Authors

Bhuvan Jain

Bhuvan Jain

Bhuvan is a Senior Technical Account Manager at Amazon Web Services, supporting independent software vendor (ISV) customers. He is passionate about helping customers build Well-Architected solutions on AWS, with a focus on enterprise-scale networking. As a subject matter expert, Bhuvan offers guidance on designing network architectures that are highly available, resilient, and cost-effective. He holds a Master’s degree in Electrical and Computer Engineering from the University of Illinois at Chicago (UIC). In his free time, he enjoys playing basketball and volleyball, as well as watching movies and TV series.

Vikram Venkataraman

Vikram Venkataraman

Vikram Venkataraman is a Principal Specialist Solutions Architect at Amazon Web Services. He helps customers modernize, scale, and adopt best practices for containerized workloads on Amazon EKS. With the emergence of AI-powered automation, Vikram has been actively working with customers to leverage AWS AI/ML services to solve complex operational challenges, streamline monitoring workflows, and enhance incident response through intelligent automation. He designed and built the POC architecture described in this post and is a co-author of the EKS knowledge graphs blog.

OpenSearch Agent Skills bring built-in intelligence to your agentic IDE

Post Syndicated from Bobby Mohammed original https://aws.amazon.com/blogs/big-data/opensearch-agent-skills-bring-built-in-intelligence-to-your-agentic-ide/

Today, we’re launching OpenSearch Agent Skills, a repository of open, composable skills that bring built-in intelligence to developer workflows with OpenSearch, directly inside your favorite agentic IDE. By embedding OpenSearch expertise into the developer’s existing workflow, Agent Skills reduce setup time, eliminate unnecessary tool-hopping, and let teams focus on building rather than configuring.

Developers today can go from idea to working prototype in minutes using agentic IDEs like Claude, Cursor, and Kiro. They can spin up applications, generate APIs, and build end-to-end workflows with a prompt. But whether you’re experimenting with a new idea, building a POC, or running production systems, the experience quickly becomes more complex. For example, improving relevance in OpenSearch still requires deep expertise in query Domain-Specific Language (DSL), ranking logic, and hybrid search tuning. Troubleshooting latency or cluster health issues often means manually piecing together signals from logs, traces, shards, and infrastructure metrics. Even migrations from Elasticsearch or Solr can become complex and time-consuming because of schema conversion, compatibility gaps, and performance optimization challenges. As AI agents become a primary interface for building and operating applications on OpenSearch, a deeper gap emerges. Translating high-level intent into query DSLs, index configurations, and multi-step workflows still requires significant expertise. At the same time, workflows remain fragmented across domains like search, logs, and observability, forcing teams into siloed tooling and disconnected reasoning. The result is repeated trial-and-error, lack of standardized approaches, and slower time-to-value, despite the promise of faster development.

What are Agent Skills?

Agent Skills, developed by Anthropic, are a lightweight, open format for extending AI agent capabilities with specialized knowledge and workflows. They’re supported by a growing number of AI tools and agentic clients, including Kiro, Claude Code, Cursor, VS Code, GitHub Copilot, Codex and others.

At their core, Agent Skills are pre-built intelligence you can call, extend, and reuse. Each skill encapsulates domain knowledge, execution logic with multi-step workflows, and guidance with explainability, so you not only get results but understand how they’re achieved. Instead of stitching together tools and writing custom logic, you can invoke a skill to handle an entire task, from analysis to recommendation to execution.

At launch, OpenSearch Agent Skills introduces three foundational skills designed to address some of the most common and complex developer workflows: Search, Logs, and Solr to OpenSearch Migrations.

Search skill

The Search Skill builds on the foundation introduced by OpenSearch Launchpad, and brings an agentic, intent-driven experience to building and optimizing search applications with OpenSearch. Developers can go from a simple requirement or sample document to a fully working search application in minutes, whether lexical, semantic, hybrid, or agentic, with no

deep OpenSearch expertise required.

What it does:

  • Translates natural language requirements or sample data into search configurations.
  • Automatically creates index mappings, ingest pipelines, and ML model integrations.
  • Sets up keyword, semantic, and hybrid search capabilities out of the box.

Example

Build a semantic search application for product documentation

Output:

  • Fully configured OpenSearch index with optimized mappings.
  • Integrated embedding models and ingest pipeline.
  • Working search experience (API + UI) ready to test and iterate.

The Search Skill builds on the foundation introduced by OpenSearch Launchpad, extending the same capabilities into an agent-native workflow. You can move from idea to a production-ready search application in minutes, eliminating manual setup and accelerating both prototyping and deployment in OpenSearch.

Logs skill

The Log Skill analyzes log data and investigates distributed traces directly within OpenSearch, bringing agentic intelligence to observability workflows. Instead of manually crafting PPL queries or piecing together trace data across services, developers can express their intent and let the skill

handle the complexity.

What it does:

  • Queries and analyzes log data using PPL, including error patterns, log volume trends, and anomaly detection.
  • Investigates distributed traces, identifying slow spans, error spans, service dependencies, and agent invocations.
  • Correlates logs and traces using traceId to surface root causes across the full observability stack.

Example:

Investigate why my service is returning 500s and correlate with recent traces

Output:

  • PPL query results surfacing error patterns and log volume anomalies.
  • Trace analysis identifying slow or failing spans and service dependencies.
  • Correlated view linking log errors to specific trace IDs for faster root cause analysis.

With the Logs Skill, you can move from a vague symptom to a pinpointed root cause in minutes without needing to master PPL syntax or manually navigate trace data.

Solr to OpenSearch migration skill

The Migration Skill streamlines the complex process of migrating from Solr to OpenSearch. Migrations typically involve cluster discovery, compatibility checks, schema translation, data movement, and validation. These steps often require deep expertise and manual coordination. The

Migration skill turns all these steps into a guided, automated workflow.

What it does:

  • Discovers and analyzes source clusters, including indices, mappings, and configurations.
  • Performs compatibility assessment and highlights breaking changes or required transformations.
  • Translates schemas, index settings, and queries into OpenSearch-compatible formats.

Example:

How can I migrate from Solr to OpenSearch?

Output:

  • Detailed migration plan with compatibility report and required changes.
  • Translated index mappings and configurations ready for OpenSearch.
  • Executed data migration pipeline with progress tracking.
  • Validation report confirming data integrity and query parity between source and target.

With the Migration Skill, developers can move from a fragmented, high-risk migration process to a structured, automated workflow. This approach provides faster transitions, reduced downtime, and confidence in production readiness.

How it works

OpenSearch Agent Skills are organized as a tree of SKILL.md files, structured by domain category. Rather than one monolithic skill that loads everything, the repo is broken into focused, independently installable skills. Each skill is small enough to stay within a tight context window, but

complete enough to handle real end-to-end workflows.

The top-level structure currently groups skills into three categories:

  • Search: opensearch-launchpad for building BM25, semantic, and hybrid search applications from scratch.
  • Observability: log-analytics for PPL-based log querying and error analysis, and trace-analytics for distributed trace investigation and span analysis.
  • Cloud: aws-setup for deploying to Amazon OpenSearch Service (managed) or Amazon OpenSearch Serverless, with separate manifests for each.

Each skill bundles everything the agent needs: step-by-step workflows, reference docs (like PPL syntax guides and CLI references), and executable scripts that run directly against your cluster.

When you say “build a hybrid search app” or “why is my service throwing 500 errors?”, the agent activates only the matching skill, follows its instructions, and executes the right OpenSearch APIs. It returns results alongside clear explanations of what was configured and why. Because skills load on demand, you can have the full collection installed without bloating your agent’s context window.

We’re continuously expanding the skill library. Categories like Dashboard and Migration are already on the roadmap, with more to come as the ecosystem grows.

Getting started

Getting started with OpenSearch Agent Skills is straightforward. No MCP server or extras are required. Skills are installed using npx skills and work directly with your existing agentic IDE.

Prerequisites:

  • Python 3.11+ and uv.
  • Docker installed and running.
  • AWS credentials configured (optional, for cloud deployment).

Install all skills:

npx skills add opensearch-project/opensearch-agent-skills

Or install a specific skill: (e.g. opensearch-launchpad)

npx skills add opensearch-project/opensearch-agent-skills@opensearch-launchpad --full-depth

npx skills add opensearch-project/opensearch-agent-skills@log-analytics --full-depth

npx skills add opensearch-project/opensearch-agent-skills@trace-analytics --full-depth

npx skills add opensearch-project/opensearch-agent-skills@migration-companion --full-depth

Once installed, simply express your intent to your agent, for example, “I want to build a semantic search app with OpenSearch,” and the agent reads the skill instructions and runs the scripts automatically.

Skills can also be installed to a specific agent (-a claude-code), globally across all projects (-g), or to all detected agents (--all). Explore available skills before installing with --list.

Looking ahead

This is just the beginning. We’re actively expanding the OpenSearch Agent Skills ecosystem with new capabilities across advanced relevance tuning, cost-aware performance optimization, index lifecycle and schema evolution, and cross-domain workflows that unify search, logs, and analytics.

Over time, we see Agent Skills becoming a community-driven knowledge layer across OpenSearch domains where solving a complex problem once means everyone benefits. More importantly, Agent Skills mark a fundamental shift in how developers build and operate with OpenSearch: moving away from manual, fragmented workflows toward intelligent, reusable capabilities that guide, optimize, and accelerate development at every stage.

Get involved

OpenSearch Agent Skills is designed to be an open, evolving ecosystem, and we’re getting started. Here’s how you can participate:

  • Try it in your workflow. Install the skills in Claude, Cursor, or Kiro and start interacting with OpenSearch using natural language. Build new applications, investigate issues, or run migrations, and see how far intent-driven workflows can go.
  • Build and extend skills. Agent Skills are intentionally modular and extensible. Create your own skills to encode domain-specific workflows, internal best practices, or repeatable operational playbooks. Whether it’s a custom relevance tuning flow or a specialized observability pipeline, your contributions can become reusable intelligence for others.
  • Contribute to the ecosystem. We welcome contributions across all levels, from improving documentation and fixing bugs to adding entirely new skills. If you’ve solved a complex problem with OpenSearch, consider turning it into a skill and contribute to the Git repo.
  • Share feedback and ideas. Let us know what worked, what didn’t, and what capabilities you’d like to see next, whether it’s deeper integrations, new domains, or more advanced automation.
  • Join the conversation. Engage with the OpenSearch community through GitHub discussions, community forums, and working groups. Collaborate with others building similar workflows and help define the future of agent-driven search and observability.

With OpenSearch Agent Skills, we’re moving toward a world where developers don’t only use tools but use shared intelligence. If that resonates with you, we’d love for you to be part of the journey.

Star and get involved in the OpenSearch Agent Skills repo. Join the conversation on the OpenSearch community forum and connect with us in the OpenSearch Slack channel.

Acknowledgments

We would like to extend our sincere gratitude to the following contributors for their valuable contributions to this project Arjun kumar Giri, Sarat Vemulapalli, Chenyang Li, Fen Qin, Janelle Arita, Kaituo Li, Krishna Kondaka, Owais Kazi, Peter Zhu and Zhichao Geng. Your dedication, expertise, and collaborative spirit have been instrumental in making this project successful. Thank you for your time and contributions.


About the authors

Bobby Mohammed

Bobby Mohammed

Bobby is a Principal Product Manager at AWS, leading product initiatives at the intersection of Search, Generative AI, and Agentic AI. His work focuses on next-generation intelligent applications, from retrieval-augmented generation (RAG) to agent-driven workflows and long-term memory systems. Previously, he helped build foundational AI and data capabilities on Amazon SageMaker, spanning data, analytics, and machine learning at scale. Prior to AWS, he served as Director of Product at Intel, leading deep learning training and inference platforms powering high-performance AI infrastructure. Bobby holds an MBA from the Kellogg School of Management at Northwestern University, and Master’s and Bachelor’s degrees in Electrical Engineering.

Sean Zheng

Sean Zheng

Sean is a Senior Engineering Manager at AWS, where he leads ML/GenAI and search relevancy components within AWS OpenSearch. His team owns plugins including ML Commons, Neural Search, and Search Relevancy Workbench, serving as the primary driver of ML and agentic capabilities for OpenSearch. Recent deliveries under his team include Agentic Search, Agentic Memory, and a Python-based agentic service. Prior to his role with AWS OpenSearch, Sean worked across multiple teams in Amazon’s retail organization, focusing on machine learning and data analytics. His experience spans Core ML, Product Graph, and Search Engine Optimization teams. Sean holds a PhD degree in Computer Science from State University of New York.

Detect and resolve HBase inconsistencies faster with AI on Amazon EMR

Post Syndicated from Yu-Ting Su original https://aws.amazon.com/blogs/big-data/detect-and-resolve-hbase-inconsistencies-faster-with-ai-on-amazon-emr/

HBase operations teams spend hours manually correlating logs, metadata, and consistency reports to identify root causes. Traditional approaches require deep expertise and extensive investigation across scattered data sources, directly impacting MTTR and operational efficiency. As HBase deployments scale and expertise becomes increasingly scarce, organizations face mounting pressure to maintain service reliability while managing growing operational complexity. The manual nature of troubleshooting creates bottlenecks that delay incident resolution, increase operational costs, and risk service degradation during critical business periods.

In this post, we show you how to build an AI-powered troubleshooting solution using Amazon OpenSearch Service vector search and intelligent analysis. This solution reduces HBase inconsistency resolution from hours to minutes and root cause identification from days to hours through natural language queries over operational data. This democratizes HBase troubleshooting capabilities across teams and reducing dependency on specialized expertise.

Solution overview

The solution addresses HBase troubleshooting challenges through data processing, vector search, and AI-powered analysis. It processes operational data from Amazon EMR clusters, generates semantic vector embeddings, and enables natural language queries for intelligent troubleshooting.
Key components include:

  • Amazon EMR HBase: Runs HBase workloads with Amazon S3 as the HBase rootdir for durable, scalable storage
  • Data Processing: Extracts and processes HBase logs, HBCK reports, and metadata with vector embeddings
  • Amazon OpenSearch Service: Provides vector search capabilities with k-NN algorithms for semantic analysis
  • AI Analysis Interface: Enables natural language queries with context-aware recommendations
  • Custom Knowledge Base: Supports organization-specific runbooks and troubleshooting procedures by ingesting Git repositories via Kiro CLI‘s /knowledge add command, enabling the AI assistant to reference custom operational guides alongside HBase source code and operational tools

AWS cloud architecture diagram showing an HBase log analysis system with EMR cluster, VPC networking, IAM roles, Lambda functions, OpenSearch domain, and supporting services for scalable log processing and analytics.

The preceding diagram illustrates how the HBase log analysis system troubleshoots inconsistencies through automated workflows across AWS services.

When an operations team needs to investigate HBase issues, the engineer connects over SSH to the Amazon EMR primary node and runs the error collection script, which gathers logs from HBase master and RegionServer nodes and uploads them to Amazon S3. Next, the engineer connects to the Analytics Amazon Elastic Compute Cloud (Amazon EC2) instance and executes the automated processing script, which downloads logs from Amazon S3, generates semantic vector embeddings, and injects them into Amazon OpenSearch Service for k-NN-based semantic search. The engineer then queries the Kiro CLI AI Assistant using natural language to investigate. Kiro searches Amazon OpenSearch Service for relevant log entries and uses Amazon Bedrock to analyze patterns, correlate errors across components, and provide actionable recommendations. This reduces troubleshooting time from hours to minutes. The system operates within an Amazon Virtual Private Cloud (Amazon VPC) with private subnets for Amazon EMR and Analytics Amazon EC2, AWS Identity and Access Management (AWS IAM) roles for access control, Parameter Store for configuration, and Amazon CloudWatch for monitoring.

Prerequisites

For this walkthrough, you need the following prerequisites:

AWS account setup

  • An AWS account with administrative access for initial deployment
  • AWS Command Line Interface (AWS CLI) configured with administrative credentials

Required AWS IAM permissions

For infrastructure deployment

Your deployment user or role needs the following permissions:

  • Your deployment user or role requires sufficient access to AWS CloudFormation, Amazon S3, AWS IAM, and AWS System Manager.
  • The user or role must have the ability to create AWS CloudFormation stacks.

Infrastructure deployment:

  • For infrastructure deployment, you need AWS CloudFormation stack management permissions.
  • You also require sufficient access to create and manage the following resources:
    • Amazon OpenSearch Service domains
    • Amazon EC2 instances, Amazon VPCs, security groups, and networking components
    • AWS IAM roles and policies
    • AWS Systems Manager Parameter Store entries
    • Amazon CloudWatch Logs groups
    • Amazon S3 bucket for access logs and session logs

Runtime service roles

The AWS CloudFormation stack automatically creates two specialized AWS IAM roles designed with least-privilege access principles.

The first role is the Amazon OpenSearch Service Role, which manages Amazon VPC networking and Amazon CloudWatch logging for the Amazon OpenSearch Service domain.

The second role is the Application Role, which provides minimal Amazon OpenSearch Service and Amazon S3 access specifically for log processing applications and secure log ingestion operations.

Network requirements

  • Amazon VPC with private subnets for secure Amazon OpenSearch Service deployment
  • NAT Gateway for outbound internet access from private subnets
  • Security groups configured for HTTPS-only communication

Running Kiro CLI on Amazon EC2

Kiro platform requirements:

Kiro subscription

  • Active Kiro License: Valid subscription to Kiro platform
  • User Account: Registered Kiro user account with appropriate permissions
  • API Access: Kiro API keys or authentication tokens for CLI access

AWS Identity Center integration

  • AWS IAM Identity Center Setup: AWS IAM Identity Center enabled in your AWS organization
  • Permission Sets: Configured permission sets for Kiro users with appropriate AWS access
  • User Assignment: Users assigned to relevant AWS accounts and permission sets
  • SAML/OIDC Configuration: Identity provider integration if using external identity systems

Additional prerequisites

  • Python 3.7+ and Node.js installed locally
  • Python 3.11+ for AWS Lambda runtime environment (required for OpenSearch MCP server compatibility)
  • Sufficient service quotas for Amazon OpenSearch Service instances and Amazon EC2 resources
  • Recommended access to the analysis instance via AWS Systems Manager Session Manager (recommended). Amazon EMR clusters running HBase workloads
  • EMR_EC2_Default_Role of Amazon EMR EC2 instance profile can execute describe-stacks on AWS CloudFormation stacks in us-east-1
  • Basic familiarity with HBase operations

The deployment follows AWS security best practices with resource-specific permissions, regional restrictions, and encrypted data storage. All AWS IAM policies implement least-privilege access patterns to help secure operation of the log analysis pipeline.

Walkthrough

This walkthrough demonstrates deploying and configuring the AI-powered HBase troubleshooting solution in five key steps:

  1. Deploy AWS infrastructure using AWS CloudFormation
  2. Configure Amazon EMR analysis log collection
  3. Process and index HBase data
  4. Enable AI-powered analysis
  5. Add custom knowledge base (optional)

The complete solution is available in our GitHub repository.

Step 1: Deploy the infrastructure

Deploy the required AWS infrastructure including Amazon OpenSearch Service domain, Amazon EC2 instances, and AWS IAM roles.

To deploy the infrastructure

  1. Deploy AWS CloudFormation stack. Please update [email protected] to an email address for security alerts and Advanced Intrusion Detection Environment (AIDE) reports:
# Deploy to development environment
aws cloudformation create-stack \
  --stack-name dev-hbase-log-analysis \
  --template-body file://cloudformation/hbase-log-analysis-simple.yaml \
  --parameters \
    ParameterKey=EnvironmentName,ParameterValue=dev \
    ParameterKey=EC2InstanceType,ParameterValue=m7g.xlarge \
    ParameterKey=SecurityAlertEmail,[email protected] \
  --capabilities CAPABILITY_IAM \
  --region us-east-1
# Wait for deployment to complete (~15-20 minutes)
aws cloudformation wait stack-create-complete \
  --stack-name dev-hbase-log-analysis \
  --region us-east-1
  1. Note the deployment outputs including Amazon OpenSearch Service endpoint and Amazon EC2 instance details in the AWS CloudFormation console.

AWS CloudFormation stack outputs table displaying infrastructure resource identifiers including IAM roles, EC2 instances, security groups, S3 buckets, OpenSearch domain configuration, and VPC details for an HBase log analysis application in the development environment.

The deployment creates:

  • Amazon OpenSearch Service domain with vector search capabilities
  • Amazon EC2 instance for data processing and AI analysis
  • AWS IAM roles with appropriate permissions
  • Security groups and Amazon VPC configuration

Step 2: Connect to Amazon EC2 instance and set up system

Connect to the Amazon EC2 instance using AWS Systems Manager (SSM) and set up the required components.

To connect and set up the system

  1. Run the following commands to get the instance ID from AWS CloudFormation outputs and connect via AWS Systems Manager (SSM):
# Get instance ID
INSTANCE_ID=$(aws cloudformation describe-stacks \
  --stack-name dev-hbase-log-analysis \
  --query 'Stacks[0].Outputs[?OutputKey==`EC2InstanceId`].OutputValue' \
  --output text \
  --region us-east-1)
# Connect via SSM
aws ssm start-session --target $INSTANCE_ID --region us-east-1

Terminal screenshot showing AWS CLI commands to retrieve an EC2 instance ID from CloudFormation stack outputs and establish an AWS Systems Manager Session Manager connection to the instance in the us-east-1 region.

  1. Clone the repository and run automated setup:
# On EC2 instance
sudo su - ec2-user

# Re-install aws cli
sudo dnf remove awscli -y

# For ARM64 (Graviton instances - default)
curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"

# For x86_64 (if using non-Graviton instances)
# curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"

unzip awscliv2.zip
sudo ./aws/install

# update $PATH in ~/.bashrc
echo 'export PATH=$PATH:/usr/local/bin/' >> ~/.bashrc

# Reload ~/.bashrc
source ~/.bashrc

# Fork and clone the source code repository on GitHub: sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro
git clone https://github.com/YOUR_USERNAME/sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro.git hbase-analysis
cd hbase-analysis

# Run automated setup
chmod +x ./scripts/setup/automated-system-setup.sh
./scripts/setup/automated-system-setup.sh \
  --emr-version emr-7.12.0 \
  --stack-name dev-hbase-log-analysis \
  --region us-east-1

The automated setup script installs:

  • System dependencies (awscli, git, unzip)
  • uv package manager and OpenSearch MCP Server
  • Kiro CLI and configuration with AWS IAM Identity Center authentication. The script will automatically add Apache HBase open source repo and Apache HBase open source operational tools to knowledge bases
  • HBase source repositories for your Amazon EMR version
  • Python dependencies and MCP server configuration
  1. Add your own knowledge base to Kiro CLI

To enhance Kiro CLI’s analysis capabilities with Apache HBase open-source repositories, your organization’s HBase runbooks and troubleshooting guides, you can add your own knowledge base repositories. Here are the commands. Please periodically validate and maintain your runbook contents so that they remain accurate and up-to-date, reflecting any changes in your HBase environment, configurations, or operational procedures.:

# Navigate to the HBase repositories directory
cd /opt/hbase-repositories
# Clone your organization's HBase runbook repository
git clone <runbook-repository-url> <your-own-runbook-repo>
# Example:
# git clone https://github.com/your-org/hbase-runbooks.git hbase-runbooks
# git clone https://gitlab.company.com/ops/hbase-troubleshooting.git hbase-troubleshooting
# Add your custom repositories to Kiro CLI knowledge base manually (run these commands inside kiro-cli):
echo "/knowledge add --name \"Your custom HBase knowledge base\" --path /opt/hbase-repositories/<your-own-runbook-repo>" | kiro-cli
# Example:
# echo "/knowledge add --name \"Company HBase runbooks\" --path /opt/hbase-repositories/hbase-runbooks" | kiro-cli
# echo "/knowledge add --name \"HBase troubleshooting guides\" --path /opt/hbase-repositories/hbase-troubleshooting" | kiro-cli

Step 3: Configure Amazon EMR log analysis collection

Set up data collection from your Amazon EMR clusters to gather HBase logs, metadata, and consistency reports using the recommended direct collection method.
To configure Amazon EMR log analysis collection

  1. On your Amazon EMR cluster primary node, run the following commands to download the collection scripts:
# On EMR primary node
sudo su - hadoop

# Fork and clone the source code repository on GitHub: sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro
git clone https://github.com/YOUR_USERNAME/sample-emr-hbase-inconsistencies-detection-recovery-mcp-kiro.git hbase-analysis
cd hbase-analysis
  1. Run the interactive collection wizard:
# Run collection wizard
python3 scripts/utilities/emr_log_collection/emr_cluster_wizard_v2.py

Input the parameters like the EMR cluster’s jobflow ID, the log analysis Amazon S3 bucket name, and the lookback hours. The default value of the lookback hours is 4 hours.

Terminal screenshot of EMR Cluster Log Collection Wizard V2 showing an interactive command-line interface for configuring HBase diagnostic log collection from Amazon EMR clusters, with step indicators, input fields for job flow ID and S3 bucket, validation confirmations, and lookback hour configuration.

  1. The collection wizard performs these actions:
  • Collects HBase logs from local filesystem. Please reference to prerequisites for the access permission.
  • Runs sudo -u hbase hbase hbck -details (or hbck2 for HBase 2.x)
  • Runs hdfs dfs -ls -R /hbase or aws s3 ls <hbase-root-dir> –recursive
  • Runs hbase shell <<< 'scan "hbase:meta"'
  • Creates properly named files matching analysis system requirements
  • Uploads to Amazon S3 with correct naming conventions

Here’s the data collection summary:

Terminal screenshot showing EMR Cluster Log Collection Wizard V2 completion summary with job flow ID, S3 bucket location, 4-hour lookback period, green success confirmation message, S3 file path, and detailed listing of seven collected diagnostic files including HBCK reports, HBase meta table scans, root directory paths, process information, log collection summary, node logs from all servers, and collection metadata in JSON format.

You can check the uploaded contents through AWS CLI.

aws s3 ls s3://<log-path> --recursive

Here’s a screenshot of the outputs.

Terminal screenshot showing AWS CLI command output listing HBase diagnostic files and logs collected from an EMR cluster and stored in Amazon S3, displaying timestamps, file sizes, and complete S3 object paths including diagnostics directory with HBCK reports, meta table scans, root directory listings, process information, and logs directory with compressed application logs from HBase master and regionserver nodes.

  1. On the Analysis Amazon EC2 instance, download collected files to the Analysis Amazon EC2 instance.
# On analytics EC2 instance
sudo su - ec2-user

# Download logs from S3
mkdir -p /tmp/hbase-log-analysis
cd /tmp/hbase-log-analysis
aws s3 sync s3://<S3-BUCKET-NAME>/emr-logs/<EMR-JOBFLOW-ID>/ .

You can get your jobflow ID from Amazon EMR console:

Amazon EMR clusters management dashboard displaying a table with clusters, showing one cluster entry named "test" in waiting status with green indicator, creation time, elapsed time, normalized instances, along with filter controls, search functionality, pagination showing page 1, and action buttons for View details, Terminate, Clone, and Create cluster operations.

The generated files (hbase-hbase-master-ip-xxx-xxx-xxx-xxx.ec2.internal.log.gz, hbase-hbase-regionserver-ip-xxx-xxx-xxx-xxx.ec2.internal.log.gz, hbck_report.txt, hbase_rootdir_paths.txt, hbase_meta.txt, hbase_processes.txt, log_copy_summary.txt) should be aligned with the automated processing script requirements as following.

Terminal screenshot showing recursive ls -lRt command output listing HBase diagnostic files and logs in /tmp/hbase-log-analysis/ directory, displaying file permissions, ownership by ec2-user, file sizes, timestamps, and complete directory structure including diagnostics directory with text files (manifest.json, HBCK report, meta table scan, process information, root directory paths, log copy summary), logs directory with nested nodes subdirectory containing redacted instance IDs, and applications/hbase subdirectories with compressed RegionServer and Master log files.

Step 4: Process and index data

Process the collected HBase data and create vector embeddings for intelligent search capabilities.To process and index the data, please navigate to the project directory on the Analysis EC2 instance, and run automated-log-processing.sh:

sudo su – ec2-user
cd ~/hbase-analysis
chmod +x ./scripts/processing/automated-log-processing.sh
./scripts/processing/automated-log-processing.sh \
  --job-flow-id j-YOUR-JOB-FLOW-ID \
  --stack-name dev-hbase-log-analysis

The processing scripts extract and parse HBase logs and generate dimensional vector embeddings from HBase log messages using sentence transformer models to enable semantic search beyond keyword matching. The system uses the all-MiniLM-L6-v2 model by default (producing 384-dimensional embeddings), but supports configurable models with different embedding dimensions, automatically adapting the OpenSearch vector index to match the chosen model’s output. The system processes comprehensive HBase operational data including region operations, compaction activities, Write-Ahead Log events, memstore operations, and cluster management information from HMaster and RegionServer logs. Vector embeddings capture error messages, exception stack traces, performance warnings, and multi-line log entries through intelligent text preprocessing. This semantic representation enables advanced troubleshooting where users can query conceptually for “region server performance issues” or “memory pressure” and receive contextually relevant results across different log files and time periods. The vector search capabilities support error correlation by grouping similar exceptions, performance analysis by identifying related bottlenecks, and operational pattern recognition. Each log entry is stored in Amazon OpenSearch Service with original metadata (timestamp, log level, source file, job flow ID) alongside the embedding vector, enabling both structured queries and AI-powered semantic analysis. This approach transforms raw HBase logs into a searchable knowledge base supporting anomaly detection, trend analysis, and predictive insights for proactive cluster management and troubleshooting.

All scripts use AWS IAM authentication automatically. Here’s a screenshot of the data processing outputs.

Terminal screenshot showing successful completion of HBase log analysis processing, green checkmark, confirmation message "Successfully processed 4 file(s)", and next steps section displaying three numbered instructions with redacted URLs for accessing OpenSearch Dashboards, starting Kiro CLI for AI-powered analysis, and querying data using job flow ID, followed by troubleshooting documentation references for HBase inconsistency analysis and log analysis guides.

Step 5: Enable AI-powered analysis

Configure the AI analysis interface to enable natural language queries against your HBase operational data.

To set up AI-powered analysis

  1. Launch Kiro CLI (already configured by automated setup):

kiro-cliCheck mcp and knowledge bases. /mcp list

Terminal screenshot showing MCP list command output displaying one configured MCP server named "opensearch-mcp-server" with command "uvx" in green and white text on dark background with pink shell prompt, featuring a purple "Configured MCP Servers" header with checkbox icon and green horizontal separator line.

/knowledge show

Terminal screenshot showing "/knowledge show" command output displaying Agent kiro_default's knowledge base with repositories: Apache HBase source code, and HBase operational tools

If you cannot see these 2 knowledge bases, you can manually add them through the following commands:

# Note: Large repositories (~500MB) may take a while to index. Check progress with: /knowledge show
/knowledge add --name "HBase operational tools" --path /opt/hbase-repositories/hbase-operator-tools"
/knowledge add --name "Apache HBase source code" --path /opt/hbase-repositories/hbase"
  1. Use natural language queries to analyze your HBase data. The AI analysis uses both the OpenSearch MCP Server for querying indexed data and the Filesystem knowledge bases for accessing HBase source code. You can add your custom runbooks for Kiro’s reference as well.

For HBase inconsistency analysis:

# HBase Inconsistency Detection and Remediation Guidelines
## Search Strategy
- Use fuzzy search for case variations/typos, term query for exact region IDs, match_phrase for paths, query_string for logs
- Always use .keyword subfields for exact text matching
- Cross-reference filesystem (wildcard: {"wildcard": {"path": "*<region_id>*"}}) with hbase:meta (match: {"match": {"row_key": "<region_id>"}})
- The total region count in hbase meta must match the total matched document count of wildcard path like "*/.regioninfo" in hbase rootdir path.  
- All terms of region_name.keyword for a region encoded name must match a wildcard path like "*/.regioninfo"
- All terms of table_name.keyword for a table must match a wildcard path like "*/.tabledesc*"
- 1595e783b53d99cd5eef43b6debb2682 is the master store region that will locate in <hbase-root-dir>/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/
- May cross check with the raw logs in /tmp/hbase-log-analysis/
## Issue Types
Orphan regions, missing .regioninfo, missing/extra regions in hbase:meta, rowkey holes, stuck RIT, master initialization failures
## Analysis Steps
### 1. Cross-Reference Meta vs Filesystem
- Filesystem regions NOT in hbase:meta → ORPHAN REGION
- Meta regions NOT in filesystem → MISSING REGION
### 2. Validate Region Chain Continuity
- Sort regions by STARTKEY, verify region[i].ENDKEY == region[i+1].STARTKEY
- First STARTKEY must be '', last ENDKEY must be ''
- Gaps → ROWKEY HOLE
### 3. Check Region States
- state != 'OPEN' → Check RIT
- Missing server assignment → UNASSIGNED
- Multiple servers → SPLIT BRAIN
- "deployed_servers" field must have only one region server address like "ip-xxx-xxx-xxx-xxx.ec2.internal,16020,1770781485397" . The value should not be null or have multiple values. 
### 4. Validate .regioninfo Files
- Missing .regioninfo in region directory → CORRUPT REGION
### 5. Cross-Check HBCK Report
- Compare orphan counts, RIT regions, filesystem vs meta region counts
### 6. Analyze Logs
- Search: "updating hbase:meta row=<region>", "STUCK", "RIT", "Failed" + "<region>", "Split"/"Merge" + "<region>"
## Remediation
- Reference knowledge bases: "Apache HBase source code", "HBase operational tools"
- Use hbck2: /usr/lib/hbase-operator-tools/hbase-hbck2.jar
- Prefix commands with sudo -u hbase
- Use aws s3 for S3-based rootdir
- Wait 300s after creating holes before hbck fixMeta (catalog janitor cycle)
- Use unassign instead of deprecated close_region
- If the region does not have .regioninfo in  <hbase-root-dir>/data/<namespace>/<table-name>/<region-encoded-name>/ but hbase:meta has that region's information and that region has been deployed on a healthy region server, you can use hbase shell to unassign and assign the region to re-generate .regioninfo
- Always add "sudo -u hbase hbase" before "hbase shell" and "hbase hbck" commands
## Job flow
Target: <your-job-flow-id>
Inconsistency to detect: All kinds of inconsistencies

You can trust or input “y” or “t” to grant Kiro to search through mcp and knowledge bases.

Terminal screenshot showing MCP tool execution authorization prompt.

You may get some outputs like this: Kiro checked for any HBase issue.

Terminal screenshot showing HBase database query results for user table entries with server configuration details and an HBase Inconsistency Detection Framework analysis report

Kiro summarized the examination results.

Terminal screenshot displaying HBase inconsistency detection analysis results for job flow, showing one critical missing .regioninfo file issue for HBase region in a HBase table, with cluster health metrics, risk assessment, recommended fixes, and generated diagnostic reports.

Kiro provided mitigation commands after Kiro summarized the issue.

Terminal screenshot displaying a structured HBase quick fix guide with three sections: recommended fix procedure with sequential steps for region reassignment, verification steps using AWS S3 and HBCK2 tools, and impact assessment showing 30-60 second downtime, zero data loss risk, and isolated region scope for fixing missing .regioninfo file in HBase region.

Cleaning up

To avoid incurring future charges, delete the resources created during this walkthrough.

To clean up the resources

  1. Delete the AWS CloudFormation stack from AWS Management Console:

AWS CloudFormation Stacks management console displaying a list view with stacks, showing the "dev-hbase-log-analysis" stack with CREATE_COMPLETE status, along with action buttons for Delete, Update stack, Stack actions, and Create stack.

  1. Clean up Amazon EMR cluster resources (if created only for this walkthrough):
AWS EMR Clusters management console showing page clusters with a cluster in "Waiting" status
  1. Verify resource cleanup in the AWS Console to verify that all resources are deleted and review your AWS bill to confirm no unexpected charges.

Important considerations:

  • Amazon OpenSearch Service domains take several minutes to fully delete
  • Amazon S3 buckets with versioning retain object versions
  • Use smaller instance types for development to optimize costs
  • Monitor usage with AWS Cost Explorer

Conclusion

In this post, we showed you how to build an AI-powered HBase troubleshooting solution that transforms manual log analysis into an automated workflow. By combining Amazon OpenSearch Service vector search with Amazon Bedrock-powered analysis through the Kiro CLI, operations teams can resolve complex HBase inconsistencies faster and gain deeper operational insights. The solution demonstrates how AI augments human expertise to improve operational efficiency, reducing HBase inconsistency resolution from hours to minutes and root cause identification from days to hours. Ready to transform your HBase operations? Get started with the GitHub repository and explore the Amazon OpenSearch Service documentation for additional guidance on vector search capabilities.

Acknowledgments

The author would like to thank Xi Yang, Anirudh Chawla, and Sasidhar Puthambakkam for their contributions to developing the technical solution. Xi Yang is a Senior Hadoop System Engineer and Amazon EMR subject matter expert at AWS. Anirudh Chawla is an AWS Analytics Specialist Solution Architect who helps organizations empower businesses to harness their data effectively through AWS’s analytics platform. Sasidhar Puthambakkam is a Senior Hadoop Systems Engineer and Amazon EMR Subject Matter Expert who provides architectural guidance for complex BigData workloads.


About the authors

Yu-Ting Su

Yu-ting Su, Sr. Hadoop System Engineer, AWS Support Engineering. Yu-Ting is a Sr. Hadoop Systems Engineer at Amazon Web Services (AWS). Her expertise is in Amazon EMR and Amazon OpenSearch Service. She’s passionate about distributing computation and helping people to bring their ideas to life.

How to consolidate cross-Region S3 data into OpenSearch

Post Syndicated from David Venable original https://aws.amazon.com/blogs/big-data/how-to-consolidate-cross-region-s3-data-into-opensearch/

You might have data in Amazon Simple Storage Service (Amazon S3) buckets in different AWS Regions that you want available in a single Amazon OpenSearch Service domain or collection. Consolidating data across Regions provides unified analytics and searches, reduce operation complexity, and streamline your search infrastructure. We’re happy to announce that Amazon OpenSearch Ingestion pipelines can now read from S3 buckets in different Regions to ingest and consolidate data into a single OpenSearch Service domain or collection.

To consolidate this data across AWS Regions, you previously had to provide your own solution. Now Amazon OpenSearch Ingestion can help you accomplish this. In this post, I’ll show you how to use the new cross-Region support to ingest data from S3 buckets across multiple AWS Regions into a single OpenSearch Service domain or collection.

Amazon OpenSearch Ingestion (OSI) is a feature-rich data ingestion pipeline that you can use for many different purposes: observability, analytics, and zero-ETL search. Many customers use OpenSearch Ingestion to ingest data from Amazon S3 into OpenSearch Service domains and Amazon OpenSearch Serverless collections. Until now, you could only ingest from a single AWS Region at a time. Now that you can use OpenSearch Ingestion for cross-Region S3 ingestion, I’ll show you how you can use it in two scenarios: batch processing using S3 scan, and streaming ingestion using Amazon Simple Queue Service (Amazon SQS) queues for AWS vended logs like Amazon Virtual Private Cloud (Amazon VPC) Flow Logs and AWS CloudTrail.

Prerequisites

Complete the following prerequisite steps:

  1. Deploy an OpenSearch Service domain or OpenSearch Serverless collection in the Regions where you want to perform your search or analytics.
  2. You need S3 buckets in at least two different Regions. You can use existing ones or create S3 buckets. You can use one in the same AWS Region as your OpenSearch Service domain or collection, or use two completely different Regions.
  3. Upload objects with data into your S3 buckets. The data can be JSON, ND-JSON, Parquet, CSV, or plaintext formats.
  4. Configure AWS Identity and Access Management (IAM) permissions needed for OSI. For instructions, see Amazon S3 as a source.
  5. For cross-Region ingestion, you must now also include the s3:GetBucketLocation permission. This gives the pipeline the ability to determine which AWS Region the bucket is located in.

After you complete these steps, you can either set up your Amazon OpenSearch Ingestion pipelines for batch or streaming scenarios. In the following sections, I’ll give you recommendations on when to choose which approach, and I outline the steps for creating your pipeline.

Batch scenarios

You can use the OpenSearch Ingestion S3 scan capability to read batch data from S3. You might find this approach useful when your data is written to S3 on a schedule. To perform a cross-Region S3 scan, you only specify the buckets that you’re reading from when you create the OpenSearch Ingestion pipeline.

The following diagram shows the design for an OpenSearch Ingestion pipeline in us-west-2 reading from S3 buckets in us-east-1 and eu-west-1 and writing that data into an OpenSearch Service domain in us-west-2.

Next, you will create an OpenSearch Ingestion pipeline. You must create this pipeline in the same Region as your OpenSearch Service domain or collection.

version: "2"
s3-scan-cross-region:
  source:
    s3:
      compression: automatic
      codec:
        json:
      scan:
        buckets:
          - bucket:
              name: amzn-s3-demo-bucket1
          - bucket:
              name: amzn-s3-demo-bucket2
      aws:
        region: us-west-2

  sink:
    - opensearch:
        hosts: [ "https://search-mydomain-abcdefghijklmn.us-west-2.es.amazonaws.com" ]
        index: s3_scan_cross_region
        aws:
          region: us-west-2

The previous pipeline configuration supports the JSON codec. You might want to configure a different codec if your data isn’t a large JSON object.

You can now query your OpenSearch Service domain or collection to see the data that you ingested.

Streaming scenarios: AWS vended logs

Like many of our customers, you might want to ingest S3 data from different AWS Regions into OpenSearch Service. A common reason is to consolidate AWS vended logs. For example, VPC Flow Logs, CloudTrail data, and load balancer logs. For these scenarios, you can configure OpenSearch Ingestion pipelines to read from an Amazon SQS queue to stream data into your OpenSearch Service domain or collection.

These AWS vended logs write to Amazon S3 in the same AWS Region as the service running it. For example, VPC Flow Logs will be in the same AWS Region as your Amazon VPC. You can use OpenSearch Ingestion to consolidate these logs into one AWS Region. In the VPC Flow Logs example, you can consolidate your VPC Flow Logs from multiple AWS Regions into a single OpenSearch Service domain or collection to analyze network patterns from your different Amazon VPCs.

The following diagram outlines the overall setup. It shows an example of sending AWS vended logs from us-east-1 and eu-west-1 to an OpenSearch Service domain in us-west-2. You can change the AWS Regions depending on your specific needs.

  1. You must configure your vended logs to write log events to Amazon S3 buckets in their respective AWS Regions. Using VPC Flow Logs as our example, you can configure VPC Flow Logs for your VPCs.
  2. Create an Amazon SQS queue in the same AWS Region as your OpenSearch Service domain.
  3. Amazon S3 doesn’t send notifications to cross-Region Amazon SQS queues, so you will use intermediate Amazon Simple Notification Service (Amazon SNS) topics to consolidate the notifications from multiple Regions into one queue. For each S3 bucket, create an SNS topic.
  4. Configure S3 Event Notifications for SNS. You will do this for each S3 bucket and each SNS topic.
  5. SNS can send cross-Region notifications to SQS. Create a subscription from each SNS topic that you created in step 3 to the single SQS queue you created in step 2.
  6. Configure your pipeline role to read from SQS and read from the relevant S3 buckets.

Now create an OpenSearch Ingestion pipeline in the same AWS Region as your OpenSearch Service domain.

version: "2"
s3-sqs-cross-region:
  source:
    s3:
      notification_type: sqs
      codec:
        newline:
      sqs:
        queue_url: https://sqs.us-west-2.amazonaws.com/123456789012/amzn-s3-demo-all-regions
      aws:
        region: us-west-2

  sink:
    - opensearch:
        hosts: [ "https://search-mydomain-abcdefghijklmn.us-west-2.es.amazonaws.com" ]
        index: s3_sqs_cross_region
        aws:
          region: us-west-2

The previous pipeline configuration supports the JSON codec. You might want to configure a different codec if your data is not a large JSON object.

Next, upload objects with data into your S3 buckets. By uploading data, S3 will send notifications to SNS and then the SQS queue.

You can now query your OpenSearch Service domain or collection to see the data that you ingested.

Here is what makes this possible and what is different. The SQS queue receives the event notifications for the buckets. Before the cross-Region feature of OpenSearch Ingestion, the pipeline could see these events, but couldn’t access the S3 bucket even if the permissions were granted. Now, the pipeline will determine the AWS Region that the bucket is in, access an AWS Security Token Service (AWS STS) token for the AWS Region of the bucket. Using the STS token from the same Region as the S3 bucket allows the pipeline to read and access the data.

Using the AWS Console

When you create the pipeline using the OpenSearch Ingestion console, you will have options to select a blueprint for your use-case. These blueprints help you create pipelines for various vended log types only by selecting your SQS queue and OpenSearch domain. The blueprint handles the data type mappings for you by including appropriate processors. You can use these blueprints as a starting point and modify your processors for your specific requirements.

Clean up resources

When you’re done testing this out, use the following resources to delete the resources that you created.

If you set up a batch pipeline:

  • Delete the OpenSearch Ingestion pipeline.

If you set up a streaming pipeline:

For both pipelines, these steps help you delete the common resources.

Conclusion

In this post, I showed you how you can use Amazon OpenSearch Ingestion to ingest data from Amazon S3 buckets in different AWS Regions. I showed that this works for both batch scan and streaming scenarios. The feature offers you a straightforward way to consolidate your data from other Regions into one OpenSearch Service domain or collection.

To get started with the cross-Region S3 source, refer to the OpenSearch Ingestion documentation or try creating a pipeline from one of our blueprints using the OpenSearch Ingestion console. You can read about the codecs that OpenSearch Ingestion offers for parsing your S3 objects. You can also learn how about the various processors that OpenSearch Ingestion offers, so you can transform and enrich your data to meet your needs.

You can also use OpenSearch Ingestion for cross-Region and cross-account. To do this, you must grant cross-account permissions on your S3 bucket. You must also make some changes to your pipeline configuration. Combining what I showed you in this post with the existing cross-account features greatly expands your ingestion options.

If you’re ready to take your streaming ingestion analytics to the next level you can read about how to generate metrics from logs and even how to send those derived metrics to Amazon Managed Service for Prometheus.

Have you tried out the cross-Region capabilities of OpenSearch Ingestion? Share your use-cases and questions in the comments.


About the authors

David is a senior software engineer working on observability in OpenSearch at Amazon Web Services. He is a maintainer on the Data Prepper project.

Unified observability in Amazon OpenSearch Service: metrics, traces, and AI agent debugging in a single interface

Post Syndicated from Muthu Pitchaimani original https://aws.amazon.com/blogs/big-data/unified-observability-in-amazon-opensearch-service-metrics-traces-and-ai-agent-debugging-in-a-single-interface/

Amazon OpenSearch Service now brings application monitoring, native Amazon Managed Service for Prometheus integration, and AI agent tracing together in OpenSearch UI‘s observability workspace. You can query Prometheus metrics with PromQL alongside logs and traces stored in Amazon OpenSearch Service, trace an AI agent’s full reasoning chain down to the failing tool call, and drill from a service-level health view to the exact span that caused a checkout failure, all without leaving the interface.

In this post, we walk through two real-world scenarios using the OpenTelemetry sample app: a multi-agent travel planner facing slow processing, and a checkout flow quietly failing on one microservice. We chase each one to its root cause using these new capabilities.

Scenario 1: An underperforming AI agent

Your multi-agent travel planner is live and users start reporting slow responses. With the new AI agent tracing capability in Amazon OpenSearch Service, you can trace the agent’s full processing path to pinpoint exactly where things went wrong.

In any observability workspace in OpenSearch UI, navigate to Application Map in the left navigation pane.

OpenSearch Service application map

You can see the full topology of your system including the travel agent and the sub-agents it calls. The travel agent node shows elevated latency and occasional errors. Select it, and the side panel confirms that latency is up but the latency chart shows intermittent spikes rather than consistent degradation.

System topology with service health metrics

The application map tells you something is wrong, but understanding why an AI agent is underperforming requires seeing its reasoning chain. Select Agent Traces in the left navigation pane, then filter by service name and time range.

Agent processing steps with invocation data

Select one of the traces to see the trace tree. Unlike a traditional span waterfall, this view organizes around the agent’s reasoning chain: the root agent span, the LLM calls it made, the tools it invoked, and how they nested each step color-coded by type. The trace map provides a visual directed graph of the same execution. You can see which model was called, how many input and output tokens were consumed, and the actual messages sent to and received from the model.

A tool call inside the weather agent errored out. The agent then spent additional time reasoning about the failure before returning a partial response explaining the intermittent latency spikes and occasional faults.

Why this matters for AI agents

Agents make autonomous decisions based on LLM responses, tool results, and chained reasoning. Unlike traditional microservices with deterministic code paths, agent behavior varies across executions. Without semantic tracing that captures these AI-specific signals, root-cause analysis is guesswork. The trace tree surfaced the model name, token counts, and failing tool call because the travel planner was instrumented with OpenTelemetry’s generative AI semantic conventions. The next section describes how.

Instrumenting AI agents

OpenTelemetry auto-instrumentation enriches spans with well-known attributes for HTTP, database, and gRPC calls. AI agents need a different set of attributes such as which LLM was called, what tokens were consumed, which tools were invoked, that standard instrumentation doesn’t cover.

The OpenTelemetry gen_ai semantic conventions define standard attributes for these signals, including gen_ai.operation.name, gen_ai.usage.input_tokens, gen_ai.request.model, and gen_ai.tool.name. When Amazon OpenSearch Service receives spans with these attributes, it categorizes them by operation type (agent, LLM, tool, embeddings, retrieval) and renders the agent trace tree and trace map views.

The Python SDK provides one way to generate these spans. To send traces to Amazon OpenSearch Ingestion, configure the SDK with AWS Signature Version 4 (SigV4) authentication. The AWSSigV4OTLPExporter cryptographically signs each HTTP request to help prevent unauthorized data ingestion. The calling identity needs an IAM policy that grants osis:Ingest on your pipeline’s ARN. Credentials are resolved through the standard AWS credential provider chain.

from opensearch_genai_observability_sdk_py import register, AWSSigV4OTLPExporter

exporter = AWSSigV4OTLPExporter(
    endpoint="https://pipeline.us-east-1.osis.amazonaws.com/v1/traces",
    service="osis",
    region="us-east-1",
)

register(service_name="my-agent", exporter=exporter)

Use the @observe decorator to trace agent functions and enrich() to add model metadata:

@observe(op=Op.EXECUTE_TOOL)
def get_weather(city: str) -> dict:
    return {"city": city, "temp": 22, "condition": "sunny"}

@observe(op=Op.INVOKE_AGENT)
def assistant(query: str) -> str:
    enrich(model="gpt-4o", provider="openai")
    data = get_weather("Paris")
    return f"{data['condition']}, {data['temp']}C"

result = assistant("What's the weather?")

The SDK also supports auto-instrumentation for OpenAI, Anthropic, Amazon Bedrock, LangChain, LlamaIndex, and others. Because the instrumentation is built on OpenTelemetry standards, any agent framework that emits spans with gen_ai.* attributes is compatible with OpenSearch UI.

Scenario 2: Investigating a microservice issue

AI agents are only one part of most production environments. The same interface surfaces telemetry from conventional microservices, where the troubleshooting workflow follows a more familiar path.

Your ecommerce checkout begins paging during a busy traffic window. From OpenSearch UI, navigate to APM Services in the left navigation pane. Every instrumented service is listed alongside its health indicators. The checkout service shows an elevated error rate.

Service overview panel with request, error, duration metrics

Select the affected service. The detail view shows Request, Error, and Duration (RED) metrics: request rate is climbing, fault rate has spiked in the last 15 minutes, and p99 duration has doubled. You can see exactly when the degradation started.

Service drilldown health dashboard

Drill into the correlated spans for the affected time window. The span list shows multiple failed requests, all hitting the same endpoint. Select one to see the full trace waterfall. The checkout service called prepareOrder, which failed trying to retrieve a product from the catalog. The error message in the span details tells you exactly what went wrong, that’s your root cause.

Waterfall transaction view of spans

Checking the infrastructure with PromQL

In both scenarios, the natural next question is whether the problem originates in the application or in the infrastructure beneath it. With the new Amazon Managed Service for Prometheus integration, you can answer that question without leaving OpenSearch UI.

Prometheus metrics are now queryable directly from the same workspace using native PromQL syntax, alongside the logs and traces you’ve already been navigating.

Metric query showing Prometheus Query Language

For the database timeout in Scenario 2, run a PromQL query to check the database instance’s read/write throughput for the same time window. For the agent latency issue in Scenario 1, check the LLM endpoint’s response time metrics to see if the slowness originates from the model provider.

This is a key architectural decision: metrics continue to live in Amazon Managed Service for Prometheus, logs and traces continue to live in Amazon OpenSearch Service, and neither signal is copied or warehoused into a second store. Each backend remains the single store for the data type it’s purpose-built to handle, while OpenSearch UI federates queries across both at runtime. The cost, retention, and operational model of each store stay intact while the troubleshooting workflow collapses into a single interface.

To configure the OpenTelemetry Collector and OpenSearch Ingestion pipelines that route metrics into Amazon Managed Service for Prometheus, see Ingesting application telemetry.

How it’s wired together

The following diagram shows the end-to-end architecture. Applications instrumented with OpenTelemetry send traces, logs, and metrics over OTLP to Amazon OpenSearch Ingestion. OpenSearch Ingestion routes each signal to the appropriate store: traces and logs land in Amazon OpenSearch Service, while metrics flow into Amazon Managed Service for Prometheus. OpenSearch UI then queries both stores to render the Application Map, Services catalog, Agent Traces, and Metrics views.

OpenSearch Observability Stack Architecture

The entire experience rests on open-source foundations, Prometheus for metrics, OpenSearch for logs and traces, and OpenTelemetry for instrumentation, so teams already running an OpenTelemetry collector can adopt it by updating the collector’s export configuration to point at Amazon OpenSearch Ingestion, with no proprietary agents or rewritten instrumentation required.

Getting started

To enable these capabilities, log in to OpenSearch UI’s observability workspace, select the Gear icon in the bottom left corner to open Settings and setup, and verify that the Observability:apmEnabled toggle is on under the Observability section. OpenSearch UI is available at no additional charge for Amazon OpenSearch Service customers.

Explore locally first. The OpenSearch Observability Stack gives you a fully configured environment including application monitoring, agent tracing, and Prometheus integration, running on your machine with a single install command. It ships with sample instrumented services, including a multi-agent travel planner, so you can explore the full workflow with real telemetry data out of the box.

For AI agent development. Agent Health is an open-source, evaluation-driven observability tool designed for local development. It gives you execution flow graphs, token tracking, and tool invocation visibility right in your development loop, before you push to production.

For production. The Python SDK provides one-line setup and decorator-based tracing with gen_ai semantic conventions, with auto-instrumentation support for OpenAI, Anthropic, Amazon Bedrock, LangChain, LlamaIndex, and others. See the Amazon OpenSearch Service documentation and the Amazon Managed Service for Prometheus integration guide for the full managed experience.


About the authors

Muthu Pitchaimani

Muthu is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search applications and solutions. Muthu is interested in the topics of networking and security, and is based out of Austin, Texas.

Raaga N.G

Raaga is a Solutions Architect at AWS with over 5 years of experience helping enterprises modernize their technology landscape and build scalable, cloud-native solutions. She partners with customers to translate business requirements into efficient cloud architectures that drive measurable outcomes, supporting their journey from application modernization to AI adoption through thoughtful, customer-centric solutions.

Rekha Thottan

Rekha Thottan is a Senior Technical Product Manager at AWS OpenSearch, contributing to AI agent observability and evaluation for the OpenSearch Project.

Kevin Lewin

Kevin is a Cloud Operations Specialist Solution Architect at Amazon Web Services. He focuses on helping customers achieve their operational goals through observability and automation.

Enhancing Identity Intelligence with Babel Street Match and Amazon OpenSearch

Post Syndicated from Kunal Sharma original https://aws.amazon.com/blogs/big-data/enhancing-identity-intelligence-with-babel-street-match-and-amazon-opensearch/

This post is co-authored with Gil Irizarry, Mae Wells-Kress and Craig Harmon from Babel Street. 

Can your system tell “John Smith” apart from “John Smith”?

Organizations requiring identity intelligence increasingly face challenges due to complexity of matching names and entities across vast, multilingual, and constantly evolving datasets. Whether helping border security, combating financial crimes, or maintaining regulatory compliance, the accuracy of identity and entity resolution directly determines whether threats are detected, investigations succeed, and regulatory requirements are met. Yet, linguistic diversity, transliterations, inconsistent data formats, and legacy system limitations continue to create friction, leading to false positives, missed matches, and costly manual reviews. As customers ingest and analyze petabytes of unstructured and structured data in Amazon OpenSearch Service, the need for intelligent, scalable, and multilingual matching becomes increasingly important. This is where the integration of Babel Street (an AWS Partner) with OpenSearch Service provides a solution that helps organizations enhance precision, reduce noise, and accelerate insights from their high-volume data environments.

This post explores how combining Babel Street Match with OpenSearch Service provides a solution that helps your organization to handle large-scale, multilingual data.

The growing complexity of identity and entity resolution

As organizations ingest and analyze massive volumes of multilingual and inconsistently formatted data, accurately matching names and entities becomes increasingly difficult. Variations in spelling, transliterations, semantic differences, cultural naming conventions, and incomplete or noisy records can contribute to mismatches. These challenges are compounded by legacy systems, fragmented data pipelines, operational inefficiencies, and evolving regulatory requirements—especially in sectors where precision is a requirement.

Evaluating and enhancing identity in high-volume enterprise environments

Amazon OpenSearch Service is a fully managed, scalable search and analytics service that enables organizations to ingest, search, visualize, and analyze massive volumes of data in near real time. Built to handle structured and unstructured information from diverse sources, it powers use cases ranging from security analytics and log monitoring to enterprise search and advanced analytical applications.

Babel Street delivers risk intelligence trusted by organizations across government, defense, and the private sector. The offering combines access to vast volumes of multilingual data with advanced analytics to uncover hidden identities, secure vendor networks, and identify emerging risks with precision, speed, and scale. From national security to regulatory compliance and enterprise resilience, Babel Street provides the strategic advantage needed to stay ahead of risk, safeguard operations, and protect missions.

Babel Street Match, an offering from Babel Street incorporates advanced identity risk intelligence capabilities, which enhance the precision and reliability of screening processes. This advanced solution uses sophisticated matching techniques to verify identities and identify variations in personal data—including aliases, alternate spellings, and differences in biographical details, helping organizations separate legitimate individuals from potential threats. The ability to screen names, addresses, dates, and other identifiers across different scripts and languages helps reduce false positives and negatives, helps accurately detect critical risks with transparent scoring to meet compliance and audit requirements. Further, Babel Street Match streamlines screening workflows, reduces the burden of manual reviews, and elevates the accuracy of threat detection.

The following diagram shows the details of OpenSearch Service and Babel Street Match Plugin integration.

Architecture diagram showing Babel Street Match Plugin integration with AWS services, including AWS Marketplace, Amazon S3, and Amazon OpenSearch Service across two AWS accounts for secure entity matching.

Babel Street Match integrates directly with the OpenSearch Service domain through a lightweight plugin that runs inside your own AWS account where you have full control of your data. The Match plugin sends encrypted match requests to Babel Street’s fully managed Match engine, where the core matching engine performs the entity-resolution logic. The results return to you in real time, enhancing your existing OpenSearch Service workflows with advanced name- and entity-matching capabilities. Meanwhile, Babel Street’s control plane handles licensing, monitoring, and AWS Marketplace integration behind the scenes, provides continuous validation, automated updates, and a seamless operational experience.

Example use cases

The solution combines enterprise-scale search and analytics with AI-powered, multilingual identity intelligence. This section showcases example use cases where integration has enhanced organizations’ capabilities.

  • Border Screening: Help agencies identify high-risk travelers, cargo, and networks to strengthen point-of-entry security with faster, automated risk assessment.
  • Financial Services Compliance: Help Financial institutions and the FinTechs that serve them by offering AI-driven solutions for name screening, adverse media monitoring, and know your customer (KYC)/know your vendor (KYV) due diligence.
  • Identity and Organization Screening: Help businesses needing identity and organization screening by providing AI, analytics, and advanced matching technologies to assist in addressing complex screening challenges.
  • Customer and Vendor Onboarding: Help governments and financial institutions by providing research, analytics, and advanced matching technologies needed to quickly and confidently onboard customers and vendors at scale.

Customer Success Stories

Here’s how leading organizations are leveraging Babel Street Match and Amazon OpenSearch Service to solve real-world identity challenges:

  • A European online brokerage faced AML (anti-money laundering) compliance challenges with its outdated name-matching system, which produced excessive false positives and couldn’t process longer multilingual names. After implementing Babel Street Match on OpenSearch Service, the firm achieved up to 70% better accuracy across 25 languages—significantly reducing manual work and speeding customer payments.
    Babel Street Match Improves FI’s Name-Matching Accuracy by Up to 70% on OpenSearch
  • A major border agency struggled with an outdated screening system that flagged 15% of travelers as potential watchlist matches—overwhelming agents and creating long queues. After implementing Babel Street Match, false positives dropped dramatically (from 80,000 to just 100 in one test), hardware needs fell by 70%, and travelers with common names can now pass through faster. As one stakeholder put it: “Name matching is not our biggest problem anymore.”
    Enabling Stronger, Safer Borders with AI-powered Screening by Babel Street Match

Getting Started with Babel Street Match for Amazon OpenSearch Service

Amazon OpenSearch Service supports third-party plugins like Babel Street Match for OpenSearch. This plugin is supported on OpenSearch version 2.15 or higher and licenses can be obtained through AWS Marketplace.

Installing Babel Street Match for Amazon OpenSearch Service

Prerequisites: Obtain the license file from Babel Street and upload it to an S3 bucket in the same AWS Region as your OpenSearch domain.

Installation Steps:

  1. Create packages – In the OpenSearch Service console, create a package for your license file and select the Babel Street Match plugin from the available options
  2. Associate packages – Link both the license and plugin packages to your OpenSearch domain
  3. Verify – Monitor the domain update and confirm the plugin is active

For details, refer to AWS documentation “Installing third-party plugins in Amazon OpenSearch Service” and Babel Street installation guide which provides detailed guidance on pre-requisites, installation and using the plugin.

Conclusion

Together, Babel Street Match and OpenSearch Service help organizations cut through false positives and catch true matches faster. The result? Greater precision, efficiency, and speed—whether protecting entities, maintaining compliance, or securing supply chains. That’s business-critical identity intelligence in action.

Explore how Babel Street Match on Amazon OpenSearch Service can elevate your organization’s identity intelligence capabilities and transform the screening operations through an interactive or customized demo on Babel Street’s website.

Portions of this content describing Babel Street products and services are provided by Babel Street. AWS is not responsible for the accuracy of third-party product information.


About the Authors

Kunal Sharma

Kunal Sharma is a Sr. Solutions Architect at AWS. He works with AWS Worldwide Public Sector (WWPS) partners to build and scale cloud-native solutions. As an SA, he thrives on turning complex customer challenges into elegant, well-architected solutions — one whiteboard session at a time.

Gil Irizarry

Gil is the Chief Innovation Officer at Babel Street. He specializes in applying natural language processing and AI to identity resolution use cases. Gil’s work combines computational linguistics, machine learning and AI to produce state-of-the-art entity extraction and resolution applications. Gil’s focus on innovation led to his winning of Babel Street’s internal hackathon two years in a row.

Mae Wells-Kress

Mae Wells-Kress is the Vice President of Strategic Marketing at Babel Street. She has extensive experience across strategic and creative marketing roles, she implements process-driven lead generation efforts and develops strategic campaigns, events, and messaging that connect with audiences and helps organizations advance their missions in high stakes environments.

Craig Harmon

Craig is the Director of Partner Management at Babel Street. He leads the company’s strategic alliance with Amazon Web Services (AWS). A former Senior Partner Account Manager at AWS, Craig brings a hyperscaler‑native perspective to building and scaling partnerships that drive revenue growth and deepen technical collaboration. He is passionate about operational excellence and the design of high‑performance partner models that translate cloud innovation into measurable outcomes for customers and partners.

AWS Weekly Roundup: Claude Mythos Preview in Amazon Bedrock, AWS Agent Registry, and more (April 13, 2026)

Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-mythos-preview-in-amazon-bedrock-aws-agent-registry-and-more-april-13-2026/

In my last Week in Review post, I mentioned how much time I’ve been spending on AI-Driven Development Lifecycle (AI-DLC) workshops with customers this year. A common theme in those sessions is the need for better cost visibility. Teams are moving fast with AI, but as they go from experimenting to full production, finance and leadership really need to know who is using which resources and at what cost. That’s why I was so excited to see the launch of Amazon Bedrock new support for cost allocation by IAM user and role this week. This lets you tag IAM principals with attributes like team or cost center and then activate those tags in your Billing and Cost Management console. The resulting cost data flows into AWS Cost Explorer and the detailed Cost and Usage Report, giving you a clear line of sight into model inference spending. Whether you’re scaling agents across teams, tracking foundation model use by department, or running tools like Claude Code on Amazon Bedrock, this new feature is a game changer for tracking and managing your AI investments. You can get all the details on setting this up in the IAM principal cost allocation documentation.

Now, let’s get into this week’s AWS news…

Headlines
Amazon Bedrock now offers Claude Mythos Preview Anthropic’s most sophisticated AI model to date is now available on Amazon Bedrock as a gated research preview through Project Glasswing. Claude Mythos introduces a new model class focused on cybersecurity, capable of identifying sophisticated security vulnerabilities in software, analyzing large codebases, and delivering state of the art performance across cybersecurity, coding, and complex reasoning tasks. Security teams can use it to discover and address vulnerabilities in critical software before threats emerge. Access is currently limited to allowlisted organizations, with Anthropic and AWS prioritizing internet critical companies and open source maintainers.

AWS Agent Registry for centralized agent discovery and governance now in preview AWS launched Agent Registry through Amazon Bedrock AgentCore, providing organizations with a private catalog for discovering and managing AI agents, tools, skills, MCP servers, and custom resources. The registry helps teams locate existing capabilities rather than duplicating them, with semantic and keyword search, approval workflows, and CloudTrail audit trails. It is accessible via the AgentCore Console, AWS CLI, SDK, and as an MCP server queryable from IDEs.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Announcing Amazon S3 Files, making S3 buckets accessible as file systems — Amazon S3 Files transforms S3 buckets into shared file systems that connect any AWS compute resource directly with your S3 data. Built on Amazon EFS technology, it delivers full file system semantics with low latency performance, caching actively used data and providing multiple terabytes per second of aggregate read throughput. Applications can access S3 data through both file system and S3 APIs simultaneously without code modifications or data migration.
  • Amazon OpenSearch Service supports Managed Prometheus and agent tracing —Amazon OpenSearch Service now provides a unified observability platform that consolidates metrics, logs, traces, and AI agent tracing into a single interface. The update includes native Prometheus integration with direct PromQL query support, RED metrics monitoring, and OpenTelemetry GenAI semantic convention support for LLM execution visibility. Operations teams can correlate slow traces to logs and overlay Prometheus metrics on dashboards without switching between tools.
  • Amazon WorkSpaces Advisor now available for AI powered troubleshooting— AWS launched Amazon WorkSpaces Advisor, an AI powered administrative tool that uses generative AI to help IT administrators troubleshoot Amazon WorkSpaces Personal deployments. It analyzes WorkSpace configurations, detects problems automatically, and provides actionable recommendations to restore service and optimize performance.
  • Amazon Braket adds support for Rigetti’s 108 qubit Cepheus QPU — Amazon Braket now offers access to Rigetti’s Cepheus-1-108Q device, the first 100+ qubit superconducting quantum processor on the platform. The modular design features twelve 9 qubit chiplets with CZ gates that offer enhanced resilience to phase errors. It supports multiple frameworks including Braket SDK, Qiskit, CUDA-Q, and Pennylane, with pulse level control for researchers.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • What’s Next with AWS (April 28, Virtual) Join this livestream at 9am PT for a candid discussion about how agentic AI is transforming how businesses operate. Featuring AWS CEO Matt Garman, SVP Colleen Aubrey, and OpenAI leaders discussing emerging agent capabilities, Amazon’s internal experiences, and new agentic solutions and platform capabilities.

Browse here for upcoming AWS led in person and virtual events, startup events, and developer focused events.


That’s all for this week. Check back next Monday for another Weekly Roundup!

~ micah

Agentic AI for observability and troubleshooting with Amazon OpenSearch Service

Post Syndicated from Muthu Pitchaimani original https://aws.amazon.com/blogs/big-data/agentic-ai-for-observability-and-troubleshooting-with-amazon-opensearch-service/

Amazon OpenSearch Service powers observability workflows for organizations, giving their Site Reliability Engineering (SRE) and DevOps teams a single pane of glass to aggregate and analyze telemetry data. During incidents, correlating signals and identifying root causes demand deep expertise in log analytics and hours of manual work. Identifying the root cause remains largely manual. For many teams, this is the bottleneck that delays service recovery and burns engineering resources.

We recently showed how to build an Observability Agent using Amazon OpenSearch Service and Amazon Bedrock to reduce Mean time to Resolution (MTTR).  Now, Amazon OpenSearch Service brings many of these functions to the OpenSearch UI—no additional infrastructure required. Three new agentic AI features are offered to streamline and accelerate MTTR:

  • An Agentic Chatbot that can access the context and the underlying data that you’re looking at, apply agentic reasoning, and use tools to query data and generate insights on your behalf.
  • An Investigation Agent that deep-dives across signal data with hypothesis-driven analysis, explaining its reasoning at every step.
  • An Agentic Memory that supports both agents, so their accuracy and speed improve the more you use them.

In this post, we show how these capabilities work together to help engineers go from alert to root cause in minutes. We also walk through a sample scenario where the Investigation Agent automatically correlates data across multiple indices to surface a root cause hypothesis.

How the agentic AI capabilities work together

These AI capabilities are accessible from OpenSearch UI through an Ask AI button, as shown in the following diagram, which gives an entry point for the Agentic Chatbot.

Agentic Chatbot

To open the chatbot interface, choose Ask AI.

The chatbot understands the context of the current page, so it understands what you’re looking at before you ask a question. You can ask questions about your data, initiate an investigation, or ask the chatbot to explain a concept. After it understands your request, the chatbot plans and uses tools to access data, including generating and running queries in the Discover page, and applies reasoning to produce a data-driven answer. You can also use the chatbot in the Dashboard page, initiating conversations from a particular visualization to get a summary as shown in the following image.

Investigation agent

Many incidents are too complex to resolve with one or two queries. Now you can get the help of the investigation agent to handle these complex situations. The investigation agent uses the plan-execute-reflect agent, which is designed for solving complex tasks that require iterative reasoning and step-by-step execution. It uses a Large Language Model (LLM) as a planner and another LLM as an executor. When an engineer identifies a suspicious observation, like an error rate spike or a latency anomaly, they can ask the investigation agent to investigate. One of the important steps the investigation agent performs is re-evaluation. The agent, after executing each step, reevaluates the plan using the planner and the intermediate results. The planner can adjust the plan if necessary or skip a step or dynamically add steps based on this new information. Using the planner, the agent generates a root cause analysis report led by the most likely hypothesis and recommendations, with full agent traces showing every reasoning step, all findings, and how they support the final hypotheses. You can provide feedback, add your own findings, iterate on the investigation goal, and review and validate each step of the agent’s reasoning. This approach mirrors how experienced incident responders work, but completes automatically in minutes. You can also use the “/investigate” slash command to initiate an investigation directly from the chatbot, building on an ongoing conversation or starting with a different investigation goal.

Agent in action

Automatic query generation

Consider a situation where you’re an SRE or DevOps engineer and received an alert that a key service is experiencing elevated latency. You log in to the OpenSearch UI, navigate to the Discover page, and select the Ask AI button. Without any expertise in the Piped Processing Language (PPL) query language, you enter the question “find all requests with latency greater than 10 seconds”. The chatbot understands the context and the data that you’re looking at, thinks through the request, generates the right PPL command, and updates it in the query bar to get you the results. And if the query runs into any errors, the chatbot can learn about the error, self-correct, and iterate on the query to get the results for you.

Investigation and investigation management

For complex incidents that normally require manually analyzing and correlating multiple logs for the possible root cause, you can choose Start Investigation to initiate the investigation agent. You can provide a goal for the investigation, along with any context or hypothesis that you want to instruct the investigation. For example, “identify the root cause of widespread high latency across services. Use TraceIDs from slow spans to correlate with detailed log entries in the related log indices. Analyze affected services, operations, error patterns, and any infrastructure or application-level bottlenecks without sampling”.

The agent, as part of the conversation, will offer to investigate any issue that you’re trying to debug.

The agent sets goals for itself along with any other relevant information like indices, associated time range, and other, and asks for your confirmation before creating a Notebook for this investigation. A Notebook is a way within the OpenSearch UI to develop a rich report that’s live and collaborative. This helps with the management of the investigation and allows for reinvestigation at a later date if necessary.

After the investigation starts, the agent will perform a quick analysis by log sequence and data distribution to surface outliers. Then, it will plan for the investigation into a series of actions, and then performs each action, such as query for a specific log type and time range. It will reflect on the results at every step, and iterate on the plan until it reaches the most likely hypotheses. Intermediate results will appear on the same page as the agent works so that you can follow the reasoning in real time. For example, you find that the Investigation Agent accurately mapped out the service topology and used it as a key intermediary steps for the investigation.

As the investigation completes, the investigation agent concludes that the most likely hypothesis is a fraud detection timeout. The associated finding shows a log entry from the payment service: “currency amount is too big, waiting for fraud detection”. This matches a known system design where large transactions trigger a fraud detection call that blocks the request until the transaction is scored and assessed. The agent arrived at this finding by correlating data across two separate indices, a metrics index where the original duration data lived, and a correlated log index where the payment service entries were stored. The agent linked these indices using trace IDs, connecting the latency measurement to the specific log entry that explained it.

After reviewing the hypothesis and the supporting evidence, you find the result reasonable and aligns with your domain knowledge and past experiences with similar issues. You can now accept the hypothesis and review the request flow topology for the affected traces that were provided as part of the hypothesis investigation.

Alternatively, if you find that the initial hypothesis wasn’t helpful, you can review the alternative hypothesis at the bottom of the report and select any of the alternative hypotheses if there’s one that’s more accurate. You can also trigger a re-investigation with additional inputs, or corrections from previous input so that the Investigation Agent can rework it.

Getting started

You can use any of the new agentic AI features (limits apply) in the OpenSearch UI at no cost. You will find the new agentic AI features ready to use in your OpenSearch UI applications, unless you have previously disabled AI features in any OpenSearch Service domains in your account. To enable or disable the AI features, you can navigate to the details page of the OpenSearch UI application in AWS Management Console and update the AI settings from there. Alternatively, you can also use the registerCapability API to enable the AI features or use the deregisterCapability API to disable them. Learn more at Agentic AI in Amazon OpenSearch Services.

The agentic AI feature uses the identity and permissions of the logged in users for authorizing access to the connected data sources. Make sure that your users have the necessary permissions to access the data sources. For more information, see Getting Started with OpenSearch UI.

The investigation results are saved in the metadata system of OpenSearch UI and encrypted with a service managed key. Optionally, you can configure a customer managed key to encrypt all of the metadata with your own key. For more information, see Encryption and Customer Managed Key with OpenSearch UI.

The AI features are powered by Claude Sonnet 4.6 model in Amazon Bedrock. Learn more at Amazon Bedrock Data Protection.

Conclusion

The new agentic AI capabilities announced for Amazon OpenSearch Service help reduce Mean Time to Resolution by providing context-aware agentic chatbot for assistance, hypothesis-driven investigations with full explainability, and agentic memory for context consistency. With the new agentic AI capabilities, your engineering team can spend less time writing queries and correlating signals, and more time acting on confirmed root causes. We invite you to explore these capabilities and experiment with your applications today.


About the authors

Muthu Pitchaimani

Muthu is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search applications and solutions. Muthu is interested in the topics of networking and security, and is based out of Austin, Texas.

Hang (Arthur) Zuo

Arthur is a Senior Product Manager with Amazon OpenSearch Service. Arthur leads OpenSearch UI platform and agentic AI features for observability and search use cases. Arthur is interested in the topics of Agentic AI and data products.

Mikhail Vaynshteyn

Mikhail is a Solutions Architect with Amazon Web Services. Mikhail works with healthcare and life sciences customers and specializes in data analytics services. Mikhail has more than 20 years of industry experience covering a wide range of technologies and sectors.

Designing centralized and distributed network connectivity patterns for Amazon OpenSearch Serverless – Part 2

Post Syndicated from Ankush Goyal original https://aws.amazon.com/blogs/big-data/designing-centralized-and-distributed-network-connectivity-patterns-for-amazon-opensearch-serverless-part-2/

This post is Part 2 of our two-part series on hybrid multi-account access patterns for Amazon OpenSearch Serverless. In Part 1, we explored a centralized architecture where a single account hosts multiple OpenSearch Serverless collections and a shared VPC endpoint. This approach works well when a single business unit or team manages collections on behalf of the organization.

However, many enterprises have multiple business units that need independent ownership of their OpenSearch Serverless infrastructure. When each business unit wants to manage their own collections, security policies, and VPC endpoints within their own AWS accounts, the centralized model from Part 1 no longer fits.

In this post, we address this multi-business unit scenario by introducing a pattern where the central networking account manages a custom private hosted zone (PHZ) with CNAME records pointing to each business unit’s VPC endpoint. This approach maintains centralized DNS management and connectivity while giving each business unit full autonomy over their collections and infrastructure.

The challenge with multiple business units

When multiple business units independently manage their own OpenSearch Serverless collections in separate AWS accounts, each account has its own VPC endpoint with its own private hosted zones. These private hosted zones only work within their respective VPCs, creating DNS fragmentation across the organization. Consumers in spoke accounts and on-premises environments can’t resolve collection endpoints in other accounts without additional DNS configuration.

Managing individual PHZ associations for each consumer VPC doesn’t scale, and asking each business unit to coordinate DNS with every consumer creates operational overhead. You need a network architecture that gives each team autonomy while keeping DNS management and connectivity centralized.

Solution overview

This architecture solves the problem by centralizing DNS management in the networking account while leaving collection and VPC endpoint ownership with each business unit. The networking account maintains a custom PHZ with CNAME records that map each collection endpoint to the regional DNS name of its corresponding VPC endpoint. This custom PHZ is associated with a Route 53 Profile and shared through AWS Resource Access Manager (AWS RAM) to spoke accounts. On-premises DNS resolution flows through the Route 53 Resolver inbound endpoint in the central networking VPC, which uses the same custom PHZ.

We cover two complementary patterns: Pattern 1 for on-premises access to collections across multiple business unit accounts, and Pattern 2 for spoke account access to those same collections. Both patterns rely on the centralized custom PHZ managed by your networking team.

Pattern 1: On-premises access to OpenSearch Serverless collections across multiple business unit accounts

With this pattern, your on-premises clients can privately access OpenSearch Serverless collections hosted across multiple business unit accounts, each with its own VPC endpoint. The following diagram illustrates this multi-business-unit architecture. It shows how on-premises DNS queries are resolved through the custom PHZ in the central networking account and routed to the correct business unit’s VPC endpoint through AWS PrivateLink.

(A) The Route 53 Profiles are created in the central networking account and shared through AWS Resource Access Manager (AWS RAM) with the central OpenSearch Serverless account.

(B) The central networking account has a custom PHZ with domain us-east-1.aoss.amazonaws.com and associated with central networking account VPC and with Route 53 Profiles.

(C) This PHZ contains CNAME records pointing to each business unit’s VPC endpoints.

(D) Each business unit account has Private DNS enabled on its VPC endpoint.

(E) This automatically creates the following PHZs during VPC endpoint creation and associates them with the business unit’s VPC, so DNS resolution works locally within that VPC without depending on the Route 53 Profiles.

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

DNS resolution flow

  1. Your on-premises client initiates a request to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com.
  2. The on-premises DNS resolver has a conditional forwarder for us-east-1.aoss.amazonaws.com and forwards the query over AWS Direct Connect or AWS Site-to-Site VPN to the Route 53 Resolver inbound endpoint IPs in the central networking VPC.
  3. The inbound Resolver endpoint passes the query to the Route 53 VPC Resolver in the central networking VPC.
  4. The VPC Resolver finds the custom PHZ (bu-1-collection-id-1.us-east-1.aoss.amazonaws.com) associated with the central networking VPC. The CNAME record for bu-1-collection-id-1.us-east-1.aoss.amazonaws.com resolves to the regional DNS name of BU1’s VPC endpoint (for example, vpce-1234567890abcdefghi.a2oselk.vpce-svc-0c3ebf9a1a3ad247b.us-east-1.vpce.amazonaws.com).
  5. The VPC Resolver then resolves the VPC endpoint regional DNS name to its elastic network interfaces (ENIs) private IP addresses.
  6. The traffic reaches BU1’s VPC endpoint elastic network interfaces (ENIs) through private network connectivity because the on-premises client connects over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.

Data flow

  1. Your on-premises client sends an HTTPS request to the resolved IP address with the TLS Server Name Indication (SNI) header set to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com, over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.
  2. Traffic reaches the VPC endpoint ENIs in BU1’s OpenSearch Serverless VPC.
  3. The VPC endpoint forwards the request to the OpenSearch Serverless service, which inspects the hostname and routes to BU1 Collection 1.

To access a collection in BU2, your client follows the same flow using bu-2-collection-id-1.us-east-1.aoss.amazonaws.com. The custom PHZ contains a separate CNAME record pointing to BU2’s VPC endpoint, and the OpenSearch Serverless service routes to the correct collection based on the hostname.

Pattern 2: Spoke account access to OpenSearch Serverless collections across multiple business unit accounts

While Pattern 1 addresses on-premises access, you might also need to provide access from compute resources and distributed applications in spoke accounts to OpenSearch Serverless collections across multiple business unit accounts. With this pattern, compute resources in spoke account VPCs can privately access OpenSearch Serverless collections across multiple business unit accounts through the centralized private hosted zone (PHZ) in the networking account.

In the following example, we use an Amazon Elastic Compute Cloud (Amazon EC2) instance as a compute resource to illustrate the pattern. However, the same approach applies to any compute resource within the spoke VPC. The following diagram illustrates this multi-business-unit, multi-spoke architecture, showing how spoke VPCs resolve DNS through the shared Route 53 Profile and custom PHZ, then route traffic to the correct business unit’s OpenSearch Serverless collections through AWS PrivateLink.

(A) The Route 53 Profiles, created in the central networking account, are shared through AWS RAM with all spoke accounts and with the central OpenSearch Serverless account.

(B) The central networking account has a custom PHZ with domain us-east-1.aoss.amazonaws.com and associated with central networking account VPC and with Route 53 Profiles.

(C) This PHZ contains CNAME records pointing to each business unit’s VPC endpoints.

(D) Each business unit account has Private DNS enabled on its VPC endpoint.

(E) This automatically creates the following PHZs during VPC endpoint creation and associates them with the business unit’s VPC, so DNS resolution works locally within that VPC without depending on the Route 53 Profiles.

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

DNS resolution flow

  1. An Amazon EC2 instance in BU1 Spoke VPC 1 initiates a request to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com and sends a DNS query to the Route 53 VPC Resolver.
  2. The VPC Resolver finds the Route 53 Profiles associated with the spoke VPC.
  3. The Profiles reference the custom PHZ (us-east-1.aoss.amazonaws.com) managed in the central networking account. The CNAME record for bu-1-collection-id-1.us-east-1.aoss.amazonaws.com resolves to the regional DNS name of BU1’s VPC endpoint.
  4. The VPC Resolver then resolves the VPC endpoint regional DNS name to its elastic network interfaces (ENIs) private IP addresses.
  5. Traffic reaches the VPC endpoint ENIs through private network connectivity because the spoke VPC connects to BU1’s VPC through AWS Transit Gateway or AWS Cloud WAN.

Data flow

  1. Your Amazon EC2 instance sends an HTTPS request to the resolved IP address with the TLS SNI header set to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com, routed through AWS Transit Gateway or AWS Cloud WAN to BU1’s OpenSearch Serverless VPC.
  2. The request arrives at the VPC endpoint ENIs in BU1’s OpenSearch Serverless VPC.
  3. The VPC endpoint forwards the request to the OpenSearch Serverless service, which inspects the hostname and routes to BU1 Collection 1.

To access a collection in BU2, the same flow applies using bu-2-collection-id-1.us-east-1.aoss.amazonaws.com. The custom PHZ resolves to BU2’s VPC endpoint, and routes traffic through the transit gateway to BU2’s VPC. The same applies to resources in other spoke accounts with the Route 53 Profiles associated.

Custom PHZ record structure

The custom PHZ in the central networking account uses the domain us-east-1.aoss.amazonaws.com and contains CNAME records that map each collection endpoint to the regional DNS name of its corresponding VPC endpoint. Note that collections within the same business unit account share the same VPC endpoint, so their CNAME records point to the same regional DNS name. Collections in different business unit accounts point to different VPC endpoints.

Custom PHZ management

Unlike Part 1, where the auto-created PHZs from the VPC endpoint handle DNS resolution, this pattern requires your networking team to manually maintain the custom PHZ. When a business unit adds a new collection, the networking team must add a corresponding CNAME record to the custom PHZ.

Cost considerations

The architecture patterns described in this post use several AWS services that can contribute to your overall costs, including Amazon Route 53 (hosted zones, DNS queries, and Resolver endpoints), and Route 53 Profiles. We recommend reviewing the official AWS pricing pages for the most current rates:

For a full cost estimate tailored to your workload, use the AWS Pricing Calculator.

Conclusion

In this post, we showed how you can give on-premises clients and spoke account resources private access to OpenSearch Serverless collections distributed across multiple business unit accounts. By centralizing DNS management through a custom PHZ in the networking account and sharing it through Route 53 Profiles, you avoid coordinating PHZ associations across accounts while giving each business unit full ownership of their collections and VPC endpoints.

Combined with Part 1, you now have two architectural approaches for hybrid multi-account access to OpenSearch Serverless: a centralized model where one account owns all collections and a shared VPC endpoint, and a distributed model where multiple business units each manage their own collections and VPC endpoints. Choose the centralized model when a single team manages collections on behalf of the organization. Choose the distributed model when business units need independent ownership of their OpenSearch Serverless infrastructure.

For additional details, refer to the Amazon OpenSearch Serverless VPC endpoint documentation and Route 53 Profiles documentation.


About the authors

Ankush Goyal

Ankush Goyal

Ankush is a Senior Technical Account Manager at AWS Enterprise Support, specializing in helping customers in the travel and hospitality industries optimize their cloud infrastructure. With over 20 years of IT experience, he focuses on leveraging AWS networking services to drive operational efficiency and cloud adoption. Ankush is passionate about delivering impactful solutions and enabling clients to streamline their cloud operations.

author name

Salman Ahmed

Salman is a Senior Technical Account Manager at AWS. He specializes in guiding customers through the design, implementation, and support of AWS solutions. Combining his networking expertise with a drive to explore new technologies, he helps organizations successfully navigate their cloud journey. Outside of work, he enjoys photography, traveling, and watching his favorite sports teams.

Designing centralized and distributed network connectivity patterns for Amazon OpenSearch Serverless – Part 1

Post Syndicated from Ankush Goyal original https://aws.amazon.com/blogs/big-data/designing-centralized-and-distributed-network-connectivity-patterns-for-amazon-opensearch-serverless-part-1/

Amazon OpenSearch Serverless is a fully managed, serverless option for Amazon OpenSearch Service that removes the operational complexity of provisioning, configuring, and tuning OpenSearch clusters. When you run OpenSearch Serverless collections in a central account and need secure, private access from both on-premises environments and multiple AWS accounts, network architecture becomes critical. In this post, we explore two patterns to help you achieve this connectivity securely.

Solution overview

Working with customers implementing OpenSearch Serverless, we published blog posts addressing various network connectivity patterns to meet their evolving requirements:

In this post, we build on those patterns to address an additional enterprise requirement. When you manage many OpenSearch Serverless collections centrally but need access from multiple accounts and on-premises, you face several key challenges:

  • Coordinating VPC endpoints across accounts: managing endpoint provisioning and lifecycle across many consumer accounts adds operational overhead
  • Managing DNS configurations for each consumer: each new account or on-premises environment requires its own DNS setup, increasing complexity
  • Separating networking responsibilities from application ownership: without clear boundaries, networking and application teams become tightly coupled, slowing down both

This architecture solves these challenges with a clear separation of responsibilities. The central networking account shares Route 53 Profiles to manage DNS propagation across spoke accounts. The OpenSearch Serverless account owner maintains full control over their VPC endpoint and the associated private hosted zones (PHZs). Application owners retain autonomy over DNS configuration and collection management.

A single VPC endpoint handles multiple OpenSearch Serverless collections in an AWS Region, which reduces complexity and cost. Your networking team manages connectivity infrastructure while your application teams independently manage their OpenSearch Serverless collections, data access policies, and collection-specific DNS configurations. This gives you connectivity from on-premises networks (through AWS Direct Connect or AWS Site-to-Site VPN) and from compute resources across multiple AWS accounts through a unified network path.

This separation means that your network administrators and application teams can work independently. The result is a governance model that scales with your organization. We cover two complementary patterns that together give you complete hybrid access coverage, Pattern 1 for on-premises access and Pattern 2 for multi-account access, both using centralized interface VPC endpoints and Route 53 Profiles.

Before proceeding, you should be familiar with OpenSearch Serverless interface VPC endpoint DNS resolution. When creating an OpenSearch Serverless interface VPC endpoint, AWS automatically provisions four private hosted zones. The zones are three visible private hosted zones (for collections, dashboards, and FIPS endpoints) and one hidden internal private hosted zone that work together to resolve collection endpoints to private IP addresses. For more details on this DNS resolution mechanism, customers can review our previous blog post.

Pattern 1: Accessing multiple OpenSearch Serverless collections from on-premises through a centralized VPC endpoint and Route 53 Profiles in a multi-account architecture

The architecture spans three components:

  • A central OpenSearch Serverless account that hosts the collections and interface VPC endpoint.
  • A central networking account that owns the Route 53 Profiles and Inbound Resolver.
  • An on-premises environment connected using AWS Direct Connect or AWS Site-to-Site VPN.

The following diagram illustrates the architecture across these three components. It shows how DNS queries from on-premises clients are resolved through the Route 53 Profile and Inbound Resolver, and how data traffic reaches the OpenSearch Serverless collections using AWS PrivateLink.

(A) The Route 53 Profiles are created in the central networking account and shared through AWS Resource Access Manager (AWS RAM) with the central OpenSearch Serverless account. The central OpenSearch Serverless account associates the PHZs and interface VPC endpoint association with the shared Route 53 Profiles because that’s needed for end-to-end private DNS resolution.

(B) The central AOSS VPC contains an interface VPC endpoint for OpenSearch Serverless with Private DNS enabled. There are four Private Hosted Zones (PHZs):

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

(C) The first three PHZs must be manually associated with the Route 53 Profile.

(D) The fourth is automatically associated when the VPC endpoint is associated with the profile.

(E) On the on-premises side, the DNS resolver is configured with conditional forwarding for us-east-1.aoss.amazonaws.com, directing queries to the Route 53 Resolver Inbound Endpoint in the central networking account.

DNS resolution flow

  1. Your on-premises client initiates a request to collection-id-1.us-east-1.aoss.amazonaws.com.
  2. The on-premises DNS resolver has a conditional forwarder for us-east-1.aoss.amazonaws.com and forwards the query over AWS Direct Connect or AWS Site-to-Site VPN to the Route 53 Resolver inbound endpoint IPs in the central networking VPC.
  3. The inbound resolver receives the query and passes it to the Route 53 VPC Resolver.
  4. The VPC Resolver checks the Route 53 Profiles associated with the central networking VPC. The Profiles provide access to the visible and hidden PHZs from the central OpenSearch Serverless VPC.
  5. The VPC Resolver uses the visible PHZ to match the wildcard CNAME *.us-east-1.aoss.amazonaws.com to the interface VPC endpoint (VPCE) DNS name. It then uses the hidden PHZ to resolve the VPCE DNS name to the private elastic network interface (ENI) IP addresses of the interface VPC endpoint in the central OpenSearch Serverless VPC.
  6. The private ENI IP addresses are returned through the inbound resolver endpoint to the on-premises DNS resolver and back to the on-premises client.

Data flow

  1. The on-premises client sends an HTTPS request to the resolved private ENI IP address with the TLS Server Name Indication (SNI) header set to collection-id-1.us-east-1.aoss.amazonaws.com, over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.
  2. Traffic reaches the interface VPC endpoint ENIs in the central OpenSearch Serverless VPC.
  3. The interface VPC endpoint forwards the request to the OpenSearch Serverless service, which routes to Collection 1.

To access Collection 2, the client follows the same flow using collection-id-2.us-east-1.aoss.amazonaws.com. The wildcard DNS resolves to the same interface VPC endpoint, and the OpenSearch Serverless service routes to the correct collection based on the hostname.

Pattern 2: Accessing multiple OpenSearch Serverless collections from spoke accounts using a centralized VPC endpoint and Route 53 Profiles

While Pattern 1 addresses on-premises access, you might also need to provide access from compute resources and distributed applications across multiple AWS accounts. With this pattern, any compute resource running within spoke account VPCs can privately access multiple OpenSearch Serverless collections hosted in a central OpenSearch Serverless VPC through a single shared interface VPC endpoint.

In the following example, we use an Amazon Elastic Compute Cloud (Amazon EC2) instance as a compute resource to illustrate the pattern. However, the same approach applies to any compute resource within the spoke VPC.

The following diagram illustrates this multi-account architecture, showing how spoke account VPCs resolve DNS and route data traffic to the central OpenSearch Serverless collections through the shared Route 53 Profile and AWS PrivateLink, alongside the on-premises access path from Pattern 1.

(A) The Route 53 Profiles, created in the central networking account, are shared via AWS RAM with both the OpenSearch Serverless account and the spoke accounts. The OpenSearch Serverless account associates its interface VPC endpoint and private hosted zones (PHZs) with the Profiles, while each spoke account associates the Profiles with its VPC. Spoke VPCs get full DNS resolution for OpenSearch Serverless collection endpoints without requiring their own interface VPC endpoints, PHZs, or manual DNS configuration.

(B) The central AOSS VPC contains an interface VPC endpoint for OpenSearch Serverless with Private DNS enabled. There are four Private Hosted Zones (PHZs):

  • us-east-1.aoss.amazonaws.com
  • us-east-1.opensearch.amazonaws.com
  • us-east-1.aoss-fips.amazonaws.com
  • privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

(C) The first three PHZs must be manually associated with the Route 53 Profile.

(D) The fourth is automatically associated when the VPC endpoint is associated with the profile.

DNS resolution flow

  1. An Amazon EC2 instance in Spoke VPC 1 initiates a request to collection-id-1.us-east-1.aoss.amazonaws.com and sends a DNS query to the Route 53 VPC Resolver.
  2. The VPC Resolver finds the Route 53 Profiles associated with the spoke VPC, which carries the PHZs from the central OpenSearch Serverless account.
  3. The visible PHZ matches the wildcard CNAME *.us-east-1.aoss.amazonaws.com to the VPCE DNS name. The hidden PHZ resolves the VPCE DNS name to the private ENI IP addresses of the interface VPC endpoint in the central OpenSearch Serverless VPC.
  4. Route 53 returns the private ENI IP addresses to the Amazon EC2 instance.

Data flow

  1. Your Amazon EC2 instance sends an HTTPS request to the resolved private ENI IP address with the TLS SNI header set to collection-id-1.us-east-1.aoss.amazonaws.com. This is routed through AWS Transit Gateway or AWS Cloud WAN to the central OpenSearch Serverless VPC.
  2. The request arrives at the interface VPC endpoint ENIs in the central OpenSearch Serverless VPC.
  3. The interface VPC endpoint forwards the request to the OpenSearch Serverless service, which routes to Collection 1.

To access Collection 2, the same flow applies using collection-id-2.us-east-1.aoss.amazonaws.com. The same applies to resources in Spoke Account 2 or other spoke accounts with the Route 53 Profiles associated.

AWS RAM Permission Configuration for Resource Association

When the central networking account shares the Route 53 Profiles through AWS RAM, the default AWS managed permissions policy (AWSRAMPermissionRoute53ProfileAllowAssociationActions) only grants actions for associating and disassociating the Profile with VPCs, viewing Profiles details, and listing associations. It does not include the route53profiles:AssociateResourceToProfile or route53profiles:DisassociateResourceFromProfile actions required for the OpenSearch Serverless account to associate its interface VPC endpoint and PHZs with the shared Profiles.

To enable this, the central networking account must create a custom managed permission in AWS RAM with the following actions:

  • route53profiles:AssociateProfile
  • route53profiles:AssociateResourceToProfile
  • route53profiles:DisassociateProfile
  • route53profiles:DisassociateResourceFromProfile
  • route53profiles:GetProfile
  • route53profiles:GetProfileResourceAssociation
  • route53profiles:ListProfileAssociations
  • route53profiles:ListProfileResourceAssociations
  • route53profiles:ListProfiles

This custom permission must be attached to the RAM resource share before the central OpenSearch Serverless account can associate its PHZs and interface VPC endpoint with the Profiles.

Cost considerations

The architecture patterns described in this post use several AWS services that may contribute to your overall costs, including Amazon Route 53 (hosted zones, DNS queries, and Resolver endpoints), and Route 53 Profiles. We recommend reviewing the official AWS pricing pages for the most current rates:

For a full cost estimate tailored to your workload, use the AWS Pricing Calculator.

Conclusion

In this post, we showed how organizations can provide secure, private access to multiple OpenSearch Serverless collections from both on-premises environments and distributed AWS accounts using a single centralized interface VPC endpoint and Route 53 Profiles. This architecture centralizes OpenSearch Serverless collections and network infrastructure in a dedicated account, using Route 53 Profiles to propagate DNS across accounts. This eliminates per-account VPC endpoints, manual PHZ associations, and custom DNS configuration in spoke accounts.

This pattern is a good fit when a single team or business unit manages OpenSearch Serverless collections centrally. However, many enterprises have business units that need to independently manage their own OpenSearch Serverless collections in separate AWS accounts, each with their own interface VPC endpoints, security policies, and collection lifecycle. In Part 2, we explore how the architecture changes to support this distributed ownership model, where each business unit runs OpenSearch Serverless in their own account while still relying on centralized network connectivity and DNS management through Route 53 Profiles. We will cover how DNS resolution and the Route 53 Profiles configuration adapt when interface VPC endpoints and collections are spread across multiple accounts.

For additional details, refer to the OpenSearch Serverless VPC endpoint documentation and Route 53 Profiles documentation.


About the authors

Ankush Goyal

Ankush Goyal

Ankush is a Senior Technical Account Manager at AWS Enterprise Support, specializing in helping customers in the travel and hospitality industries optimize their cloud infrastructure. With over 20 years of IT experience, he focuses on leveraging AWS networking services to drive operational efficiency and cloud adoption. Ankush is passionate about delivering impactful solutions and enabling clients to streamline their cloud operations.

author name

Salman Ahmed

Salman is a Senior Technical Account Manager at AWS. He specializes in guiding customers through the design, implementation, and support of AWS solutions. Combining his networking expertise with a drive to explore new technologies, he helps organizations successfully navigate their cloud journey. Outside of work, he enjoys photography, traveling, and watching his favorite sports teams.

AWS Weekly Roundup: Amazon Connect Health, Bedrock AgentCore Policy, GameDay Europe, and more (March 9, 2026)

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-connect-health-bedrock-agentcore-policy-gameday-europe-and-more-march-9-2026/

Fiti AWS Student Community Kenya!

Last week was an incredible whirlwind: a round of meetups, hands-on workshops, and career discussions across Kenya that culminated with the AWS Student Community Day at Meru University of Science and Technology, with keynotes from my colleagues Veliswa and Tiffany, and sessions on everything from GitOps to cloud-native engineering, and a whole lot of AI agent building.

JAWS Days 2026 is the largest AWS Community Day in the world, with over 1,500 attendees on March 7th. This event started with a keynote speech on building an AI-driven development team by Jeff Barr, and included over 100 technical and community experience sessions, lightning talks, and workshops such as Game Days, Builders Card Challenges, and networking parties.

Now, let’s get into this week’s AWS news…

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Introducing Amazon Connect Health, Agentic AI Built for Healthcare — Amazon Connect Health is now generally available with five purpose-built AI agents for healthcare: patient verification, appointment management, patient insights, ambient documentation, and medical coding. All features are HIPAA-eligible and deployable within existing clinical workflows in days.
  • Policy in Amazon Bedrock AgentCore is now generally available — You can now use centralized, fine-grained controls for agent-tool interactions that operate outside your agent code. Security and compliance teams can define tool access and input validation rules using natural language that automatically converts to Cedar, the AWS open-source policy language.
  • Introducing OpenClaw on Amazon Lightsail to run your autonomous private AI agents — You can deploy a private AI assistant on your own cloud infrastructure with built-in security controls, sandboxed agent sessions, one-click HTTPS, and device pairing authentication. Amazon Bedrock serves as the default model provider, and you can connect to Slack, Telegram, WhatsApp, and Discord.
  • AWS announces pricing for VPC Encryption Controls — Starting March 1, 2026, VPC Encryption Controls transitions from free preview to a paid feature. You can audit and enforce encryption-in-transit of all traffic flows within and across VPCs in a region, with monitor mode to detect unencrypted traffic and enforce mode to prevent it.
  • Database Savings Plans now supports Amazon OpenSearch Service and Amazon Neptune Analytics — You can save up to 35% on eligible serverless and provisioned instance usage with a one-year commitment. Savings Plans automatically apply regardless of engine, instance family, size, or AWS Region.
  • AWS Elastic Beanstalk now offers AI-powered environment analysis — When your environment health is degraded, Elastic Beanstalk can now collect recent events, instance health, and logs and send them to Amazon Bedrock for analysis, providing step-by-step troubleshooting recommendations tailored to your environment’s current state.
  • AWS simplifies IAM role creation and setup in service workflows — You can now create and configure IAM roles directly within service workflows through a new in-console panel, without switching to the IAM console. The feature supports Amazon EC2, Lambda, EKS, ECS, Glue, CloudFormation, and more.
  • Accelerate Lambda durable functions development with new Kiro power — You can now build resilient, long-running multi-step applications and AI workflows faster with AI agent-assisted development in Kiro. The power dynamically loads guidance on replay models, step and wait operations, concurrent execution patterns, error handling, and deployment best practices.
  • Amazon GameLift Servers launches DDoS Protection — You can now protect session-based multiplayer games against DDoS attacks with a co-located relay network that authenticates client traffic using access tokens and enforces per-player traffic limits, at no additional cost to GameLift Servers customers.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

From AWS community
Here are my personal favorite posts from AWS community and my colleagues:

  • I Built a Portable AI Memory Layer with MCP, AWS Bedrock, and a Chrome Extension — Learn how to build a persistent memory layer for AI agents using MCP and Amazon Bedrock, packaged as a Chrome extension that carries context across sessions and applications.
  • When the Model Is the Machine — Mike Chambers built an experimental app where an AI agent generates a complete, interactive web application at runtime from a single prompt — no codebase, no framework, no persistent state. A thought-provoking exploration of what happens when the model becomes the runtime.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Community GameDay Europe — Think you know AWS? Prove it at the AWS Community GameDay Europe on March 17, a gamified learning event where teams compete to solve real-world technical challenges using AWS services.
  • AWS at NVIDIA GTC 2026 — Join us at our AWS sessions, booths, demos, and ancillary events in NVIDIA GTC 2026 on March 16 – 19, 2026 in San Jose. You can receive 20% off event passes through AWS and request a 1:1 meeting at GTC.
  • AWS Summits — Join AWS Summits in 2026: free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), and Bengaluru (April 23–24).
  • AWS Community Days — Community-led conferences where content is planned, sourced, and delivered by community leaders. Upcoming events include Slovakia (March 11), Pune (March 21), and the AWSome Women Summit LATAM in Mexico City (March 28)

Browse here for upcoming AWS led in-person and virtual events, startup events, and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— seb

How Amplitude implemented natural language-powered analytics using Amazon OpenSearch Service as a vector database

Post Syndicated from Jeffrey Wang original https://aws.amazon.com/blogs/big-data/how-amplitude-implemented-natural-language-powered-analytics-using-amazon-opensearch-service-as-a-vector-database/

This is a guest post by Jeffrey Wang, Co-Founder and Chief Architect at Amplitude in partnership with AWS.

Amplitude is a product and customer journey analytics platform. Our customers wanted to ask deep questions about their product usage. Ask Amplitude is an AI assistant that uses large language models (LLMs). It combines schema search and content search to provide a customized, accurate, low latency, natural language-based visualization experience to end customers. Ask Amplitude has knowledge of a user’s product, taxonomy, and language to frame an analysis. It uses a series of LLM prompts to convert the user’s question into a JSON definition that can be passed to a custom query engine. The query engine then renders a chart with the answer, as illustrated in the following figure.

Ask Amplitude generates charts in response to user queries

Amplitude’s search architecture evolved to scale, simplify, and cost-optimize for our customers, by implementing semantic search and Retrieval Augmented Generation (RAG) powered by Amazon OpenSearch Service. In this post, we walk you through Amplitude’s iterative architectural journey and explore how we address several critical challenges in building a scalable semantic search and analytics platform.

Our primary focus was on enabling semantic search capabilities and natural language chart generation at scale, while implementing a cost-effective multi-tenant system with granular access controls. A key objective was optimizing the end-to-end search latency to deliver rapid results. We also tackled the challenge of empowering end customers to securely search and use their existing charts and content for more sophisticated analytical inquiries. Additionally, we developed solutions to handle real-time data synchronization at scale, making sure constant updates to incoming data could be processed while maintaining consistently low search latency across the entire system.

RAG and vector search with Ask Amplitude

Let’s take a brief look at why Ask Amplitude uses RAG. Amplitude collects omnichannel customer data. Our end customers send data on user actions that are performed in their platforms. These actions are recorded as user-generated events. For example, in the case of retail and ecommerce customers, the types of user events include “product search,” “add to cart,” “checked out,” “shipping option,” “purchase,” and more. These events help define the customer’s database schema, outlining the tables, columns, and relationships between them. Let’s consider a user question such as “How many people used 2-day shipping?” The LLM needs to determine which elements of the captured user events are pertinent to formulating an accurate response to the query. When users ask a question to Ask Amplitude, the first step is to filter the relevant events from OpenSearch Service. Rather than feeding all event data to the LLM, we take a more selective approach for both cost and accuracy reasons. Because LLM usage is billed based on token count, sending complete event data would be unnecessarily expensive. More importantly, providing too much context can degrade the LLM’s performance—when faced with thousands of schema elements, the model struggles to reliably identify and focus on the relevant information. This information overload can distract the LLM from the core question, potentially leading to hallucinations or inaccurate responses. This is why RAG is the preferred approach. To retrieve the most relevant items from the product usage schema, a vector search is performed. This is effective even in situations when the question might not refer to the exact words that are in the customer’s schema. The following sections walk through the iterations of Amplitude’s search journey.

Initial solution: No semantic search

We used Amazon Relational Database Service (Amazon RDS) for PostgreSQL as the primary database to store our people, events, and properties data. However, as the following diagram shows, we had a separate, third-party store to implement keyword search. We had to bring in data from PostgreSQL to this third-party search index and keep it updated.

Initial Solution: No Semantic Search

This architecture was simple but had two key shortcomings: there were no natural language capabilities in our search index, and the search index supported only keyword search.

Iteration 1: Brute force cosine similarity

To improve our search capability, we considered several prototypes. Because data volumes for most customers were not very large, it was quick to build a vector search prototype using PostgreSQL. We transformed user interaction data into vector embeddings and used array cosine similarity to compute similarity metrics across the dataset. This alleviated the need for custom similarity computation. The vector embeddings captured nuanced user behavior patterns using PostgreSQL capabilities without additional infrastructure overhead. This is generally called the brute force method, where an incoming query is matched against all embeddings to find its top (K) neighbors by a distance measure (cosine similarity in this case). The following diagram illustrates this architecture.

Iteration 1: Brute force cosine similarity

Enabling semantic search was a big improvement over traditional search for users who might use different terms to refer to the same concepts, such as “hours of video streamed” or “total watch time”. However, although this worked for small datasets, it was slow because the brute force method had to compute cosine similarity for all pairs of vectors. This was amplified as the number of elements in the events schema, the complexity of questions, and expectations of quality grew. Additionally, Ask Amplitude answers needed to blend both semantic and keyword search. To support this, each search query had to be implemented as a three-step process involving multiple calls to separate databases:

  1. Retrieve the semantic search results from PostgreSQL.
  2. Retrieve the keyword search results from our search index.
  3. In the application, semantic search results and keyword search results were combined using pre-assigned weights, and this output was dispatched to the Ask Amplitude UI.

This multi-step manual approach made the search process more complex.

Iteration 2: ANN search with pgvector

As Amplitude’s customer base grew, Ask Amplitude needed to scale to accommodate more customers and larger schemas. The goal was not just to answer the question at hand, but to teach the user how to build an end-to-end analysis by guiding them iteratively. To this end, the embeddings needed to store and index contextually rich semantic content. The team experimented with bigger, higher dimensionality embeddings and had anecdotal observations of vector dimensionality appearing to impact the effectiveness of the retrieval. Another requirement was to support multilingual embeddings.

To support a more scalable k-NN search, the team switched to pgvector, a PostgreSQL extension that provides powerful functionalities for with vectors in high-dimensional space. The following diagram illustrates this architecture.

Iteration 2: ANN search with pgvector

Pgvector was able to support k-nearest neighbor (k-NN) similarity search for larger dimensionality vectors. As the number of vectors grew, we switched to indexes that allowed approximate nearest neighbor (ANN) search, such as HNSW and IVFFlat.

For customers with larger schemas, calculating brute force cosine similarity was slow and expensive. We found a performance difference when we moved to ANN enabled by pgvector. However, we still needed to deal with the complexity introduced by the three-step process of querying PostgreSQL for semantic search, a separate search index for keyword search, and then stitching it all together.

Iteration 3: Dual sync to keyword and semantic search with OpenSearch Service

As the number of customers grew, so did the number of schemas. There were hundreds of millions of schema entries in the database, so we sought a performant, scalable, and cost-effective solution for k-NN search. We explored OpenSearch Service and Pinecone. We chose OpenSearch Service because we could combine keyword and vector search capabilities. This was convenient for four reasons:

  • Simpler architecture – Positioning semantic search as a capability in an existing search solution, as we observed in OpenSearch Service, makes for a simpler architecture than treating it as a separate specialized service.
  • Lower-latency search – The ability to effectively organize and catalog search data was fundamental to how we generated answers. Augmenting semantic search to our existing pipeline by combining both into one query provided lower latency querying.
  • Reduced need for data synchronization – Keeping the database in sync with the search index was critical to the accuracy and quality of answers. With the alternatives that we looked at, we would have to maintain two synchronization pipelines, one for keyword search index and the other for a semantic search index, complicating the architecture and increasing the chances of experiencing out-of-sync results between keyword and semantic search results. Synchronizing them into one place was easier than synchronizing them into multiple places and then combining the signals at query time. With a combined keyword and vector search capabilities of OpenSearch Service, we now needed to synchronize only one primary database on PostgreSQL with the search index.
  • Minimized performance impact to source data updates – We found that synchronizing data to another search index is a complex problem because our dataset changes constantly. With every new customer, we had hundreds of updates every second. We had to make sure the latency of these updates wasn’t impacted by the sync process. Collocating search data with vector embeddings obviated the need for multiple sync processes. This helped us avoid additional latency in the primary database, due to the sync processes encroaching upon database update traffic.

Although our previous third-party search engine specialized in fast ecommerce search, this wasn’t aligned with Amplitude’s specific needs. By migrating to OpenSearch Service, we simplified our architecture by reducing two synchronization processes to one. We phased out the current search platform gradually. This meant we temporarily continued to have two synchronization processes, one with current platform and another to the combined keyword and semantic search index on OpenSearch Service, as shown in the following diagram.

Iteration 3: Dual sync to keyword and semantic search with OpenSearch Service

In addition to the pros of k-NN search identified in the previous iteration, moving to OpenSearch Service helped us realize three key benefits:

  • Reduced latency – Instead of collocating the embeddings with primary data, we were able to collocate with our search index. The search index is where our application needed to run our queries to pick out user events that are relevant to the question being asked and send this as context sent to the LLM. Because the search text, metadata, and embeddings were all in one place, we needed only one hop for all our search requirements, thereby improving latency.
  • Reduced compute power – We had anywhere between 5,000–20,000 elements in the user events schema. We didn’t need to send the entire schema to the LLM, because each user query required only 20–50 relevant elements. With the efficient filtering capabilities of OpenSearch Service, we were able to narrow down the vector search space by using tenant-specific metadata, significantly reducing compute requirements across our multi-tenant environment.
  • Improved scalability – With OpenSearch Service, we could take advantage of additional capabilities such as HNSW product quantization (PQ) and byte quantization. Byte quantization made it possible to handle the scale of millions of vector entries with minimal reduction in recall, but with improvement to cost and latency.

However, in this interim solution, our data wasn’t fully migrated to OpenSearch Service yet. We still had the old pipeline along with the new pipeline, and had to perform dual syncing. This was only temporary, as we phased out the old search index, and the old pipeline served as a baseline to compare with in terms of performance and recall.

Iteration 4: Hybrid search with OpenSearch Service

In the final architecture, we were able to migrate all our data to OpenSearch Service, which also served as our vector database, as shown in the following diagram.

Iteration 4: Hybrid search with OpenSearch Service

We now had to perform just one data synchronization from the PostgreSQL database to the combined search and vector index, allowing the resources on the database to focus on transactional traffic. OpenSearch Service provides merging, weighting, and ranking of the search results as part of the same query. This obviated the need to implement them as a separate module in our application, effectively resulting in a single, scalable hybrid search (combined keyword-based (lexical) search and vector-based (semantic) search). With OpenSearch Service, we could also experiment with the new integration with Amazon Personalize.

Evolving RAG to draw upon user-generated content

Our customers wanted to ask deeper questions about their product usage that couldn’t be answered just by looking at the schema (the structure and names of the data columns) alone. Simply knowing the column names in a database doesn’t necessarily reveal the meaning, values, or proper interpretation of that data. The schema alone provides an incomplete picture. A naïve approach would be to index and search all data values instead of searching just the schema. Amplitude avoids this for scalability reasons. The cardinality and volume of event data (potentially trillions of event records) makes indexing all values cost prohibitive. Amplitudes hosts about 20 million charts and dashboards across all Amplitude customers. This user-generated content is valuable. We observed that we can better understand the meaning and context by analyzing how other users have previously visualized data.For example, if a user asks about “2-day shipping,” Amplitude first checks if the data schema contains columns with relevant names like “shipping” or “shipping method”. If such columns exist, it then examines the potential values in those columns to find values related to 2-day shipping. Amplitude also searches user-created content (charts, dashboards, and more) to see if anyone else at the company has already visualized data related to 2-day shipping. If so, it can use that existing chart as a reference for how to properly filter and analyze the data to answer the question. To search this content efficiently, Amplitude employs a hybrid approach combining keyword and vector similarity (semantic) searches. For tenant isolation and pruning, we use metadata to filter by customer first, and then vector search.

Conclusion

In this post, we showed you how Amplitude built Ask Amplitude, an AI assistant using OpenSearch Service as a vector database to enable natural language queries of product analytics data. We evolved our system through four iterations, ultimately consolidating keyword and semantic search into OpenSearch Service, which simplified our architecture from multiple sync pipelines to one, reduced query latency by combining search operations, and enabled efficient multi-tenant vector search at scale using features like HNSW PQ and byte quantization. We extended the system beyond schema search to index 20 million user-generated charts and dashboards, using hybrid search to provide richer context for answering customer questions about product usage.

As natural language interfaces become increasingly prevalent, Amplitude’s iterative journey demonstrates the potential for harnessing LLMs and RAG using vector databases such as OpenSearch Service to unlock rich conversational customer experiences. By gradually transitioning to a unified search solution that combines keyword and semantic vector search capabilities, Amplitude overcame scalability and performance challenges while reducing architecture complexity. The final architecture using OpenSearch Service enabled efficient multi-tenancy and fine-grained access control and also facilitated low-latency hybrid search. Amplitude is able to deliver more natural and intuitive analytics capabilities to its customers by generating deeper insights and contextualizing data.

To learn more about how Ask Amplitude helps you express Amplitude-related concepts and questions in natural language, refer to Ask Amplitude. To get started with OpenSearch Service as a vector database, refer to Amazon OpenSearch Service as a Vector Database.


About the authors

Jeffrey Wang

Jeffrey Wang

Jeffrey is a Co-founder & Former Chief Architect, Amplitude. He originated the infrastructure that enables us to scan billions of events every second at Amplitude. He studied Computer Science at Stanford and brings experience building infrastructure from Palantir and Sumo Logic.

Preethi Kumaresan

Preethi Kumaresan

Preethi is a technology leader in machine learning, GenAI, and end-to-end cloud solutions. Currently a Sr. GenAI Solutions Architect at AWS, she brings over 15 years of experience leading teams and products at Google, Cisco, and VMware, as well as high-growth startups. Preethi holds a Master’s degree from the University of California, Santa Cruz, and in her free time, she is an avid traveler, outdoors enthusiast, and snowboarder.

Sekar Srinivasan

Sekar Srinivasan

Sekar is a Sr. Specialist Solutions Architect at AWS focused on Big Data and Analytics. Sekar has over 20 years of experience working with data. He is passionate about helping customers build scalable solutions modernizing their architecture and generating insights from their data. In his spare time he likes to work on non-profit projects, especially those focused on underprivileged Children’s education.

Zero-ETL integrations with Amazon OpenSearch Service

Post Syndicated from Omama Khurshid original https://aws.amazon.com/blogs/big-data/zero-etl-integrations-with-amazon-opensearch-service/

Amazon OpenSearch Service is a fully managed service that reduces operational overhead, provides enterprise-grade security, high availability, and scalability, and enables you to quickly deploy real-time search, analytics, and generative AI applications. OpenSearch itself is an open-source, distributed search and analytics suite that supports a wide range of use cases, including real-time monitoring, log analytics, and full-text search. OpenSearch Service offers zero-ETL integrations with other Amazon Web Service (AWS) services, enabling seamless data access and analysis without the need for maintaining complex data pipelines.

Zero-ETL refers to a set of integrations designed to minimize or eliminate the need to build traditional extract, transform, load (ETL) pipelines. Traditional ETL processes can be time-consuming and difficult to develop, maintain, and scale. In contrast, zero-ETL integrations allow direct, point-to-point data movement and can also support querying across data silos without physically moving the data.

In this post, we explore various zero-ETL integrations available with OpenSearch Service that can help you accelerate innovation and improve operational efficiency. We cover following types of integrations, their key features, architecture, benefits, pricing, limitation and some general best practices.

  1. Log and storage integrations
  2. Database integrations

The following diagram illustrates the zero-ETL integration architecture in AWS, showing how various AWS services feed data into OpenSearch Service and its associated dashboards:

Zero ETL with Amazon OpenSearch Service

Zero-ETL integration with Amazon S3

Amazon OpenSearch Service direct queries with Amazon S3 provides a zero-ETL integration to reduce the operational complexity of duplicating data or managing multiple analytics tools by enabling you to directly query their operational data, reducing costs and time to action.

Key features of this integration include:

  1. In-place querying: You can use rich analytics capabilities of OpenSearch Service SQL and PPL directly on infrequently-queried data stored outside of OpenSearch Service in Amazon S3.
  2. Selective data ingestion: You can choose which data to bring into OpenSearch Service for detailed analysis, optimizing costs and speeding up queries with indexes like skipping or covering indexes.

The zero-ETL integration with Amazon S3 supports OpenSearch Service. For more information on architecture and feature see the post Modernize your data observability with Amazon OpenSearch Service zero-ETL integration with Amazon S3.

In log analytics use cases, we categorize operational log data into two types:

  • Primary data includes the most recent and frequently accessed logs used for real-time monitoring and analysis.
  • Secondary data consists of historical logs that are accessed less frequently but retained for compliance or trend analysis.

You can offload infrequently queried data, such as archival or compliance data, to Amazon S3. With direct query, you can analyze analytics from Amazon S3 without data movement or duplication. However, query performance in OpenSearch Service might slow down when you’re accessing external data sources due to factors like network latency, data transformation, or large data volumes. You can optimize your query performance by using OpenSearch indexes, such as a skipping index, covering index, or materialized view.

While Amazon S3 direct query integration with OpenSearch Service provides on-demand access to data stored in Amazon S3, it is important to remember that OpenSearch’s alerting, monitoring, anomaly detection, and security analytics capabilities can only operate on data that has been explicitly ingested into OpenSearch Service indices. These capabilities would not work with direct query with Amazon S3. However, it will work if the data is indexed with covering or materialized index.

Benefits

With direct queries with Amazon S3, you no longer need to build complex ETL pipelines or incur the expense of duplicating data in both OpenSearch Service and Amazon S3 storage. You also save time and effort by not having to move back and forth between different tools during your analysis.

Pricing

OpenSearch Service separately charges for the compute needed to query your external data in addition to maintaining indexes in OpenSearch Service. Costs for Direct Query is based on the data volume scanned, query execution time, query frequency and frequency with which the indexed data in OpenSearch is kept updated. For more information, see Amazon OpenSearch Service Pricing.

Considerations

In case you are using OpenSearch service to query directly data on Amazon S3, consider the limitations with Direct Query.

Best practices

These are some general and Amazon S3 recommendations for using direct queries in OpenSearch Service. For more information, see Recommendations for using direct queries in Amazon OpenSearch Service.

  • Use the COALESCE SQL function to handle missing columns and ensure results are returned.
  • Use limits on your queries to ensure you aren’t pulling too much data back.
  • If you plan to analyze the same dataset many times, create an indexed view to fully ingest and index the data into OpenSearch Service and drop it when you have completed the analysis.
  • Drop acceleration jobs and indexes when they’re no longer needed.
  • Ingest data into Amazon S3 using partition formats of year, month, day, hour to speed up queries.
  • When you build skipping indexes, use Bloom filters for fields with high cardinality and min/max indexes for fields with large value ranges. Bloom filters are a space efficient probabilistic data structure that lets you quickly check whether an item is possibly in a set. For high-cardinality fields, consider using a value-based approach to improve query efficiency.
  • Use Index State Management to maintain storage for materialized views and covering indexes.

Zero-ETL integration with Amazon CloudWatch Logs

Amazon CloudWatch Logs serves as a centralized monitoring and storage solution for log files generated across various AWS services. This unified logging service offers a highly scalable platform where all your logging data converges into one manageable system. It provides comprehensive functionality for log management, including real-time viewing, pattern searching, field-based filtering, and secure archival capabilities. By presenting all logs chronologically in a unified stream, CloudWatch Logs eliminates the complexity of managing multiple log sources, transforming diverse logging data into a coherent, time-ordered sequence of events.

The zero-ETL integration between Amazon CloudWatch and Amazon OpenSearch Service enables direct log analysis and visualization while avoiding data redundancy, thereby reducing both technical complexity and costs. You can now leverage two additional query languages alongside the existing CloudWatch Logs Insights QL when using CloudWatch Logs, while as an OpenSearch user, you gain the ability to query CloudWatch logs directly.

Review New Amazon CloudWatch and Amazon OpenSearch Service launch an integrated analytics experience, to explore how the integration works between OpenSearch Service and Amazon CloudWatch Logs.

Benefits

  • The enhanced CloudWatch Logs Insights console now incorporates OpenSearch PPL and SQL functionality. Users can perform complex log analysis using SQL JOIN operations and various functions (including JSON, mathematical, datetime, and string operations). The PPL option provides additional data filtering and analysis capabilities.
  • The integration offers ready-to-use dashboards for various AWS services like Amazon Virtual Private Cloud (VPC), AWS CloudTrail, and AWS Web Application Firewall (WAF). These pre-configured visualizations enable quick insights into metrics such as flow patterns, top users, data transfer volumes, and temporal analysis, without requiring manual dashboard configuration.
  • You can now analyze CloudWatch logs through OpenSearch UI Discover and execute SQL and PPL queries. At the writing of this post, the query execution is limited to 50 log groups.
  • The direct access and analysis of CloudWatch data within OpenSearch Service removes the need for traditional ETL processes, eliminates separate data ingestion pipelines and avoids data duplication. This streamlined approach significantly reduces both storage expenses and operational complexity. It delivers a more efficient data management solution that simplifies the entire workflow while maintaining cost-effectiveness.

Pricing

When you use OpenSearch Service direct queries, you incur separate charges for OpenSearch Service and the resource used to process and store your data on Amazon CloudWatch Logs. As you run direct queries, you see charges for OpenSearch Compute Units (OCUs) per hour, listed as DirectQuery OCU usage type on your bill.

  • For interactive queries, OpenSearch Service handles each query with a separate pre-warmed job, without maintaining an extended session.
  • For indexed view queries, the indexed data is stored in an OpenSearch Serverless collection where you are charged for data indexed (IndexingOCU), data searched (SearchOCU), and data stored in GB.

You can find a pricing example on running an OpenSearch dashboard from either OpenSearch UI or CloudWatch Logs (pricing example n°7).

For more pricing information, see Amazon OpenSearch Service Direct Query pricing.

Considerations

In addition to the OpenSearch Service “direct queries” general limitations, if you are direct querying data in CloudWatch Logs, the following limitations apply:

  • The direct query integration with CloudWatch Logs is only available on OpenSearch Service collections and the OpenSearch user interface.
  • OpenSearch Serverless collections have networked payload limitations of 100 MiB.
  • CloudWatch Logs supports VPC Flow Logs, CloudTrail, and AWS WAF dashboard integrations installed from the console.

Best practices

Besides the general recommendations of OpenSearch Service direct querying, when using OpenSearch Service to direct query data in CloudWatch Logs, the following is recommended:

  • Specify the log group names within logGroupIdentifier in logGroups command to query multiple log groups in one query, see Multi-log group functions.
  • Enclose certain fields in backticks to successfully query them when using SQL or PPL commands. Backticks are needed for fields with special characters, such as `@SessionToken` or `LogGroup-A` (non-alphabetic and non-numeric). Refer to CloudWatch Logs Recommendations to see an example.

Zero-ETL integration with Amazon DynamoDB

Amazon DynamoDB zero-ETL integration with OpenSearch Service lets you perform a search on your DynamoDB data by automatically replicating and transforming it without custom code or infrastructure. This zero-ETL integration uses Amazon OpenSearch Ingestion to synchronize data between Amazon DynamoDB and OpenSearch Service cluster or OpenSearch Serverless collection within seconds of it being available.

It uses DynamoDB export to Amazon S3 to create an initial snapshot to load into OpenSearch Service. After the snapshot has been loaded, the plugin uses DynamoDB Streams to replicate any further changes in near real time. Turn on point-in-time recovery (PITR) for export and the DynamoDB Streams feature for ongoing replication.

This feature allows you to capture item-level changes in your table and push the changes to a stream. Every item in tables is processed as an event in OpenSearch Ingestion and can be modified with processors. You can also specify index mapping templates within ingestion pipelines to ensure that your Amazon DynamoDB fields are mapped to the correct fields in your OpenSearch indices.

To learn more, see DynamoDB zero-ETL integration with Amazon OpenSearch Service in the AWS documentation.

When configuring zero-ETL between DynamoDB and OpenSearch Service, consider the differences between the data models. You have the following options with data layout:

  1. Passthrough: Each item in DynamoDB table is directly mapped to one document in OpenSearch Index.
  2. Routing: A single DynamoDB table mapped to multiple OpenSearch Service indices. In DynamoDB, it is common to store denormalized data in one table to optimize for access patterns. For example, a single DynamoDB table containing both customer profiles and order information can be routed to separate OpenSearch Service indices:
    • Customer attributes → ‘customers’ index
    • Order attributes → ‘orders’ index

    You can achieve this by using the conditional routing feature in the OpenSearch ingestion pipeline.

  3. Merge: In some use cases, you need to combine data from multiple DynamoDB tables into a single OpenSearch index. You can use AWS Lambda integration with OpenSearch Ingestion to perform lookups on other DynamoDB tables and merge data from multiple DynamoDB tables.

Pricing

There is no additional cost to use this feature apart from the cost of the existing underlying components, including OpenSearch Ingestion charges OpenSearch Compute Units (OCUs) which is used to replicate data between Amazon DynamoDB and OpenSearch Service. Furthermore, this feature uses Amazon DynamoDB Streams for the change data capture (CDC), and you incur the standard costs for Amazon DynamoDB Streams.

Considerations

Consider the following limitations when you set up an OpenSearch Ingestion pipeline for DynamoDB:

  • At the writing of this post, the OpenSearch Ingestion integration with DynamoDB doesn’t support cross-Region and cross-account ingestion.
  • An OpenSearch Ingestion pipeline supports only one DynamoDB table as its source.

Best practices

For complete information, see Best practices for working with DynamoDB zero-ETL integration and OpenSearch Service

Integration with Amazon Aurora and Amazon RDS

Amazon RDS and Amazon Aurora integration with OpenSearch Service eliminates complex data pipelines and enables near real-time data synchronization between Amazon Aurora and Amazon RDS databases (including RDS for MySQL and RDS for PostgreSQL) with advanced search capabilities on transactional databases. You can use an OpenSearch Ingestion pipeline with Amazon RDS or Amazon Aurora to export existing data and stream changes (such as create, update, and delete) to OpenSearch Service domains and collections. The OpenSearch Ingestion pipeline incorporates change data capture (CDC) infrastructure to provide a high-scale, low-latency way to continuously stream data from Amazon RDS or Amazon Aurora.

This automated process keeps your data consistently up to date in OpenSearch Service, making it readily available for search and analysis purpose. The pipeline ensures data consistency by continuously polling or receiving changes from the Amazon Aurora cluster or Amazon RDS and updating the corresponding documents in the OpenSearch index. OpenSearch Ingestion supports end-to-end acknowledgement to ensure data durability. An OpenSearch Ingestion pipeline also maps incoming event actions into corresponding bulk indexing actions to help ingest documents. This keeps data consistent, so that every data change in Amazon RDS is reconciled with the corresponding document changes in OpenSearch.

For details on the architecture, refer to Integrating Amazon OpenSearch Ingestion with Amazon RDS and Amazon Aurora. To get started, refer to OpenSearch Ingestion pipeline with Amazon RDS or Using an OpenSearch Ingestion pipeline with Amazon Aurora.

Pricing

There is no additional charge for using this feature beyond the cost of your existing underlying resources, such as OpenSearch Service, OpenSearch Ingestion pipelines (OCUs), and Amazon RDS or Amazon Aurora. Additional costs may include storage used for enabling enhanced binlogs for MySQL and WAL logs for PostgreSQL for change data capture. You also incur storage costs for snapshot exports from your database to Amazon S3 used for the initial data.

Considerations

Consider the following limitations when you set up the integration for Amazon RDS or Amazon Aurora:

  • Support both Aurora MySQL or RDS for MySQL (8.0 and above) and Aurora PostgreSQL or RDS for PostgreSQL (16 and above).
  • Requires same-Region and same-account deployment, primary keys for optimal synchronization, and currently has no data definition language (DDL) statement support.
  • The integration only supports one Aurora PostgreSQL database per pipeline.
  • The existing pipeline configuration can’t be updated to ingest data from a different database and/or a different table. To update the database and/or table name of a pipeline, stop the pipeline and restart it with an updated configuration or create a new pipeline.
  • Ensure that the Amazon Aurora or Amazon RDS cluster has authentication enabled using AWS Secrets Manager, which is the only supported authentication mechanism.

Best practices

The following are some best practices to follow while setting up the integration with OpenSearch Service:

  • If a mapping template is not specified in OpenSearch, it automatically assigns field types using dynamic mapping based on the first document received. However, it is always recommended to define field types explicitly by creating a mapping template that suits your requirements.
  • To maintain data consistency, the primary and foreign keys of tables remain unchanged.
  • You can configure the dead-letter queues (DLQ) in your OpenSearch Ingestion pipeline. If you’ve configured the queue, OpenSearch Service sends all failed documents that can’t be ingested due to dynamic mapping failures to the queue.
  • Monitor recommended CloudWatch metrics to measure the performance of your ingestion pipeline.

Zero-ETL integration with Amazon DocumentDB

Amazon Document DB is a fully managed database service built for JSON data management at scale. It offers built-in text and vector search functionalities. By leveraging OpenSearch Service, you can execute search analytics, including features like fuzzy matching, synonym detection, cross-collection queries, and multilingual search capabilities on DocumentDB data.

The zero-ETL integration initiates the process with a full historical data extraction to OpenSearch using an ingestion pipeline. After the initial data load is completed, the pipelines read from Amazon DocumentDB change streams ensuring near real-time data consistency between the two systems. OpenSearch organizes the incoming data into indexes, with flexibility to either consolidate data from a DocumentDB collection into a single index or partition data across multiple indices. The ingestion pipelines synchronize all create, update, and delete operations from the DocumentDB collection, maintaining corresponding document modifications in OpenSearch. This ensures both data systems remain synchronised.

The pipelines offer configurable routing options, allowing data from a single collection to be written to one index or conditionally route to multiple indexes. Users can configure ingestion pipelines to stream data from Amazon DocumentDB to OpenSearch Service through three primary modes namely full load only, streaming change events without initial full load and full load followed by change streams. You can also monitor the state of ingestion pipelines in the OpenSearch service console. Additionally, you can use Amazon Cloudwatch to provide real-time metrics and logs and setting up alerts.

Pricing

There is no additional charge for using this feature apart from the cost of your existing underlying resources, including OpenSearch Service, OpenSearch Ingestion pipelines (OCUs), and Amazon DocumentDB. The integration performs an initial full load of Amazon DocumentDB data and continuously streams ongoing changes to OpenSearch Service using change streams. The change streams feature is disabled by default and does not incur any additional charges until the feature is enabled. Using change streams on a DocumentDB cluster incurs additional read and write input/output (I/O), as well as storage costs.

To learn more on pricing see the DocumentDB pricing page.

Considerations

The following are the limitations for the DocumentDB to OpenSearch Service integration:

  • Only one Amazon DocumentDB collection as the source per pipeline is supported.
  • Cross-region and cross-account data ingestion is not supported.
  • Amazon DocumentDB elastic clusters are not supported, only instance-based clusters are supported.
  • AWS Secrets Manager is the only supported authentication mechanism.
  • You can’t update an existing pipeline configuration to ingest data from a different database and/or a different collection. To update the database and/or collection name of a pipeline, create a new pipeline.

Best practices

The following are some best practices to follow while setting up the DocumentDB zero-ETL with OpenSearch Service:

  • Configure dead-letter queues (DLQ) to handle any failed document ingestion.
  • Configure AWS Secrets Manager and enable secrets rotation to provide the pipeline secure access.
  • If you’re using change streams in DocumentDB, it’s important to extend the retention period to up to 7 days. This ensures you don’t lose any data changes during the ingestion process.

To get started, see zero-ETL integration of Amazon DocumentDB with OpenSearch Service.

Benefits for Database Integrations

With zero-ETL integrations, you can use the powerful search and analytics features of OpenSearch Service directly on your latest database data. These include full-text search, fuzzy search, auto-complete, and vector search for machine learning (ML) workloads—enabling intelligent, real-time experiences that enhance your applications and improve user satisfaction. This integration uses change streams to automate the synchronisation of transactional data from Amazon Aurora, Amazon RDS, Amazon DynamoDB and Amazon DocumentDB to OpenSearch Service without manual intervention. Once the data is available in OpenSearch Service, you can perform real-time searches to quickly retrieve relevant results for your applications.This eliminates the need for manual Extract-Transform-Load (ETL) processes, reduces operational complexity, and accelerates time-to-insight for real-time dashboards, search, and analytics.

Conclusion

In this post, you learned that zero-ETL integrations represent a significant advancement in simplifying data analytics workflows and reducing operational complexity. As you’ve explored throughout this post, these integrations offer several advantages such as elimination of complex ETL pipelines and reduced infrastructure and operational costs by removing the need for intermediate storage and processing that enhance developer productivity.

It is time to accelerate your analytics journey with OpenSearch Service zero ETL – where your data flows seamlessly, eliminating complex pipelines and delivering real-time insights. Get started with Amazon OpenSearch Service or learn more about integrations with other services and applications in the AWS documentation.


About the authors

Omama Khurshid

Omama Khurshid

Omama is GTM Specialist Solutions Architect Analytics at Amazon Web Services. She focuses on helping customers across various industries build reliable, scalable, and efficient solutions. Outside of work, she enjoys spending time with her family, listening to music, and learning new technologies.

Canberk Keles

Canberk Keles

Canberk is an Associate Solutions Architect at Amazon Web Services, helping software companies achieve their business goals by leveraging AWS technologies. He is part of OpenSearch specialist community within AWS and has been guiding customers harness the power of OpenSearch. Outside of work, he enjoys sports, reading, traveling and playing video games.

Amazon OpenSearch Serverless introduces collection groups to optimize cost for multi-tenant workloads

Post Syndicated from Madhusudhan Narayana original https://aws.amazon.com/blogs/big-data/amazon-opensearch-serverless-introduces-collection-groups-to-optimize-cost-for-multi-tenant-workloads/

Today, we’re excited to announce the general availability of the collection groups feature for Amazon OpenSearch Serverless. With this feature you can reduce compute costs for multi-tenant workloads while creating secure tenant boundaries through per-tenant encryption, giving you the flexibility to balance cost efficiency with the exact level of isolation and security your applications requires.

Amazon OpenSearch Serverless is a serverless deployment option for Amazon OpenSearch Service, that eliminates the complexity of infrastructure management for running search and analytics workloads at scale. It automatically provisions and scales resources to deliver fast data ingestion rates and millisecond response times, even as usage patterns change. For organizations that are managing multi-tenant environments, data isolation, where the tenant’s data must be encrypted and protected (often with their own encryption keys), is a compliance requirement.

Previously, OpenSearch Serverless provided maximum security through physical isolation: each AWS Key Management Service key (KMS key) required dedicated OpenSearch Compute Units (OCUs) to maintain complete physical data separation. While this architecture provided the highest level of protection, it created challenges for multi-tenant deployments at scale. For customers managing multiple tenants with shared encryption keys, OCU resources are efficiently pooled, making the economics favorable. However, customers managing large numbers of smaller tenants, each requiring their own KMS key for data isolation, faced a challenge with higher cost. With dedicated OCU resources needed per unique key, the infrastructure costs could become prohibitive when individual tenants required only a fraction of an OCU’s capacity. This particularly impacted service providers wanting to offer bring your own key (BYOK) capabilities to their customers, forcing them to either absorb unsustainable costs or limit their service offerings.

OpenSearch Serverless has always provided flexible capacity management with maximum OCU settings to help you control costs. For most workloads, this model works seamlessly capacity scales up and down in response to demand, so you only pay for what you use. However, some workload patterns are simply better served by having a guaranteed baseline of compute ready to go from the start. Workloads with sudden traffic spikes, high-speed data ingestion pipelines, or load testing scenarios benefit from having capacity pre-allocated, so that the first requests are handled with the same responsiveness as any other. Similarly, multi-tenant architectures and time-sensitive operations often require predictable, consistent performance from the moment a collection becomes active.

Flexible controls with collection groups

Collection groups give you flexible control over security boundaries and resource allocation. Instead of forcing a one-size-fits-all approach, you can now tailor your architecture to match your specific security and cost requirements. Here’s how it works:

  1. Define your security boundary that matches your need: Collection groups is a logical security construct for related collections. Each collection groups maintains strong isolation with physically separated memory, CPU and disk from other collection groups, ensuring robust security boundaries between different security constructs.
  2. Share resources across encryption keys: Allocate collections to your collection groups regardless of whether they share KMS keys or use separate ones. Collections with different encryption keys can now share OCU resources within the same security boundary, dramatically reducing costs while maintaining full encryption protection and logical separation for each tenant.
  3. Deploy with flexible network access: Collection groups support collections with different network access types, allowing you to combine collections with public endpoints and VPC endpoints within the same group. This flexibility lets you match your security and connectivity requirements while benefiting from shared resource management across all collections in the group.
  4. Control cost and performance: Set maximum OCUs to cap spending and minimum OCUs to guarantee baseline performance. This dual control gives you a defined resource envelope for each collection groups, eliminating cost surprises while ensuring consistent performance.
  5. Optimize with insights: Access detailed CloudWatch metrics showing resource consumption, relative usage patterns, and latency across collection groups. These insights help you right-size allocations, identify optimization opportunities, and tune performance based on actual workload behavior.

With collection groups, you now have full control over resource allocation through both minimum and maximum OCU settings

Maximum OCUs: Cost control

Set an upper limit on resources to prevent runaway scaling and control costs per collection groups. This helps ensure you never exceed your budget, even during unexpected traffic spikes. Collection groups capacity limits operate independently from account-level limits. Account-level maximum OCU settings apply only to collections not associated with any collection groups, while collection groups maximum OCU settings apply to collections within that specific group. The sum of (Max OCUs across all your collection groups + Max OCU setting at the account level) should be less than your Service Quota Max OCUs allowed for your account. This separation gives you granular cost control across different security contexts.

Minimum OCUs: Performance guarantees

Define the baseline compute resources that will always be allocated to your collection groups, for consistent performance and resource availability. These OCUs are reserved exclusively for your collection groups and provide:

  • Instant availability with no cold starts: Your collections benefit from instant availability without scaling delays. Resources are always warm and ready, eliminating scaling delays when traffic arrives.
  • Guaranteed capacity: Resources are always available, even during periods of low activity or when competing with other collection groups, ensuring predictable performance even during low-traffic periods.
  • Predictable costs: Minimum OCUs are charged continuously, providing you with reserved capacity in exchange for predictable billing giving you cost certainty in exchange for guaranteed performance. This reserved baseline serves as the foundation for auto-scaling, which expands capacity up to your maximum limit as demand increases.

This combination gives you the flexibility to balance cost optimization with performance guarantees based on your specific requirements.

Multi-tenant cost economics with collection groups

Managing costs in multi-tenant architectures has always required balancing isolation, performance, and efficiency often at the expense of one another. Collection groups change that equation by enabling shared capacity across collections without sacrificing security boundaries. The following details how this plays out when you work with collection groups or without.

Before collection groups: Consider a customer with 10 tenants, each requiring their own KMS key for data isolation. Most of these tenants have modest data requirements typically 10-100GB, with the majority on the smaller end of that range. Managing dedicated resources for each tenant’s encryption key, regardless of their actual capacity needs, created operational complexity and cost challenges at scale.

With collection groups: The same customer can now group their tenants with similar security requirements into the collection groups, sharing OCU resources across collections. Tenants requiring only a small portion of OCU capacity no longer force the allocation of dedicated resources, reducing costs by up to 90% for large number of smaller tenant workloads.

With minimum OCU configuration: Premium tenants can be placed in collection groups with minimum OCUs set to guarantee performance, while standard tenants use collection groups with lower minimum thresholds for cost efficiency.

The following table illustrates how these cost savings play out across different tenant configurations, comparing infrastructure costs with and without collection groups across varying data sizes and query loads.

Number of tenants with unique KMS keys

Data size and query parameters

Cost with complete data isolation (without collection groups)

Cost with collection groups

Additional comments

10

Data size: 60GB or less

Query: Not needing more than base OCU (1 for redundant collection) compute

$3,500 $350 10x Savings in cost.
10

Data size: 60GB or less

Query: More than base OCU (1 for redundant collection) compute during peak times (For example – 5 additional OCUs per tenant without collection groups & 40 OCUs across all tenants based with collection groups due to benefit of shared infra).

$3500 + Peak time scale out per tenant ($8650) $350+ Peak time scale out ($6912). The system will scale up when there is additional query load, additional OCUs are deployed during this time. However when the load scales back, the system will scale-in to base OCU’s.
10 Data size: Sample data size in GB per tenant [3, 5, 7, 8, 10, 15, 18, 25, 28, 150]

Query: Can handle queries upto certain level with minimum OCU for the data size and then scales out on load.

For the sample data sizes, minimum OCU requirement will be [2, 2, 2, 2, 2, 2, 2, 2, 2, 8] = 26 OCUs [$4492] + Peak time scale out per tenant Minimum cost is determine by the number of OCUs required to hold the data across all tenants (120GB per OCU *2) + Peak time scale out.For the sample data sizes, 8 OCUs [$1382] + Peak time scale out per tenant The system will scale up when there is additional query load, additional OCUs are deployed during this time. However when the load scales back, the system will scale-in to minimum number of OCU required to hold the data.

Note: Above calculations are made with assumption for redundant enabled collections. For non-redundant mode it will be half the above calculations.

Getting started with collection groups

Collection groups and minimum OCU configuration are available in all AWS Regions where OpenSearch Serverless is offered, at no additional charge. Collection groups offers a new organizational feature to create collection groups and add new collections directly to these groups for enhanced management capabilities. While your existing collections will continue to operate unchanged and remain independent of any collection groups, you can immediately start using collection groups for new collections to benefit from improved organization and workflow management.

Currently, only newly created collections can be associated with collection groups, and all collections within a group must be of the same type (search, time series, or vector search). Existing collections continue to operate independently with their current capacity management settings, and you cannot mix different collection types within a single collection groups. You can use the AWS Management Console, AWS CLI, AWS CloudFormation, or AWS CDK to create the collection groups. In the following section we will show you how you can create the collection groups using the OpenSearch Service console.

To create your first collection groups:

  1. Open the OpenSearch Service console.
  2. In the left navigation pane, choose Serverless, then choose Collection groups.
  3. Choose Create collection groups.
  4. For collection groups name, enter a name for your collection groups. The name must be 3-32 characters long, start with a lowercase letter, and contain only lowercase letters, numbers, and hyphens.
  5. (Optional) For Description, enter a description for your collection groups.
  6. In the Capacity management section, configure the OCU limits:
    1. Maximum indexing capacity – The maximum number of indexing OCUs that collections in this group can scale up to.
    2. Maximum search capacity – The maximum number of search OCUs that collections in this group can scale up to.
    3. Minimum indexing capacity – The minimum number of indexing OCUs to maintain for consistent performance.
    4. Minimum search capacity – The minimum number of search OCUs to maintain for consistent performance.
  7. (Optional) In the Tags section, add tags to help organize and identify your collection groups.
  8. Choose Create collection groups.

To assign collection to the collection groups

  1. Open the Amazon OpenSearch Service console.
  2. In the left navigation pane, choose Serverless, then choose Collections.
  3. Choose Create collection.
  4. For Collection name, enter a name for your collection. The name must be 3-28 characters long, start with a lowercase letter, and contain only lowercase letters, numbers, and hyphens.
  5. (Optional) For Description, enter a description for your collection.
  6. In the Collection groups section, select the collection groups you want the collection to be assigned to. A collection can only belong to one collection groups at a time.
    (Optional) You can also choose to Create a new group. This will navigate you to the Create collection groups workflow. After you finish creating the collection groups, return to the step 1 of this procedure to begin creating your new collection.
  7. Continue through the workflow to create the collection.

Managing collection groups

Once you’ve created your collection groups, you can update their settings as your architecture evolves. The Amazon OpenSearch Serverless documentation provides step-by-step guidance on how to edit and delete collection groups, including updating OCU limits and modifying group configurations using the AWS Management Console, CLI, and CloudFormation.

Conclusion

OpenSearch Serverless collection groups transform how you can architect multi-tenant deployments by offering flexible deployment modes that balance security requirements with operational efficiency. You can now choose the collection groups where you define logical security boundaries that allow collections, regardless of whether they share the same KMS key or use different KMS keys to share OCU resources.

This flexibility directly addresses the cost challenges that previously made multi-tenant deployments prohibitive. By consolidating collections within collection groups, you can reduce infrastructure costs while maintaining robust encryption and tenant isolation. Configuring both minimum and maximum OCUs for each collection groups solves the cold-start and capacity guarantee challenges: minimum OCUs ensure your collections maintain ready compute resources to handle high-speed ingestion, sudden traffic spikes, and load testing without performance degradation. Maximum OCUs provide cost predictability and spending controls. This dual configuration gives you a defined resource envelope that eliminates both the uncertainty of cold starts and the risk of runaway costs.

To dive deeper into the collection groups and minimum OCU configuration, visit the Amazon OpenSearch Serverless documentation.

About the authors

Madhusudhan Narayana

Madhusudhan Narayana

Madhusudhan is Senior Software Engineer with Amazon Web Services. He is focused on OpenSearch Service and has years of experience in software engineering, distributed and autonomous systems. He holds a MS in Computer Science.

Prashant Agrawal

Prashant Agrawal

Prashant is a Sr. Search Specialist Solutions Architect with Amazon OpenSearch Service. When not working, you can find him traveling and exploring new places. In short, he likes doing Eat → Travel → Repeat.

Xian Huang

Xian Huang

Xian is a Product Marketing Manager at AWS.

Improving order history search using semantic search with Amazon OpenSearch Service

Post Syndicated from Shwetabh . original https://aws.amazon.com/blogs/big-data/improving-order-history-search-using-semantic-search-with-amazon-opensearch-service/

If you’ve ever shopped on Amazon, you’ve used Your Orders. This feature maintains your complete order history dating back to 1995, so you can track and manage every purchase you’ve made. The order history search feature lets you find your past purchases by entering keywords in the search bar. Beyond just finding items, it provides a straightforward way to repurchase the same or similar items, saving you time and effort.

Various features across Amazon’s shopping experience, such as Rufus and Alexa, use order history search to help you find your past purchases. Therefore, it’s important that order history search can locate your past purchased items as accurately and quickly as possible.

In this post, we show you how the Your Orders team improved order history search by introducing semantic search capabilities on top of our existing lexical search system, using Amazon OpenSearch Service and Amazon SageMaker.

Limitations of lexical search

Order history search uses lexical matching to find items from the entire order history of a customer that match at least one word of the search keywords. For example, if a customer searches for “orange juice,” the system retrieves all orange juice items as well as fresh oranges and other fruit juices the customer had previously ordered. Although lexical matching can provide a high recall of items with terms matching the search keywords precisely, it doesn’t work well for related or generic search keywords, like “health drinks” in this example.

Since the launch of Rufus, Amazon’s AI-enabled shopping assistant, a growing number of customers are experiencing a streamlined and richer shopping journey, including searching for their previous purchases with Rufus. Customers can now ask “Show me healthy drinks” without worrying about using lengthy, more precise terms like “kombucha”, “green tea”, and “protein shakes”. This makes the search experience more conversational and intent-based, presenting an opportunity to make item discovery more intuitive. For Rufus to answer order history searches with the same intuitive experience such as “Show me the healthy drinks I bought last year”, the underlying order history data store (“Your Orders”) needs semantic search capability to understand the underlying semantics of search keywords beyond the conventional lexical matching.

Challenges implementing semantic search

Implementing semantic search at our scale presented several technical challenges:

  • Scale – We needed to enable semantic search across billions of records corresponding to customers’ order history globally.
  • Zero downtime – We needed to keep the system 100% available while making changes on the backend to introduce semantic search.
  • Preventing search quality degradation – Semantic search is intended to improve the quality of search results. However, in some cases, it can reduce search quality. For example, if a customer remembers their item name exactly and wants to find only items matching that name, surfacing similar items in addition to the exactly matching items will increase crowding in results and make it harder to find the relevant item. Similarly, semantic search will not work for cases where the customer intends to search by identifier values, like order ID, which lack an inherent semantic meaning. For these scenarios, we use lexical search only.

Solution overview

Semantic search is powered by large language models (LLMs), which are mostly trained on human languages. These models can be adapted to take a piece of text in any language they were trained in and emit an embedding vector of a fixed length, irrespective of the input text length. By design, embedding vectors capture the semantic meaning of input text such that two semantically similar text strings have high cosine similarity computed on their respective embedding vectors. For semantic search on order history, the input text subject to embedding generation and similarity computation are the customer search phrases and the product text of purchased items.

We divide our solution into two parts:

  • Improving system scalability and resiliency for handling requests at scale – Before implementing semantic search, we needed to ensure our infrastructure could handle the increased computational load, leading us to adopt a cell-based architecture. This step is not needed for every use case, but systems with very high scale in terms of request or data volume can benefit a lot from its use before implementing a resource-intensive use case like semantic search.
  • Implementing semantic search – We began by evaluating the available embedding models, using the offline evaluation capabilities of Amazon Bedrock to test different models. After we selected our model, we could establish the infrastructure for generating embedding vectors.

Improving system scalability and resiliency

We used the cell-based architecture design pattern for improving our scalability and resiliency. A cell-based design entails partitioning the system into identical, smaller, self-contained chunks, or cells, which handle only a part of the overall traffic received by the system. The following diagram shows a high-level representation of a cell-based design for order history search.

Cell-based architecture diagram showing customer request routing to Amazon OpenSearch Service domains via hash-based partitioning

Each cell serves a defined subset of our customers. Cells don’t need to communicate with one another to serve a customer request. Each customer is assigned to a cell and each request from that customer is routed to that cell. The OpenSearch Service domain in each cell holds data only for the subset customers that it is supposed to serve. The number of cells (N) and distribution of data among those cells depends on the business use case, but the goal is to achieve as even a distribution of data and traffic as possible.

The routing logic can be kept as simple or as sophisticated as the use case requires it to be. The cell assignment values can either be computed at runtime for each request, or they can be computed one time and written to a cache or persistent data store like Amazon DynamoDB, from where cell assignment values can be fetched for subsequent requests. For order history search, the logic was simple and quick enough to be executed at runtime for each request. Looking up cell assignment from a persistent data store is especially useful for cases where there is a risk of some cells becoming “heavier” than others over time. In such cases, it becomes easier to redistribute the heavy cell’s data by simply overriding cell assignment values for specific keys in the data store, instead of having to change the partitioning logic immediately, which might have an impact on data distribution across all the cells.

As the system’s load grows, the number of cells in the system can be increased to handle the additional traffic. Even without increasing the number of cells in the system, we can redistribute current data among the existing N cells by reassigning some keys from one or more heavily populated cells to different lightly populated cells to spread out the load more evenly across all the cells and make more efficient use of the infrastructure.

A cell-based architecture also helps make the system more resilient. For example, if we lose one cell, our capacity is diminished only by 1/N, instead of 100%. This arrangement can also be improved to reduce the capacity loss even further by assigning partitioning keys to two or more cells such that they get written to two or more cells. In such cases, loss of a single cell does not result in data loss.

Implementing semantic search

Implementing semantic search for our order history search required several key decisions and technical steps. We began by evaluating the available embedding models, using the offline evaluation capabilities of Amazon Bedrock to test different models against our specific business domain requirements. This evaluation process helped us identify which model would deliver the best performance for our use case. After we selected our model, we needed to establish the infrastructure for generating embedding vectors. We containerized our embedding model and registered it in Amazon Elastic Container Registry (Amazon ECR), then deployed it using SageMaker inference endpoints to handle the actual vector computation at scale.

For the search infrastructure itself, we chose OpenSearch Service to implement our semantic search capabilities. OpenSearch Service provided both the vector storage we needed and the search algorithms required to deliver relevant results to our users.

One of our biggest challenges was updating our historical data to support semantic search on existing orders. We built a data processing pipeline using AWS Step Functions to orchestrate the workflow and AWS Lambda functions to handle the actual vector generation for our legacy data, so we could provide semantic search for all the records we wanted to.

The following diagram illustrates the high-level architecture.

Architecture diagram showing read-flow and write-flow for semantic search using Amazon OpenSearch Service and Amazon SageMaker embedding vectors

Model evaluation and selection

Order history search uses an embedding model trained on Amazon-specific data. Domain-specific training is critical because the generated embedding vectors must work well for the business context to return quality results.

We used an LLM-as-a-judge methodology with Anthropic’s Claude on Amazon Bedrock to evaluate candidate models. Anthropic’s Claude received prompts containing anonymized item text and search phrases from customer order history, then filtered and ranked items by relevance. These results served as ground truth for comparison.

We evaluated models using standard ranking metrics:

  • Normalized Discounted Cumulative Gain (NDCG) – Measures ranking quality against ideal order
  • Mean Reciprocal Rank (MRR) – Considers position of first relevant item
  • Precision – Rates accuracy of retrieved results
  • Recall – Rates ability to retrieve all relevant items

This process helped us determine the best model.

Retrieval strategy: Customer-scoped comprehensive search

Order history search has two key requirements:

  • Search only through the requesting customer’s order history – We don’t want items from one customer’s order history showing up in search results for another customer
  • Search all of that customer’s history – We don’t want to miss showing an item that would have been relevant for the customer’s search phrase just because the search algorithm missed evaluating it for some reason

Our approach involves using OpenSearch Service to retrieve all items for the customer who issued the search query, calculating relevance scores for each of them against the search phrase, sorting by score, and returning top K results. This provides comprehensive results coverage for each customer.

Vector storage with OpenSearch Service

We used two OpenSearch Service features for efficient vector storage and search:

  • knn_vector datatype – Built-in support for storing embedding vectors. Existing domains can add this field type without reindexing, enabling exact kNN search across all records. We didn’t need approximate kNN because the number of records for most customers was small enough for exact kNN to scale.
  • Scripted scoring – Painless scripts compute vector similarity server-side, reducing client complexity and maintaining low latency.

Hybrid search

Hybrid search refers to combining the results of lexical and semantic search to benefit from the strengths of each. The hybrid query capabilities of OpenSearch Service simplify implementing hybrid search by letting clients specify both types of queries in a single request. OpenSearch Service runs both queries in parallel, merges their results, normalizes the relevance scores of the sub-queries, and sorts results by the provided sort order (relevance score by default) before returning them to clients.

This gives clients the best of both types of searches. For example, there are certain scenarios where the search phrase doesn’t make much sense semantically, like when customers search by their orderId values. Semantic search is not designed for such cases; these are best served using keyword matching.

The hybrid search functionality helped save implementation effort and potential latency increase for order history search.

Updating historical data

After the infrastructure has been set up, newly ingested records are persisted with the relevant embedding vectors and support semantic search on those records. However, when customers search, they typically search for products they had purchased earlier. Therefore, the system might not help improve customer experience much unless the older records are updated to include the relevant embeddings. The approach to populate this data depends on the scale of the problem at hand.

Releasing the change to minimize potential customer impact

Our final step was to release the change to clients in a manner such that the impact of any potential problems is as small as possible. There are multiple ways to do that, including:

  • Implementing semantic search in a manner such that any transient issues in the semantic search flow make the logic fall back to lexical-only search, instead of failing the request completely. Even if semantic search doesn’t execute, the system should still be able to return results of lexical search to the client, instead of empty results.
  • Gating the change such that the default behavior remains lexical-only search and clients who need the semantic search feature must pass an additional flag in the request, for example, which executes the semantic or hybrid flow only for those requests.
  • Keeping the new flow behind a feature flag during the initial period such that it could be turned off completely if some critical problem is detected.

Examples of improved customer experience

The following are some examples of customer interactions with Rufus that required Rufus to query the respective customer’s order history to answer their question and give them the required pieces of information.

The following screenshots show how semantic search picks up wooden spoons for a “sustainable utensils” query and different kinds of chargers despite not having the keyword “charger” in the title description, in the case of the wall connector.

Two side-by-side screenshots demonstrating semantic search results for sustainable utensils and chargers in an e-commerce interface.

The following screenshots show how semantic search picks up relevant results even though the title description doesn’t include the queried keywords.

Two side-by-side screenshots demonstrating semantic search results for healthy snacks and kids educational items in an e-commerce order interface.

The semantic search feature of order history search helped Rufus fetch them and show to the customers. Before semantic search, Rufus wasn’t able to show any results to customers for such queries.

Business impact

Our solution resulted in the following key business impacts:

  • Customer experience improvements – The solution achieved 10% improvement in query recall, increasing the percentage of searches that return relevant results. It also reduced customer service contacts for issues related to locating past orders.
  • Partner integration success – The solution strengthened natural language processing capabilities for Alexa and Rufus, enhancing their ability to interpret order history queries. It also reduced the need for reranking and postprocessing by partner teams. We improved query success rate by 20%, meaning more customer searches now return at least one relevant item. We also observed enhanced result coverage by 48%, with semantic search consistently surfacing additional relevant matches that lexical search would have missed.

Conclusion

In this post, we showed you how we evolved Amazon order history search to support semantic search capabilities. This transition involved using cutting-edge AI technology while working within existing infrastructure limitations to develop solutions that avoided disruption and maintained SLAs during the feature upgrade. The implementation also involved backfilling, where billions of documents were processed at rates multiple times higher than normal ingestion to compute embedding vectors for previously purchased items. This operation required careful engineering and took advantage of the resilience OpenSearch Service offers even under extreme load.

Beyond the immediate implementation, this foundation enables continued innovation in search technology. The embedding vectors framework can incorporate improved models as they become available, and the architecture supports expansion into new capabilities such as personalization and multi-modal search.

You can get started with exact k-NN search today following the instructions in Exact k-NN search. If you’re looking for a managed solution for your OpenSearch cluster, check out Amazon OpenSearch Service.


About the authors

Shwetabh

Shwetabh

Shwetabh is a Senior Software Engineer at Amazon with interests in distributed systems and machine learning. Outside of work, he’s an avid reader with a particular love for technical deep-dives and thought-provoking non-fiction.

Harshavardhan Miryala

Harshavardhan Miryala

Harshavardhan is a Software Engineer at Amazon. He is passionate about machine learning, with particular interest in information retrieval and distributed computing. Outside of work, he enjoys playing racquet sports and watching football.

Ayush Kumar

Ayush Kumar

Ayush is a Tech Leader at Amazon. He is a passionate builder with an experience of over 14 years and leads the Your Orders Search product. In his spare time, he enjoys watching cricket and playing with his toddler.

Best practices for right-sizing Amazon OpenSearch Service domains

Post Syndicated from Nikhil Agarwal original https://aws.amazon.com/blogs/big-data/best-practices-for-right-sizing-amazon-opensearch-service-domains/

Amazon OpenSearch Service is a fully managed service for search, analytics, and observability workloads, helping you index, search, and analyze large datasets with ease. Making sure your OpenSearch Service domain is right-sized—balancing performance, scalability, and cost—is critical to maximizing its value. An over-provisioned domain wastes resources, whereas an under-provisioned one risks performance bottlenecks like high latency or write rejections.

In this post, we guide you through the steps to determine if your OpenSearch Service domain is right-sized, using AWS tools and best practices to optimize your configuration for workloads like log analytics, search, vector search, or synthetic data testing.

Why right-sizing your OpenSearch Service domain matters

Right-sizing your OpenSearch Service domain provides optimal performance, reliability, and cost-efficiency. An undersized domain leads to high CPU utilization, memory pressure, and query latency, whereas an oversized domain drives unnecessary spend and resource waste. By continuously matching domain resources to workload characteristics such as ingestion rate, query complexity, and data growth, you can maintain predictable performance without overpaying for unused capacity.

Beyond cost and performance, right-sizing facilitates architectural agility. It helps make sure your cluster scales smoothly during traffic spikes, meets SLA targets, and sustains stability under changing workloads. Regularly tuning resources to match actual demand optimizes infrastructure efficiency and supports long-term operational resilience.

Key Amazon CloudWatch metrics

OpenSearch Service provides Amazon CloudWatch metrics that offer insights into various aspects of your domain’s performance. These metrics fall into 16 different categories, including cluster metrics, EBS volume metrics, and instance metrics. To determine if your OpenSearch Service domain is misconfigured, monitor these common symptoms that indicate resizing or optimization may be necessary. These are caused by imbalances in resource allocation, workload demands, or configuration settings. The following table summarizes these parameters:

CloudWatch Metrics Parameter
CPU Utilization Metrics CPUUtilization: Average CPU usage across all data nodes.

  • Optimal range: 60-80% for sustained workloads

Primary control plane CPU utilization (for dedicated primary nodes): Average CPU usage on primary nodes.

  • Optimal range: Under normal conditions <50%
Memory Utilization Metrics JVMMemoryPressure: Percentage of heap memory used across data nodes.

  • Optimal range: 65–85%

Note: With Garbage First Garbage Collector (G1GC), JVM may delay collections to optimize performance. Evaluate JVMMemoryPressure together with GC metrics (Old Gen usage and GC pause time) to confirm true pressure trends.

MasterJVMMemoryPressure: Heap usage on dedicated primary nodes.

  • Optimal range: <80%

Note: Occasional spikes are normal during state updates; sustained high memory pressure warrants scaling or tuning.

Storage Metrics StorageUtilization: Percentage of storage space used.

  • Optimal range: 70–85%

FreeStorageSpace: Available storage in MB.

  • Critical threshold: When approaching the read-only threshold.

Node Level Search and Indexing Performance

(These latencies are not per-request latencies or rate, but at node level based on shards assigned to a node.)

SearchLatency: Average time for search requests.

  • Baseline establishment: Monitor during normal operations.

IndexingLatency: Average time for indexing operations.

  • Impact: Can indicate CPU or I/O bottlenecks.

SearchRate and IndexingRate: Requests per minute for search and indexing.

  • Usage: Correlate with latency metrics to understand performance impact.
Cluster Health Indicators ClusterStatus.yellow and ClusterStatus.red:

  • Yellow status: Some replica shards are unassigned.
  • Red status: Some primary shards are unassigned (data loss risk).

Nodes

  • What it measures: Number of nodes in the cluster.
  • Usage: Track node failures and recovery patterns.

Signs of under-provisioning

Under-provisioned domains struggle to handle workload demands, leading to performance degradation and cluster instability. Look for sustained resource pressure and operational errors that signal the cluster is running beyond its limits. For monitoring, you can set CloudWatch alarms to catch early signals of stress and prevent outages or degraded performance. The following are critical warning signs:

  • High CPU utilization for data nodes (>80%) sustained over time (such as more than 10 minutes)
  • High CPU utilization for primary nodes (>60%) sustained over time (such as more than 10 minutes)
  • JVM memory pressure consistently high (>85%) for data and primary nodes
  • Storage utilization reaching high (>85%)
  • Increasing search latency with stable query patterns (increasing by 50% from baseline)
  • Frequent cluster status yellow/red events
  • Node failures under normal load conditions

When resources are constrained, the end-user experience suffers with slower searches, failed indexing, and system errors. The following are key performance impact indicators:

Remediation recommendations

The following table summarizes CloudWatch metric symptoms, possible causes, and potential solutions.

CloudWatch metric symptom Causes and solution
FreeStorageSpace drops <20%

Storage pressure occurs when data volume outgrows local storage due to high ingestion, long retention without cleanup, or unbalanced shards. Lack of tiering (such as UltraWarm) further worsens capacity issues.

Solution: Free up space by deleting unused indexes or automating cleanup with ISM and use force merge on read-only indexes to reclaim storage. If pressure persists, scale vertically or horizontally, use UltraWarm or cold storage for older data, and adjust shard counts at rollover for better balance.

CPUUtilization and JVMMemoryPressure consistently >70%

High CPU or JVM pressure arises when instance sizes are too small or shard counts per node are excessive, leading to frequent GC pauses. Inefficient shard strategy, uneven distribution, and poorly optimized queries or mappings further spike memory usage under heavy workloads.

Solution: Address high CPU/JVM pressure by scaling vertically to larger instances (such as from r6g.large to r6g.xlarge) or adding nodes horizontally. Optimize shard counts relative to heap size, smooth out peak traffic, and use slow logs to pinpoint and tune resource-heavy queries.

SearchLatency or IndexingLatency spikes >500 milliseconds

Thread pool rejections often stem from resource contention like high CPU/JVM pressure or GC pauses. Inefficient shard sizing, over-sharding, and overly complex queries (deep aggregations, frequent cache evictions) further increase overhead and push tasks into rejection.

Solution: Reduce query latency by optimizing queries with profiling, tuning shard sizes (10–50 GB each), and avoiding over-sharding. Improve parallelism by scaling the cluster, adding replicas for read capacity, increasing cache through larger nodes, and setting appropriate query timeouts.

ThreadpoolRejected metrics indicate queued requests

Thread pool rejections occur when high concurrent requests overflow queues beyond capacity, especially with undersized nodes limited by vCPU-based threads. Sudden unscaled traffic spikes further overwhelm pools, causing tasks to be dropped or delayed.

Solution: Mitigate thread pool rejections by enforcing shard balance across nodes, scaling horizontally to boost thread capacity, and managing client load with retries and reduced concurrency. Monitor search queues, right-size instances for vCPUs, and cautiously tune thread pool settings to handle bursty workloads.

ThroughputThrottle or IopsThrottle reach 1

I/O throttling arises when Amazon EBS or Amazon EC2 limits are exceeded, such as gp3’s 125 MBps baseline, or when burst credits are depleted due to sustained spikes. Mismatched volume types and heavy operations like bulk indexing without optimized storage further amplify throughput bottlenecks.

Solution: Address I/O throttling by upgrading to gp3 volumes with higher baseline or provisioning extra IOPS and consider I/O-optimized instances like i3/i4 families while monitoring burst balance. For sustained workloads, scale nodes or schedule heavy operations during off-peak hours to avoid hitting throughput caps.

Signs of over-provisioning

Over-provisioned clusters show consistently low utilization across CPU, memory, and storage, suggesting resources far exceed workload demands. Identifying these inefficiencies helps reduce unnecessary spend without impacting performance. You can use CloudWatch alarms to track cluster health and cost-efficiency metrics over 2–4 weeks to confirm sustained underutilization:

  • Low CPU utilization for data and primary nodes (<40%) sustained over time
  • Low JVM memory pressure for data and primary nodes (<50%)
  • Excessive free storage (>70% unused)
  • Underutilized instance types for workload patterns

Monitor cluster indexing and search latencies constantly as the cluster is being downsized—these latencies should not increase if the cluster is eliminating unused capacity. Also, it’s recommended to reduce nodes one at a time and continue to observe latencies to continue further downturn. By right-sizing instances, reducing node counts, and adopting cost-efficient storage options, you can align resources to actual usage. Optimizing shard allocation further supports balanced performance at a lower cost.

Best practices for right-sizing

In this section, we discuss best practices for right-sizing.

Iterate and optimize

Right-sizing is an ongoing process, not a one-time exercise. As workloads evolve, continuously monitor CPU, JVM memory pressure, and storage utilization using CloudWatch to make sure they remain within healthy thresholds. Rising latency, queue buildup, or unassigned shards often signal capacity or configuration issues that require attention.

Regularly review slow logs, query latency, and ingestion trends to identify performance bottlenecks early. If search or indexing performance degrades, consider scaling, rebalancing shards, or adjusting retention policies. Periodic reviews of instance sizes and node count help align cost with demand, maintaining 200-millisecond latency targets while avoiding over-provisioning. Consistent iteration helps your OpenSearch Service domain remain performant and cost-efficient over time.

Establish baselines

Monitor for 2–4 weeks after initial deployment and document peak usage patterns and seasonal variations. Record performance during different workload types. Set appropriate CloudWatch alarm thresholds based on your baselines.

Regular review process

Conduct weekly metric reviews during initial optimization and monthly assessments for stable workloads. Conduct quarterly right-sizing exercises for cost optimization.

Scaling strategies

Consider the following scaling strategies:

Vertical scaling (instance types) – Use larger instance types when performance constraints stem from CPU, memory, or JVM pressure, and overall data volume is within a single node’s capacity. Choose memory-optimized instances (such as r8g, r7g, or r7i) for heavy aggregation or indexing workloads. Use compute-optimized instances (c8g, c7g, or c7i) for CPU-bound workloads such as query-heavy or log-processing environments. Vertical scaling is ideal for smaller clusters or testing environments where simplicity and cost-efficiency are priorities.

Horizontal scaling (node count) – Add more data nodes when storage, shard count, or query concurrency increases beyond what a single node can handle. Maintain an odd number of primary-eligible nodes (typically three or five) and use dedicated primary nodes for clusters with more than 10 data nodes. Deploy across three Availability Zones for high availability in production. Horizontal scaling is preferred for large, production-grade workloads requiring fault tolerance and sustained growth. Use _cat/allocation?v to verify shard distribution and node balance:

GET /_cat/allocation/node_name_1,node_name_2,node_name_3

Optimize storage configuration

Use the latest generation of Amazon EBS General Purpose (gp) volumes for improved performance and cost-efficiency compared to earlier versions. Monitor storage growth trends using ClusterUsedSpace and FreeStorageSpace metrics. Maintain data utilization below 50% of total storage capacity to allow for growth and snapshots.

Choose storage tiers based on performance and access patterns—for example, enable UltraWarm or cold storage for large, infrequently accessed datasets. Move older or compliance-related data to cost-efficient tiers (for analytics or WORM workloads) only after ensuring the data is immutable.

Use the _cat/indices?v API to monitor index sizes and refine retention or rollover policies accordingly:

GET /_cat/indices/index1,index2,index3

Analyze shard configuration

Shards directly affect performance and resource usage, so an appropriate shard strategy should be used. The indexes that have heavy ingestion and searches should have a number of shards in the order of number of nodes for better efficiency across all data nodes in the cluster. We recommend keeping shard sizes between 10–30 GB for search workloads and up to 50 GB for log analytics workloads and limit to <20 shards per GB of JVM heap.

Run _cat/shards?v to confirm even shard distribution and no unassigned shards. Evaluate over-sharding by checking JVMMemoryPressure (>80%) or SearchLatency spikes (>200 milliseconds) from excessive shard coordination. Assess under-sharding if IndexingLatency (>200 milliseconds) or low SearchRate indicates limit parallelism. Use _cat/allocation?v to identify unbalanced shard sizes or hot spots on nodes:

GET /_cat/allocation/node_name_1,node_name_2,node_name_3

Handling unexpected traffic spikes

Even well right-sized OpenSearch Service domains can face performance challenges during sudden workload surges, such as log bursts, search traffic peaks, or seasonal load patterns. To handle such unexpected spikes effectively, consider implementing the following best practices:

  • Enable Auto-Tune – Automatically adjust cluster settings based on current usage and traffic patterns
  • Distribute shards effectively – Avoid shard hotspots by using balanced shard allocation and index rollover policies
  • Pre-warm clusters for known events – For expected peak periods (end-of-month reports, marketing campaigns), temporarily scale up before the spike and scale down afterward
  • Monitor with CloudWatch alarms – Set proactive alarms for CPU, JVM memory, and thread pool rejections to catch early stress indicators

Deploy CloudWatch alarms

CloudWatch alarms perform an action when a CloudWatch metric exceeds a specified value for some amount of time to take remediation action proactively.

Conclusion

Right-sizing is a continuous process of observing, analyzing, and optimizing. By using CloudWatch metrics, OpenSearch Dashboards, and best practices around shard sizing and workload profiling, you can make sure your domain is efficient, performant, and cost-effective. Right-sizing your OpenSearch Service domain helps provide optimal performance, cost-efficiency, and scalability. By monitoring key metrics, optimizing shards, and using AWS tools like CloudWatch, ISM, and Auto Scaling, you can maintain a high-performing cluster without over-provisioning.

For more information about right-sizing OpenSearch Service domains, refer to Sizing Amazon OpenSearch Service domains.


Nikhil Agarwal

Nikhil Agarwal

Nikhil is a Sr. Technical Manager with Amazon Web Services. He is passionate about helping customers achieve operational excellence in their cloud journey and working actively on technical solutions. He is also enthusiastic about AI/ML, generative AI, and analytics, and deep dives into customers’ generative AI and Amazon OpenSearch Service specific use cases. Outside of work, he enjoys traveling with family and exploring different gadgets.

Rick Balwani

Rick Balwani

Rick is an Enterprise Support Manager leading a team of Technical Account Managers (TAMs) dedicated to AWS independent software vendor (ISV) customer success. He partners with customers to help them use AWS services effectively while building innovative, cutting-edge solutions. With deep expertise in DevOps and systems engineering, Rick brings technical depth and strategic insight to help ISVs scale and optimize their AWS environments.

Arun Lakshmanan

Arun Lakshmanan

Arun is a Search Specialist with Amazon OpenSearch Service based out of Chicago, IL. He works closely with customers on their OpenSearch journey across various use cases, including vector search, observability, and security analytics.

Amazon OpenSearch Service 101: T-shirt size your domain for e-commerce search

Post Syndicated from Abe Raghib original https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-101-t-shirt-size-your-domain-for-e-commerce-search/

In e-commerce, delivering fast, relevant search results helps users find products quickly and accurately, improving satisfaction and increasing sales. OpenSearch is a distributed search engine that offers advanced search capabilities including advanced full-text and faceted search, customizable analyzers and tokenizers, and auto-complete to help customers quickly find the products they want. It scales to handle millions of products, catalogs and traffic surge. Amazon OpenSearch Service is a managed service that lets users build search workloads balancing search quality, performance at scale and cost. Designing and sizing an Amazon OpenSearch Service cluster correctly is required to meet these demands.

While general sizing guidelines for OpenSearch Service domains are covered in detail in OpenSearch Service documentation, in this post we specifically focus on T-shirt-sizing OpenSearch Service domains for e-commerce search workloads. T-shirt sizing simplifies complex capacity planning by categorizing workloads into sizes like XS, S, M, L, XL based on key workload parameters such as data volume and query concurrency. For e-commerce search, where data growth is moderate and read-heavy queries predominate, this approach offers a flexible, scalable way to allocate the resources without overprovisioning or underestimating needs.

How OpenSearch Service stores indexes and performs queries

E-commerce search platforms handle vast amounts of data and daily data ingestion is typically relatively small and incremental, reflecting catalog changes, price updates, inventory status and user activities like clicks and reviews. Efficiently managing this data and organizing it per OpenSearch Service best practices is crucial in achieving optimal performance. The workload is read-heavy, consisting of user queries with advanced filtering and faceting, especially during sales or seasonal spikes that require elasticity in compute and storage resources.

You ingest product and catalog updates (inventory, listings, pricing) into OpenSearch using bulk APIs or real-time streaming. You index data into logical indexes. How you create and organize indexes in e-commerce has a significant impact on search, scalability and flexibility. The approach depends on the size, diversity and operational needs of the catalog. Small to medium-sized e-commerce platforms commonly use a single, comprehensive product index that stores all product information with product category. Additional indexes may exist for orders, users, reviews and promotions depending on search requirements and data separation needs. Large, diverse catalogs may split products into category-specific indexes for tailored mappings and scaling. You split each index into primary shards, each storing a portion of the documents. To ensure high availability and enhance query throughput, you configure each primary shard with at least one replica shard stored on different data nodes.

Diagram showing Amazon OpenSearch Service cluster with three data nodes implementing primary-replica shard distribution for Products and Reviews indexes. Data Node 1 contains P0-Products (primary), P0-Reviews (primary), and R0-Reviews (replica). Data Node 2 contains P1-Products (primary), R0-Products (replica), and R1-Reviews (replica). Data Node 3 contains P1-Reviews (primary) and R1-Products (replica). Color-coded legend distinguishes primary shards (filled boxes) from replica shards (outlined boxes) for both indexes. This architecture ensures fault tolerance and high availability by distributing primary and replica shards across different nodes.
Diagram 1. How primary and replica shards are distributed among nodes

This diagram shows two indexes (Products and Reviews), each split into two primary shards with one replica. OpenSearch distributes these shards across cluster nodes to ensure that primary and replica shards for the same data do not reside on the same node. OpenSearch runs search requests using a scatter-gather mechanism. When an application submits a request, any node in the cluster can receive it. This receiving node becomes the coordinating node for that specific query. The coordinating node determines which indices and shards can serve the query. It forwards the query to either primary or replica shards and orchestrates the different phases of the search operation and returns the response. This process ensures efficient distribution and execution of search requests across the OpenSearch cluster.

Diagram showing Amazon OpenSearch Service distributed query architecture where an e-commerce application searches for "Blue running shoes." The workflow demonstrates the scatter-gather pattern: (1) Application sends query to Coordinator Node, (2) Coordinator scatters query to three Data Nodes containing index shards, (3) Data Nodes execute searches in parallel, (4) Coordinator gathers and merges results, then returns final ranked results to application. This architecture enables horizontal scalability, parallel processing, and fault tolerance for high-performance search operations across distributed data.
Diagram 2. Tracing a Search query: “blue running shoes”This diagram walks through how a search query–for example, “blue running shoes”flows through your OpenSearch Service domain .

  1. Request: The application sends the search for “blue running shoes” to the domain. One data node acts as the coordinating node.
  2.  Scatter: The coordinator broadcasts the query to either the primary or replica shard for each of the shards in the ‘Products’ index (Nodes 1, 2, and 3 in this case).
  3. Gather: Each data node searches its local shards(s) for “blue running shoes” and returns its own top results (e.g. Node 1 returns its best matches from P0).
  4. Final results: The coordinator merges these partial lists, sorts them into single definitive list of the most relevant shoes, and sends the result back to the app.

Understanding T-Shirt Sizing for E-commerce OpenSearch Service Cluster

Storage planning

Storage impacts both performance and cost. OpenSearch Service offers two main storage options based on query latency requirements and data persistence needs. Selecting the appropriate storage type in a managed OpenSearch Service improves both performance and optimizes cost of the domain. You can choose between Amazon Elastic Block Store( EBS) storage volumes and instance storage volumes (local storage) for your data nodes.

Amazon EBS gp3 volumes offer high throughput, whereas the local NVMe SSD volumes, for example, on the r8gd, i3, or i4i instance families, offer low latency, fast indexing performance and high-speed storage, making them ideal for scenarios where real time data updates and high search throughput are critical for search operations. For search workloads that require a balance between performance and cost, instances backed with EBS GP3 SSD volumes provide a reliable option. This SSD storage offers input/output operations per second (IOPS) that are well-suited for general-purpose search workloads. It also allows users to provision additional IOPS and storage as needed.

When sizing an OpenSearch cluster, start by estimating total storage needs based on catalog size and expected growth. For example, if the catalog contains 500,000 stock keeping units (SKUs), averaging 50KB each; the raw data sums to about 25GB. The size of the raw data, however, is just one aspect of the storage requirements. Also consider the Replica count, indexing overhead (10%), Linux reserves (5%), and OpenSearch Service reserves (20% up to 20GB) per instance while calculating the required storage.

In summary,if you have 25GB of data at any given time who want one replica, the minimum storage requirement is closer to 25 * 2 * 1.1 / 0.95 / 0.8 = 72.5 GB. This calculation can be generalized as follows:

Storage requirement = Raw data * (1 + number of replicas) * 1.45 

This helps ensure disk space headroom on all data nodes, preventing shard failures and maintaining search performance. Provisioning storage slightly beyond this minimum is recommended to accommodate future growth and cluster rebalancing.

Data nodes:

For search workloads, compute-optimized instances (C8g) are well-suited for central processing unit (CPU)-intensive operations like nested queries and joins. However, general-purpose instances like M8g offer a better balance between CPU and memory. Memory-optimized instances (R8g, R8gd) are recommended for memory-intensive operations like KNN search, where larger memory footprint is required. In large, complex deployments, compute-optimized instances like c8g or general-purpose m8g, handle CPU-intensive tasks, providing efficient query processing and balanced resource allocation. The balance between CPU and memory, makes them ideal for managing complex search operations for large-scale data processing. For extremely large search workloads (tens of TB) where latency is not a primary concern, consider using the new Amazon OpenSearch Service Writable warm which supports write operations on warm indices.

Instance Class Best for users who… Examples (AWS) Characteristics
General Purpose have moderate search traffic and want a well-balanced, entry-level setup M family (M8g) Balanced CPU & memory, EBS storage. Good starting point for small to medium-sized catalogs.
Compute Optimized have high queries per second (QPS) search traffic or queries involve scoring scripts or complex filtering C family (C8) High CPU-to-memory ratio. Ideal for CPU-bound workloads like many concurrent queries.
Memory Optimized work with large catalogs, need fast aggregations, or cache a lot in memory R family (R8g) More memory per core. Holds large indices in memory to speed up searches and aggregations.
Storage Optimized update inventory frequently or have so much data that disk access slows things down I family (I3, I4g), Im4gn NVMe SSD and SSD local storage. Best for I/O-heavy operations like constant indexing or large product catalogs hitting disk frequently.

Cluster manager nodes:

For production workloads, it is recommended to add dedicated cluster manager nodes to increase the cluster stability and offload cluster management tasks from the data nodes. To choose the right instance type for your cluster manager nodes, review the service recommendations based on the OpenSearch version and number of shards in the cluster.

Sharding strategy

Once storage requirements are understood, you can investigate the indexing strategy. You create shards in OpenSearch Service to distribute an index evenly across the nodes in a cluster. AWS recommends single product index with category facets for simplicity or partition indexes by category for large or distributed catalogs. The size and number of shards per index play a vital role in OpenSearch Service performance and scalability. The right configuration ensures balanced data distribution, avoids hot spotting, and minimizes coordination overhead on nodes for use cases that prioritizes query speed and data freshness.

For read-heavy workloads like e-commerce, where search latency is the key performance objective, maintain shard sizes between 10-30GB. To achieve this, calculate the number of primary shards by dividing your total index size by your target shard size. For example, if you have a 300GB index and want 20GB shards, configure 15 primary shards (300GB ÷ 20GB = 15 shards). Monitor shard sizes using the _cat/shards API and adjust the shard count during reindexing if shards grow beyond the optimal range.

Add replica shards to improve search query throughput and fault tolerance. The minimum recommendation is to have one replica; you can add more replicas for high query throughput requirements. In OpenSearch Service, a shard processes operations like querying single-threaded, meaning one thread handles a shard’s tasks at a time. Replica shards can serve read requests by distributing them across multiple threads and nodes, enabling parallel processing.

T-shirt sizing for an e-commerce workload

In an OpenSearch T-shirt sizing table, each size label (XSmall, Small, Medium, Large, XLarge) represents a generalized cluster scale category that can help teams translate technical requirements into simple, actionable capacity planning. Each size allows architects to quickly align their catalog size, storage requirements, shard planning, CPU and AWS instance choices to the cluster resources provisioned, making it easier to scale infrastructure as business grows.

By referring to this table, teams can select the category similar to their current workload and use the T-shirt size as a starting point while continuing to refine configuration as they monitor and optimize real-world performance. For example, XSmall is suited for small catalogs with hundreds of thousands of products and minimal search traffic. Small clusters are designed for growing catalogs with millions of SKUs, supporting moderate query volumes and scaling up during busy periods. Medium corresponds to mid-size e-commerce operations handling millions of products and higher search demands, while Large fits large online businesses with tens of millions of SKUs, requiring robust infrastructure for fast, reliable search. XLarge is intended for major marketplaces or global platforms with twenty million or more SKUs, enormous data storage needs, and massive concurrent usage.

T-shirt size Number of Products Catalog Size Storage needed Primary Shard Count Active Shard Count Data Nodes Instance Type Cluster Manager Node instanceType
XSmall 500K 50 GB 145 GB 2 4 [2] r8g.xlarge [3] m8g.large
Small 2M 200 GB 580 GB 8 16 [2] c8g.4xlarge [3] m8g.large
Medium 5M 500 GB 1.45 TB 20 40 [2] c8g.8xlarge [3] m8g.large
Large 10M 1 TB 2.9 TB 40 80 [4] c8g.8xlarge [3] m8g.large
XLarge 20M 2 TB 5.8 TB 80 160 [4] c8g.16xlarge [3] m8g.large
  • T-shirt size: Represents the scale of the cluster, ranging from XS up to XL for high-volume workloads.
  • Number of products: The estimated count of SKUs in the e-commerce catalog, which drives the data volume.
  • Catalog size: The total estimated disk size of all indexed product data, based on typical SKU document size.
  • Storage needed: The actual storage required after accounting for replicas and overhead, ensuring enough room for safe and efficient operation.
  • Primary shard count: The number of main index shards chosen to balance parallel processing and resource management.
  • Active shard count: The total number of live shards (primary with one replica), indicating how many shards need to be distributed for availability and performance.
  • Data node instance type: The recommended instance type to use for data nodes, selected for memory, CPU, and disk throughput.
  • Cluster manager node instance type: The recommended instance type for lightweight, dedicated master nodes which manage cluster stability and coordination.

Scaling strategies for e-commerce workloads

E-commerce platforms continually face challenges with unpredictable traffic surges and growing product catalogs. To address these challenges, OpenSearch Service automatically publishes critical performance metrics to Amazon CloudWatch, enabling users to monitor when individual nodes reach resource limits. These metrics include CPU utilization exceeding 80%, JVM memory pressure above 75%, frequent garbage collection pauses, and thread pool rejections.

OpenSearch Service also provides robust scaling solutions that maintain consistent search performance across varying workload demands. Use the vertical scaling strategy to upgrade instance types from smaller to larger configurations, such as m6g.large to m6g.2xlarge. While vertical scaling triggers a blue-green deployment, scheduling these changes during off-peak hours minimizes impact on operations.

Use the horizontal scaling strategy to add more data nodes for distributing indexing and search operations. This approach proves particularly effective when scaling for traffic growth or increasing dataset size. In domains with cluster manager nodes, adding data nodes proceeds smoothly without triggering a blue-green deployment. CloudWatch metrics guide horizontal scaling decisions by monitoring thread pool rejections across nodes, indexing latency, and cluster-wide load patterns. Though the process requires shard rebalancing and may temporarily impact performance, it effectively distributes workload across the cluster.

Temporary replicas provide a flexible solution for managing high-traffic periods. By increasing replica shards through the _settings API, read throughput can be boosted when needed. This approach offers a dynamic response to changing traffic patterns without requiring more substantial infrastructure changes.

For more information on scaling an OpenSearch Service domain, please refer to How do I scale up or scale out an OpenSearch Service domain?

Monitoring and operational best practices

Monitoring key performance CloudWatch metrics is essential to ensure a well-optimised OpenSearch service domain. One of the key factors is maintaining CPU utilization on data nodes under 80% to prevent query slowdowns. Another metric is ensuring that JVM memory pressure is maintained below 75% on data nodes to prevent garbage collection (GC) pauses that can affect search response time. OpenSearch service publishes these metrics to CloudWatch at 1 minute interval and users can create alarms on these metrics for alerts on the production workloads. Please refer recommended CloudWatch alarms for OpenSearch Service

P95 query latency should be monitored to identify slow queries and optimize performance. Another important indicator is thread pool rejections. A high number of thread pool rejections can result in failed search requests, and affecting user experience. By continuously monitoring these CloudWatch metrics, users can proactively scale resources, optimise queries, and prevent performance bottlenecks.

Conclusion

In this post, we showed how to right-size Amazon OpenSearch Service domains for e-commerce workloads using a T-shirt sizing approach. We explored key factors including storage optimization, sharding strategies, scaling methods, and essential Amazon CloudWatch metrics for monitoring performance.

To build a performant search experience, start with a smaller deployment and iterate as your business scales. Get started with these five steps:

  1. Evaluate your workload requirements in terms of storage, search throughput, and search performance
  2. Select your initial T-shirt size based on your product catalog size and traffic patterns
  3. Deploy the recommended sharding strategy for your catalog scale
  4. Load test your cluster using OpenSearch benchmark and re-iterate until performance requirements are reached
  5. Configure Amazon CloudWatch monitoring and alarms, then continue to monitor your production domain


About the authors

Raaga NG

Raaga NG

Raaga is a Solutions Architect at AWS with over 5 years of experience helping enterprises modernize their technology landscape and build scalable, cloud-native solutions. She partners with customers to translate business requirements into efficient cloud architectures that drive measurable outcomes, supporting their journey from application modernization to AI adoption through thoughtful, customer-centric solutions

Harsh Bansal

Harsh Bansal

Harsh is an Analytics and AI Solutions Architect at AWS. Bansal collaborates closely with clients, assisting in their migration to cloud platforms and optimizing cluster setups to enhance performance and reduce costs. Before joining AWS, Bansal supported clients in leveraging OpenSearch and Elasticsearch for diverse search and log analytics requirements.

Aditya Challa

Aditya Challa

Aditya is a Senior Solutions Architect at AWS. Aditya loves helping customers through their AWS journeys because he knows that journeys are always better when there’s company. He’s a big fan of travel, history, engineering marvels, and learning something new every day.

Abe Raghib

Abe Raghib

Abe is a Senior Solutions Architect at AWS. Abe helps enterprises modernize applications and build scalable, cloud-native solutions. He works with customers to translate business needs into secure, scalable, and cost-effective architectures while supporting their data modernization and AI adoption journeys to drive innovation and measurable business outcomes.

AWS Weekly Roundup: Amazon EC2 M8azn instances, new open weights models in Amazon Bedrock, and more (February 16, 2026)

Post Syndicated from Esra Kayabali original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-ec2-m8azn-instances-new-open-weights-models-in-amazon-bedrock-and-more-february-16-2026/

I joined AWS in 2021, and since then I’ve watched the Amazon Elastic Compute Cloud (Amazon EC2) instance family grow at a pace that still surprises me. From AWS Graviton-powered instances to specialized accelerated computing options, it feels like every few months there’s a new instance type landing that pushes performance boundaries further. As of February 2026, AWS offers over 1,160 Amazon EC2 instance types, and that number keeps climbing.

This week’s opening news is a good example: The general availability of Amazon EC2 M8azn instances. These are general purpose, high-frequency, high-network instances powered by fifth generation AMD EPYC processors, offering the highest maximum CPU frequency in the cloud at 5 GHz. Compared to the previous generation M5zn instances, M8azn instances deliver up to 2x compute performance, 4.3x higher memory bandwidth, and a 10x larger L3 cache. They also provide up to 2x networking throughput and up to 3x Amazon Elastic Block Store (Amazon EBS) throughput compared with M5zn.

Built on the AWS Nitro System using sixth generation Nitro Cards, M8azn instances target workloads such as real-time financial analytics, high-performance computing, high-frequency trading, CI/CD pipelines, gaming, and simulation modeling across automotive, aerospace, energy, and telecommunications. The instances feature a 4:1 ratio of memory to vCPU and are available in 9 sizes ranging from 2 to 96 vCPUs with up to 384 GiB of memory, including two bare metal variants. For more information visit the Amazon EC2 M8azn instance page.

Last week’s launches
Here are some of the other announcements from last week:

  • Amazon Bedrock adds support for six fully managed open weights models – Amazon Bedrock now supports DeepSeek V3.2, MiniMax M2.1, GLM 4.7, GLM 4.7 Flash, Kimi K2.5, and Qwen3 Coder Next. These models span frontier reasoning and agentic coding workloads. DeepSeek V3.2 and Kimi K2.5 target reasoning and agentic intelligence, GLM 4.7 and MiniMax M2.1 support autonomous coding with large output windows, and Qwen3 Coder Next and GLM 4.7 Flash provide cost-efficient alternatives for production deployment. These models are powered by Project Mantle and provide out-of-the-box compatibility with OpenAI API specifications. With the launch, you can also use new open weight models–DeepSeek v3.2 , MiniMax 2.1, and Qwen3 Coder Next in Kiro, a spec-driven AI development tool.
  • Amazon Bedrock expands support for AWS PrivateLink – Amazon Bedrock now supports AWS PrivateLink for the bedrock-mantle endpoint, in addition to existing support for the bedrock-runtime endpoint. The bedrock-mantle endpoint is powered by Project Mantle, a distributed inference engine for large-scale machine learning model serving on Amazon Bedrock. Project Mantle provides serverless inference with quality of service controls, higher default customer quotas with automated capacity management, and out-of-the-box compatibility with OpenAI API specifications. AWS PrivateLink support for OpenAI API-compatible endpoints is available in 14 AWS Regions. To get started, visit the Amazon Bedrock console or the OpenAI API compatibility documentation.
  • Amazon EKS Auto Mode announces enhanced logging for managed Kubernetes capabilities – You can now configure log delivery sources using Amazon CloudWatch Vended Logs in Amazon EKS Auto Mode. This helps you collect logs from Auto Mode’s managed Kubernetes capabilities for compute autoscaling, block storage, load balancing, and pod networking. Each Auto Mode capability can be configured as a CloudWatch Vended Logs delivery source with built-in AWS authentication and authorization at a reduced price compared to standard CloudWatch Logs. You can deliver logs to CloudWatch Logs, Amazon S3, or Amazon Data Firehose destinations. This feature is available in all Regions where EKS Auto Mode is available.
  • Amazon OpenSearch Serverless now supports Collection Groups – You can use new Collection Groups to share OpenSearch Compute Units (OCUs) across collections with different AWS Key Management Service (AWS KMS) keys. Collection Groups reduce overall OCU costs through a shared compute model while maintaining collection-level security and access controls. They also introduce the ability to specify minimum OCU allocations alongside maximum OCU limits, providing guaranteed baseline capacity at startup for latency-sensitive applications. Collection Groups are available in all Regions where Amazon OpenSearch Serverless is currently available.
  • Amazon RDS now supports backup configuration when restoring snapshots – You can view and modify the backup retention period and preferred backup window before and during snapshot restore operations. Previously, restored database instances and clusters inherited backup parameter values from snapshot metadata and could only be modified after restore was complete. You can now view backup settings as part of automated backups and snapshots, and specify or modify these values when restoring, eliminating the need for post-restoration modifications. This is available for all Amazon RDS database engines (MySQL, PostgreSQL, MariaDB, Oracle, SQL Server, and Db2) and Amazon Aurora (MySQL-Compatible and PostgreSQL-Compatible editions) in all AWS commercial Regions and AWS GovCloud (US) Regions at no additional cost.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

AWS Summits – Join AWS Summits in 2026, free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), and Bengaluru (April 23–24).

AWS AI and Data Conference 2026 – A free, single-day in-person event on March 12 at the Lyrath Convention Centre in Ireland. The conference covers designing, training, and deploying agents with Amazon Bedrock, Amazon SageMaker, and QuickSight, integrating them with AWS data services, and applying governance practices to operate them at scale. The agenda includes strategic guidance and hands-on labs for architects, developers, and business leaders.

AWS Community Days – Community-led conferences where content is planned, sourced, and delivered by community leaders, featuring technical discussions, workshops, and hands-on labs. Upcoming events include Ahmedabad (February 28), Slovakia (March 11), and Pune (March 21).

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— Esra

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

 

Matching your Ingestion Strategy with your OpenSearch Query Patterns

Post Syndicated from Rakan Kandah original https://aws.amazon.com/blogs/big-data/matching-your-ingestion-strategy-with-your-opensearch-query-patterns/

Choosing the right indexing strategy for your Amazon OpenSearch Service clusters helps deliver low-latency, accurate results while maintaining efficiency. If your access patterns require complex queries, it’s best to re-evaluate your indexing strategy.

In this post, we demonstrate how you can create a custom index analyzer in OpenSearch to implement autocomplete functionality efficiently by using the Edge n-gram tokenizer to match prefix queries without using wildcards.

What is an index analyzer?

Index analyzers are used to analyze text fields during ingestion of a document. The analyzer outputs the terms you can use to match queries

By default, OpenSearch indexes your data using the standard index analyzer. The standard index analyzer splits tokens on spaces, converts tokens to lowercase, and removes most punctuation. For some use cases (like log analytics), the standard index analyzer might be all you need.

Standard Index Analyzer

Let’s look at what the standard index analyzer does. We’ll use the _analyze API to test how the standard index analyzer tokenizes the sentence “Standard Index Analyzer.”

Note: You can run all the commands in this post using OpenSearch DevTools in the OpenSearch Dashboard.

GET /_analyze
{
  "analyzer": "standard",
  "text": "Standard Index Analyzer."
}
#========
#Results
#========
{
  "tokens": [
    {
      "token": "standard",
      "start_offset": 0,
      "end_offset": 8,
      "type": "<ALPHANUM>",
      "position": 0
    },
    {
      "token": "index",
      "start_offset": 9,
      "end_offset": 14,
      "type": "<ALPHANUM>",
      "position": 1
    },
    {
      "token": "analyzer",
      "start_offset": 15,
      "end_offset": 23,
      "type": "<ALPHANUM>",
      "position": 2
    }
  ]
}

Notice how each word was lowercased and the period (punctuation) was removed.

Creating your own index analyzer

OpenSearch offers a large number of built in analyzers that you can use for different access patterns. It also lets you build your own custom analyzer, configured for your specific search needs. In the following example, we are going to configure a custom analyzer that returns partial word matches for a list of addresses. The analyzer is specifically designed for autocomplete functionality, enabling end users to quickly find addresses without having to type out (or remember) an entire address. Autocomplete allows OpenSearch to effectively complete the search term based off matched prefixes.

First, create an index called standard_index_test:

PUT standard_index_test
{
  "mappings": {
    "properties": {
      "text_entry": {
        "type": "text",
        "analyzer": "standard"
      }
    }
  }
}

Specifying the analyzer as standard is not required because the standard analyzer is the default analyzer.

To test, bulk add some data to our standard_index_test that we created.

POST _bulk
{"index":{"_index":"standard_index_test"}} 
{"text_entry": "123 Amazon Street Seattle, Wa 12345 "} 
{"index":{"_index":"standard_index_test"}}
{"text_entry": "456 OpenSearch Drive Anytown, Ny 78910"}
{"index":{"_index":"standard_index_test"}}
{"text_entry": "789 Palm way Ocean Ave, Ca 33345"}
{"index":{"_index":"standard_index_test"}}
{"text_entry": "987 Openworld Street, Tx 48981"}

Query this data using the text “ope”.

GET standard_index_test/_search
{
  "query": {
    "match": {
      "text_entry": {
        "query": "ope"
      }
    }
  }
}
#========
#Results
#========
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": n`ull,
    "hits": [] # No matches 
  }
}

When searching for the term “ope”, we don’t get any matches. To see why, we can dive a little deeper into the standard index analyzer and see how our text is being tokenized. Test the standard index analyzer with the address “456 OpenSearch Drive Anytown, Ny 78910”.

POST standard_index_test/_analyze
{
  "analyzer": "standard",
  "text": "456 OpenSearch Drive Anytown, Ny 78910"
}
#========
#Results
#========
  "tokens":
      "456" 
      "opensearch" 
      "drive" 
      "anytown"
      "ny" 
      "78910"

The standard index analyzer has tokenized the address into individual terms: 456, opensearch, drive and so on. That means, unless you search for an individual token (like 456 or opensearch) o, op, ope , and even open won’t yield any results. One option is to use wildcards while still using the standard index analyzer for indexing:

GET standard_index_test/_search
{
  "query": {
    "wildcard": {
      "text_entry": "ope*"
    }
  }
}

The wildcard query would match “456 OpenSearch Drive Anytown, Ny 78910” but wildcard queries can be resource intensive and slow. Querying for ope* in OpenSearch results in iterating over each term in the index, bypassing optimizations of inverted index lookups. This results in higher memory usage and slower performance. To improve the performance of our query execution and search experience, we can use an index analyzer that better suits our access patterns.

Edge n-gram

The Edge n-gram tokenizer helps you find partial matches and avoids the use of wildcards by tokenizing prefixes of a single word. For example, the input word coffee is expanded into all its prefixes, c, co , cof, and so on. It can limit the prefixes to those between a minimum (min_gram) and maximum (max_gram) length. So with min_gram=3 and max_gram=5, it will expand “coffee” to cof, coff, and coffe.

Create a new index called custom_index with our own custom index analyzer that uses Edge n-grams. Set the minimum token length (min_gram) to 3 characters, and the maximum token length (max_gram) to 20 characters. The min_gram and max_gram sets the minimum and maximum returned token length respectively. You should select the min_gram and max_gram based off your access patterns. In this example, we’re searching for the term “ope” so we don’t need to set the minimum length to anything less than 3 since we’re not searching for terms like o or op. Setting the min_gram too low can lead to high latency. Likewise, we don’t need to set the maximum length to anything greater than 20 as no individual token will exceed the length of 20. Setting the maximum length to 20 gives us room to spare in case we do eventually ingest an address with a longer token length. Note, the index we are creating here is specifically for autocomplete functionality and is likely unnecessary for a general search index.

PUT custom_index
{
  "mappings": {
    "properties": {
      "text_entry": {
        "type": "text",
        "analyzer": "autocomplete",         
        "search_analyzer": "standard"       
      }
    }
  },
  "settings": {
    "analysis": {
      "filter": {
        "edge_ngram_filter": {
          "type": "edge_ngram",
          "min_gram": 3,
          "max_gram": 20
        }
      },
      "analyzer": {
        "autocomplete": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "edge_ngram_filter"
          ]
        }
      }
    }
  }
}

In the above code, we created an index called custom_index with a custom analyzer named autocomplete. The analyzer performs the following:

  • It uses the standard tokenizer to split text into tokens
  • A lowercase filter is applied to lowercase all the tokens
  • The tokens are then further broken into smaller chunks based off the minimum and maximum values of the edge_ngram

The search analyzer is configured to use the standard analyzer to reduce query processing required at search time. We have already applied our custom analyzer to split the text for us upon ingestion, and we do not need to repeat this process when searching. Test how the custom analyzer analyzes the text Lexington Avenue:

GET custom_index/_analyze
{
  "analyzer": "autocomplete",
  "text": "Lexington Avenue"
}
#========
#Results
#========
# Minimum token length is 3 so we won't see l, or le
    "tokens": 
        "lex"  
        "lexi"  
        "lexin"  
        "lexing" 
        "lexingt" 
        "lexingto"    
        "lexington" 
        "ave"        
        "aven" 
        "avenu" 
        "avenue"

Notice how the tokens are lowercase and now support partial matches. Now that we’ve seen how our analyzer tokenizes our text, bulk add some data:

POST _bulk
{"index":{"_index":"custom_index"}} 
{"text_entry": "123 Amazon Street Seattle, Wa 12345 "} 
{"index":{"_index":"custom_index"}}
{"text_entry": "456 OpenSearch Drive Anytown, Ny 78910"}
{"index":{"_index":"custom_index"}}
{"text_entry": "789 Palm way Ocean Ave, Ca 33345"}
{"index":{"_index":"custom_index"}}
{"text_entry": "987 Openworld Street, Tx 48981"}

And test!

GET custom_index/_search
{
  "query": {
    "match": {
      "text_entry": {
        "query": "ope" 
      }
    }
  }
}
#========
#Results
#========
 "hits": [
      {
        "_index": "custom_index",
        "_id": "aYCEIJgB4vgFQw3LmByc",
        "_score": 0.9733556,
        "_source": {
          "text_entry": "456 OpenSearch Drive Anytown, Ny 78910"
        }
      },
      {
        "_index": "custom_index",
        "_id": "a4CEIJgB4vgFQw3LmByc",
        "_score": 0.4095239,
        "_source": {
          "text_entry": "987 Openworld Street, Tx 48981"
        }
      }
    ]

You have configured a custom n-gram analyzer to find partial words matches within our list of addresses.

Note, there is a tradeoff between using non-standard index analyzers and writing compute intensive queries. Analyzers can affect indexing throughput and increase the overall index size, especially if used inefficiently. For example, when creating the custom_index, the search analyzer was set to use the standard analyzer. Using n_grams for analysis upon ingestion and search would have impacted cluster performance unnecessarily. Additionally, we set the min_gram and max_gram to values that matched our access patterns, ensuring we didn’t create more n_grams than we needed to for our search use case. This allowed us to gain the benefits of optimizing search without impacting our ingestion throughput.

Conclusion

In this post, we changed how OpenSearch indexed our data to simplify and speed up autocomplete queries. In our case, using the Edge n-grams allowed OpenSearch to match parts of an address and yield precise results without compromising cluster performance with a wildcard query.

It’s always important to test your cluster before deploying in a production environment. Understanding your access patterns is essential to optimizing your cluster from both an indexing and searching perspective. Use the guidelines in this post as a starting point. Confirm your access patterns before creating an index, then begin experimenting with different index analyzers in a test environment to see how they can simplify your queries and improve overall cluster performance. For more reading on general OpenSearch cluster optimization techniques, refer to the Get started with Amazon OpenSearch Service: T-shirt-size your domain post.


About the authors

Rakan Kandah

Rakan Kandah

Rakan is a Solutions Architect at AWS. In his free time, Rakan enjoys playing guitar and reading.