Tag Archives: Amazon OpenSearch Service

How GPU acceleration builds billion-scale vector indexes on Amazon OpenSearch Service

Post Syndicated from Navneet Verma original https://aws.amazon.com/blogs/big-data/how-gpu-acceleration-builds-billion-scale-vector-indexes-on-amazon-opensearch-service/

Modern search demands high-performance vector indexing and scalability to keep pace with the rapid growth of generative AI applications. As datasets grow into the billions, traditional CPU-based indexing often becomes a bottleneck, stalling productivity and innovation velocity.

With GPU-accelerated vector (k-NN) indexing now available on Amazon OpenSearch Service and Amazon OpenSearch Serverless, you can scale to billions of vectors efficiently. Powered by NVIDIA cuVS, an open-source library for GPU-accelerated vector search, this capability offloads compute-intensive vector index building to specialized GPU workers while your existing CPU infrastructure continues serving search. The result is faster, more cost-efficient construction of large-scale vector indexes without sacrificing query performance.

Our earlier post went into those performance and cost benefits in detail. This post goes a level deeper into how the capability works. We walk through the decoupled architecture that makes this possible. We explain how a GPU-built index is converted into one that your CPU data nodes can search with no quality tradeoff. We also show how the approach holds up at scale, with a benchmark of one billion 1024-dimensional vectors. Finally, we share the operational best practices we recommend for running GPU-accelerated index builds in production.

Use cases and benefits

As companies across industries build AI-powered and agentic applications to deliver richer customer experiences, GPU acceleration for vector indexing helps across a range of use cases. A few examples:

  • Adopt new embedding models faster: When an organization upgrades to a newer embedding model, every vector has to be regenerated and reindexed. At hundreds of millions to billions of vectors, a CPU rebuild can take days or weeks. GPU acceleration shortens that rebuild to hours, so you can move to a higher-quality model while significantly reducing the reindexing window and availability risk.
  • Accelerate large-scale reindexing: A global ecommerce application managing billions of product listings, customer reviews, and behavioral signals must rebuild its vector index rapidly as new products and embeddings are added. GPU acceleration completes this within a tight operational window, keeping search relevance current.
  • Absorb bursty or high-sustained writes: A media company covering a major sporting event, such as the World Cup or Olympics, needs to index millions of real-time embeddings simultaneously. These embeddings span match highlights, commentary clips, athlete profiles, and fan-generated content, and millions of viewers search for related content at the same time. GPU workers absorb the indexing burst without competing with CPU nodes serving live search traffic, avoiding the latency spikes that usually accompany heavy writes.
  • Right-size clusters for mixed read/write workloads: A retail system traditionally over-provisions its CPU cluster to handle both peak indexing loads during catalog refreshes and concurrent search traffic, paying for peak capacity around the clock. By offloading indexing to GPUs, the CPU cluster can be right sized for search alone, reducing infrastructure costs without sacrificing performance.
  • Speed up migrations to semantic search or to OpenSearch: Whether you’re converting a text-based corpus to vector embeddings for the first time or migrating an existing vector workload from another database to Amazon OpenSearch Service, GPU-accelerated indexing compresses what would be days of index building into hours, keeping pace with upstream GPU-powered embedding generation and minimizing cutover risk.

When does GPU acceleration activate?

GPU acceleration activates automatically once you opt in. On OpenSearch Service domains, you enable it by turning on the Vector Acceleration option, and from that point on, no changes to code or API flags are needed. On OpenSearch Serverless, GPU index-build acceleration is on by default for NextGen vector search collections. Figure 1 illustrates the index build workflow. OpenSearch automatically routes vector indexing operations to GPU or CPU based on segment size, optimizing performance and falling back to CPU if issues arise.

When OpenSearch flushes or merges a segment, it compares the segment’s vector data size against a configurable window bounded by index.knn.remote_index_build.size.min and index.knn.remote_index_build.size.max. The lower bound defaults to 50 MB. Segments above the lower bound are offloaded to a remote GPU worker, and smaller segments build locally on CPU. The segment vector size is calculated as:

segment_vector_size = num_vectors × dimensions × bytes_per_element

This means two workloads with identical document counts can produce different segment sizes:

Vectors Dimensions Encoding Segment Vector Size
100,000 1536 Float32 ~586 MB
100,000 768 Byte ~74 MB

Both examples exceed the default 50 MB lower bound, so with default settings both segments would be offloaded to a GPU worker.

Index build workflow showing OpenSearch routing each segment to a GPU worker or CPU based on its vector data size

Figure 1: Simplified flow for index builds

Decoupled indexing architecture

An OpenSearch index is internally divided into segments, each containing its own vector graph. This segment-level structure is what makes GPU offload practical. Each segment’s graph can be built independently on a GPU worker without coordinating across the full index. Building on this, the key architectural insight is separating where vectors are indexed from where they are searched. Existing CPU data nodes continue handling ingestion, search, and non-vector workloads. When a segment is ready for vector index construction, the heavy graph-building work is offloaded to dedicated GPU workers, and the finished index is returned to the data node for serving.

The index build workflow

  1. Ingest – Documents with vector fields are ingested into your OpenSearch Service domain or OpenSearch Serverless collection as usual. Vectors accumulate in segments on CPU data nodes.
  2. Offload – When a segment flushes or merges and its vector data falls within the GPU activation window, the data node uploads the raw vectors to Amazon Simple Storage Service (Amazon S3) and submits a build request.
  3. Build – A GPU worker from a managed warm pool picks up the job, loads the vectors, and builds the index using CAGRA (CUDA ANN Graph), the GPU-native graph algorithm in NVIDIA cuVS. The resulting CAGRA graph is then converted to a Hierarchical Navigable Small World (HNSW) graph compatible with CPU-based search.
  4. Return – The completed HNSW index is written back to Amazon S3 and downloaded by the data node, which then uses it to serve search queries.

Fully managed GPU index builds

Enable Vector Acceleration, and Amazon OpenSearch Service handles the rest:

Automatic scaling – GPU workers scale up and down automatically based on the number of pending build jobs. During a bulk ingest or reindex, more GPU workers spin up to handle the load. When the queue drains, they scale back to zero.

Automatic instance selection – The service selects the right GPU instance type for each build job based on segment size. No capacity planning or instance selection is required on your part.

Pay only for active builds – You’re charged only when GPUs are actively building indexes, not while they are idle. Even if Vector Acceleration is enabled on your domain or collection, GPU charges, measured in OpenSearch Compute Units (OCUs), apply only when segments reach the activation threshold and trigger an index build. There is no standing GPU infrastructure cost.

Your cost therefore scales directly with indexing activity. Bursty reindexing workloads consume GPU capacity for the duration of the build, and GPU cost returns to zero until the next build.

Figure 2 illustrates the decoupled GPU workflow. Amazon S3 acts as the intermediary between data nodes and GPU workers, allowing them to operate independently. Data nodes upload raw vectors to Amazon S3, GPU workers build the CAGRA graph and convert it to HNSW, and the completed index is returned to the data nodes for serving, with search running uninterrupted throughout.

Decoupled GPU workflow with Amazon S3 as the intermediary between CPU data nodes and GPU workers that build and convert the index

Figure 2: GPU index flow architecture

Inside the CAGRA-to-HNSW conversion

In the previous section, we described how GPU workers build the vector index and return it to data nodes. But how does a GPU-built graph become searchable on CPU, and does this conversion sacrifice quality? The short answer: it doesn’t.

The CAGRA algorithm

The GPU workers use the CAGRA algorithm integrated through the cuVS GPU backend of the Facebook AI Similarity Search (Faiss) library. CAGRA is a graph-based indexing approach built from the ground up for GPU acceleration. It first builds a k-NN graph using another approximate nearest neighbors method like Inverted File with Product Quantization (IVF-PQ) or Nearest Neighbor Descent (NN-Descent). It then removes redundant paths between neighbors to form a navigable search graph.

Construction flow of the CAGRA graph, from an initial k-NN graph to a pruned, navigable search graph

Figure 3: Construction flow of the CAGRA graph

Source: CAGRA: Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs

How the GPU worker builds the index

When the GPU workers receive a vector index build request, it carries the necessary parameters for constructing the segment-specific vector index. The vector index build component initiates the process by retrieving the vector file from Amazon S3 and loading it into CPU memory. These vectors are then used to construct a CAGRA index using Faiss. After constructing the CAGRA index on GPU, the system converts it into an HNSW graph format for compatibility with CPU-based search operations. The resulting index is uploaded to Amazon S3, completing the build request.

Converting the CAGRA graph to HNSW

A typical HNSW index is a multilayered hierarchical graph. The bottom layer (layer 0) of the graph contains the vectors, and the upper layers are sparse subsets used solely for navigation. They help the search algorithm find a good entry point into the bottom layer. However, our HNSW implementation uses the CAGRA graph as the bottom layer and, similar to the CAGRA search method, starts with random entry points into the graph, avoiding the need for the upper layers altogether.

This means the GPU handles the heavy lifting of building the base-layer graph. Reusing that graph as the HNSW base layer avoids rebuilding it on the CPU, which keeps conversion overhead low. As Figure 4 shows, the CAGRA graph becomes the base layer. At query time, the search selects a random set of nodes in the graph and traverses it by following the nearest neighbor links. This is known as greedy search.

Searching an HNSW-converted CAGRA graph by starting at random entry nodes and following nearest neighbor links

Figure 4: Searching an HNSW-converted CAGRA graph

Same recall, faster build

Previous benchmarks have confirmed that GPU-built indexes achieve the same recall as CPU-built HNSW with no quality tradeoff. This is because the bottom-layer graph structure produced by CAGRA is equivalent in connectivity and search quality to what HNSW constructs on CPU. Only the build method differs.

Scaling beyond GPU memory

Out-of-core construction

Traditional GPU indexing requires the entire dataset to reside in GPU memory, creating a hard ceiling on index size based on available hardware. CAGRA removes this limitation through out-of-core k-NN graph construction. When IVF-PQ is used to build the initial k-NN graph for CAGRA, data is streamed from system memory to the GPU in batches, so the full dataset never needs to fit in GPU memory at once. Meanwhile, the GPU still handles the computationally intensive distance calculations and graph optimization.

Quantization

GPU-accelerated indexing supports the quantization levels available in OpenSearch, including 2×, 8×, 16×, and 32× compression. Quantization is applied before vectors are sent to the GPU. This reduces both the data transfer size to GPU workers and the memory footprint during graph construction. This means that you can build indexes over larger segments, improving cost efficiency.

Indexing one billion 1024-dimensional vectors on the GPU

Dataset setup

To evaluate a realistic large-scale workload, we used a dataset containing one billion vectors in 1024 dimensions. Because uniformly random vectors yield misleading results for both index construction and recall, we required data that maintained the structure of real-world embeddings. We created this dataset using the cuVS synthetic dataset generator in cuvs-bench, which outputs synthetic data whose distribution mimics an actual embedding dataset derived from Common Crawl. You can use this approach to build a realistic dataset without exposing or distributing sensitive original data. The generator can produce the complete one-billion-vector dataset, 10,000 query vectors, and the associated ground-truth labels in roughly two hours on a single Amazon Elastic Compute Cloud (Amazon EC2) g6e.16xlarge instance.

Cluster configuration

We designed the benchmark cluster on OpenSearch Service following OpenSearch vector search performance tuning best practices and conducted the benchmark using the OpenSearch Benchmark framework.

Setting Value Rationale
Data Nodes 24 × r8g.4xlarge Memory-optimized instances for large vector indexes
Primary shards 48 Keeps shard size manageable and maximizes parallelism
Replicas 0 Maximizes indexing throughput. Replicas added after build
GPU workers 10 (pre-scaled) Avoids cold-start effects during measurement
Bulk clients 160 Saturates ingestion pipeline across 24 nodes
Bulk size 500 docs/request Balances per-request overhead vs. memory pressure
Refresh interval -1 (during ingest) Prevents small segment creation. Force merge after ingesting
Merge autothrottle Disabled Avoids artificial bottleneck during benchmark

Key best practices applied

  1. Memory-optimized instances – r8g.4xlarge provides sufficient heap and native memory for loading HNSW graphs post-build.
  2. Disabled refresh during bulk ingest – Prevents creation of many small segments that would each trigger individual GPU builds.
  3. High number of bulk clients – Saturates ingestion across nodes and makes sure that GPUs are busy building the indexes.

We used the default HNSW build and search settings in OpenSearch (such as m and ef_construction) since the defaults are what most users start with, and they keep the benchmark representative.

Benchmark results

Dataset Index (min) Recall @k=100 Recall @1 P50 (search) P90 (search) P99 (search) Vector Acceleration OCU Used
1024D 1B 274 0.93 0.93 26.47ms 32.5ms 66.6ms 44

Build time scales linearly with data volume

Our earlier benchmark on OpenSearch Service indexed one billion 128-dimensional vectors (BigANN SIFT dataset) in approximately 35.5 minutes. In our latest benchmark, we scaled dimensionality 8x to 1024 dimensions and completed the index build in 274 minutes, roughly proportional to the increase in data volume. This demonstrates that GPU acceleration maintains consistent throughput efficiency as dimensionality grows: build time scales with data volume rather than fixed startup costs, so you can predictably estimate index build time from your dataset size. Search latency also stayed low at this scale, so the resulting index supported responsive queries without trading away build speed.

Optimizing bulk ingestion for GPU-accelerated indexing

When loading large volumes of vector data, temporarily adjusting index behavior can significantly reduce GPU processing overhead. This approach works if your use case can tolerate a brief period of data staleness. During full index builds, this is generally acceptable, because newly ingested vectors are not searchable until you re-enable refresh. By disabling refresh during bulk ingestion ("index.refresh_interval": "-1"), you prevent the continuous creation of small segments. Each of these would otherwise trigger an individual GPU build job. After ingestion is complete, we enable the refresh interval and complete the refresh to make the segment searchable. This means the GPU builds the vector index once across large, well-packed segments rather than repeatedly across many small ones, resulting in faster overall indexing throughput.

After enabling GPU acceleration, you can monitor builds through Amazon CloudWatch metrics (cluster-level) and the OpenSearch k-NN Stats API (per-node). If a GPU build fails, the system automatically falls back to CPU-based index building, so your data remains indexed.

Future optimization

Today, the completed HNSW index (graph structure and vectors) is transferred back from GPU workers to data nodes through Amazon S3. Because data nodes already hold the raw vectors locally, a future optimization will transfer only the graph structure (neighbor lists). This significantly reduces the data written back to Amazon S3 and the download time to data nodes.

Conclusion

GPU-accelerated indexing lets you build billion-scale vector indexes on Amazon OpenSearch Service in hours instead of days, without changing how queries are served on both OpenSearch Service domains and OpenSearch Serverless collections. In this post, we showed how OpenSearch Service offloads eligible index builds to GPU workers, builds a CAGRA graph through the NVIDIA cuVS backend in Faiss, and converts it into a CPU-searchable HNSW index. We then demonstrated the approach at scale on one billion 1024-dimensional vectors, and shared best practices for optimizing bulk ingestion and monitoring build activity and OCU usage.

Get started

Ready to try GPU-accelerated vector indexing? In a supported AWS Region, you can enable GPU acceleration when you create or update an OpenSearch Service domain running OpenSearch 3.1 or later. Use the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDK. For a new OpenSearch Serverless deployment, create a NextGen vector search collection, where GPU index-build acceleration is enabled by default and can be controlled for individual indexes. For a Classic vector collection, enable GPU acceleration at the collection level.

Acknowledgments

The authors would like to thank Ben Gardner, Manas Singh, Zack Meeks, Jiahong Liu, James Yi, Jinsol Park from NVIDIA for their contributions to this post.


About the authors

Navneet Verma

Navneet is a Principal Software Engineer at AWS working on core Vector Search in OpenSearch. He is passionate about scale, performance, and advancing the state of vector search for large-scale AI workloads.

Vamshi Vijay Nakkirtha

Vamshi Vijay Nakkirtha

Vamshi is a software engineering manager working on the OpenSearch Project and Amazon OpenSearch Service. His interests center on distributed systems.

Gowri Balasubramanian

Gowri Balasubramanian

Gowri is a Senior Manager leading the Data Specialist Solutions Architect team at Amazon Web Services. He drives customer adoption of AWS database and analytics services and develops prescriptive guidance, from reference architectures to best practices, to help enterprises accelerate their data and AI transformation journeys. He is passionate about scalable and distributed data systems.

Kshitiz Gupta

Kshitiz Gupta

Kshitiz is a Senior Solutions Architect at NVIDIA, where he helps cloud customers optimize large-scale AI workloads on GPUs. His work spans GPU-accelerated data processing, vector search, and LLM inference partnering closely with AWS and Amazon teams to bring these capabilities into production. Outside of work, he enjoys music, yoga, and hiking.

Corey Nolet

Corey Nolet

Corey is a distinguished engineer for vector search, data mining, and classical ML libraries at NVIDIA, where he focuses on building and scaling algorithms to support extreme data loads at light speed. Prior to joining NVIDIA in 2018, Corey spent many years building massive-scale exploratory data science & real-time analytics platforms for big data and HPC environments in the defense industry. Corey holds a PhD in Computer Science and has a passion for using data to make better sense of the world.

Rajeshwari Devaramani

Rajeshwari is a solutions architect at NVIDIA. Rajeshwari holds a master’s degree in computational science and engineering from the Georgia Institute of Technology. Her background includes GPU programming, high-performance computing, and deep learning.

Centralized CloudTrail monitoring across 100+ AWS accounts

Post Syndicated from Jagdish Komakula original https://aws.amazon.com/blogs/big-data/centralized-cloudtrail-monitoring-across-100-aws-accounts/

Organizations running workloads across dozens or hundreds of AWS accounts face a common challenge: centralized security monitoring at scale. Security teams need to search through hundreds of gigabytes of AWS CloudTrail logs daily to detect threats and satisfy compliance requirements for SOC 2, PCI DSS, and HIPAA audits. They also need to provide role-based access to multiple teams with different responsibilities.

Without purpose-built infrastructure, this often involves manual log searching that takes hours and compliance report generation that takes days. It also leads to fragmented code bases of custom AWS Lambda functions managing index lifecycles across environments. A single shared search domain without consistent access control compounds the problem further.

In this post, we show you how to build a centralized CloudTrail monitoring solution on Amazon OpenSearch Service. Terraform manages the full stack, from domain provisioning to access control and lifecycle policies. The solution handles 200 GB/day of CloudTrail logs, provides automated threat detection alerts, and gives 4 different teams isolated, role-appropriate access to the data.

Solution overview

The following diagram shows the architecture. CloudTrail logs flow from 100+ AWS accounts through an organization trail into a centralized S3 bucket. Amazon Simple Queue Service (Amazon SQS) notifications trigger the OpenSearch Ingestion pipeline. The pipeline auto-scales between 2 and 10 OpenSearch Compute Units (OCUs) to parse and index the logs into the OpenSearch domain. Four team-specific roles access the data through OpenSearch Dashboards with tenant isolation.

CloudTrail logs flow from 100+ accounts into S3, then through Amazon SQS and OpenSearch Ingestion into the OpenSearch domain used by 4 team roles

The key components are:

  • CloudTrail aggregation. An organization trail sends logs from 100+ accounts into a centralized Amazon Simple Storage Service (Amazon S3) bucket.
  • Ingestion. An Amazon OpenSearch Ingestion pipeline picks up new logs through Amazon SQS notifications on the S3 bucket. It automatically scales between 2 and 10 OCUs based on queue depth. Throttling on lower-environment queues prevents development and test spikes from starving production ingestion.
  • Amazon OpenSearch Service domain. 6 or1.4xlarge data nodes (OpenSearch Optimized instances) with 3 dedicated r8g.large master nodes, fine-grained access control, encryption at rest, and node-to-node encryption.
  • Infrastructure as code. Index templates, Index State Management (ISM) policies, roles, role mappings, tenants, alerting monitors, and dashboards are all declared in Terraform and applied consistently across environments.

Prerequisites

To implement this solution, you need the following:

  • An organization in AWS Organizations with CloudTrail enabled across member accounts.
  • Terraform v1.5+ with the AWS provider and the OpenSearch provider.
  • A virtual private cloud (VPC) with private subnets for the OpenSearch domain.
  • IAM roles for each team that will access the OpenSearch domain.
  • An Amazon Simple Notification Service (Amazon SNS) topic for security alert notifications.
  • Familiarity with Amazon OpenSearch Service, Terraform, and AWS CloudTrail.
  • Sample Terraform code is available in the GitHub repository

Implementation

This section walks through the Terraform code for each component of the solution, starting with the workload profile that informed our sizing decisions.

Workload profile

Before sizing the cluster, we defined the workload characteristics and SLAs for the centralized CloudTrail monitoring platform:

Metric Value
Index throughput 200 GB/day (~18,000 docs/sec)
Search queries ~2,000 queries/day (~0.023 QPS)
Average search latency < 100 ms (achieved: 76 ms)
Saved searches 600+
Dashboards and visualizations 100+
User teams 4 (Security Ops, Incident Response, Compliance, DevOps)
Retention 30 days (hot tier)
Availability target 99.9%

This is a write-heavy ingestion workload. The primary use case is automated alerting and periodic compliance queries rather than continuous interactive search. This workload profile informed the decision to use OR1 (storage-optimized) instances with zero replicas, prioritizing indexing throughput over search parallelism.

Domain provisioning

Start by provisioning the Amazon OpenSearch Service domain with encryption, fine-grained access control, and VPC placement:

resource "aws_opensearch_domain" "cloudtrail" {
  domain_name    = var.domain_name
  engine_version = "OpenSearch_3.3"

  cluster_config {
    instance_type          = "or1.4xlarge.search"
    instance_count         = 6
    zone_awareness_enabled = true
    zone_awareness_config {
      availability_zone_count = 3
    }
  }

  dedicated_master_config {
    dedicated_master_enabled = true
    dedicated_master_type    = "r6g.large.search"
    dedicated_master_count   = 3
  }

  ebs_options {
    ebs_enabled = true
    volume_type = "gp3"
    volume_size = 500
    iops        = 3000
    throughput  = 125
  }

  encrypt_at_rest { enabled = true }
  node_to_node_encryption { enabled = true }

  domain_endpoint_options {
    enforce_https       = true
    tls_security_policy = "Policy-Min-TLS-1-2-PFS-2023-10"
  }

  advanced_security_options {
    enabled                        = true
    internal_user_database_enabled = false
    master_user_options {
      master_user_arn = var.master_user_arn
    }
  }

  vpc_options {
    subnet_ids         = var.vpc_subnet_ids
    security_group_ids = var.vpc_security_group_ids
  }

  tags = {
    Environment = "production"
    Project     = "centralized-cloudtrail-monitoring"
    ManagedBy   = "terraform"
  }
}

This solution was built on OR1 instances, which are storage-optimized and use Amazon Elastic Block Store (Amazon EBS) (gp3 or io1) for local storage, with data copied synchronously to Amazon S3 as it arrives. This storage structure provides increased indexing throughput because indexing is performed exclusively on primary shards. Replicas are backed by Amazon S3 through segment replication, eliminating the CPU overhead of document replication on replica nodes. For new deployments, we recommend OR2 instances, which offer up to 26% higher indexing throughput compared to OR1 while maintaining the same storage-optimized architecture.

Ingestion pipeline

The Amazon OpenSearch Ingestion pipeline provides serverless, auto scaling ingestion from Amazon S3 into the OpenSearch domain. It picks up new CloudTrail logs through Amazon SQS notifications on the centralized S3 bucket and scales between 2 and 10 OpenSearch Compute Units (OCUs) based on queue depth:

resource "aws_iam_role" "osis_pipeline" {
  name = "cloudtrail-osis-pipeline-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action    = "sts:AssumeRole"
      Effect    = "Allow"
      Principal = { Service = "osis-pipelines.amazonaws.com" }
    }]
  })
}

resource "aws_iam_policy" "osis_pipeline" {
  name = "cloudtrail-osis-pipeline-policy"
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action   = ["s3:GetObject", "s3:ListBucket"]
        Effect   = "Allow"
        Resource = [var.cloudtrail_bucket_arn, "${var.cloudtrail_bucket_arn}/*"]
      },
      {
        Action   = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
        Effect   = "Allow"
        Resource = var.cloudtrail_sqs_queue_arn
      },
      {
        Action   = ["es:DescribeDomain", "es:ESHttp*"]
        Effect   = "Allow"
        Resource = "${aws_opensearch_domain.cloudtrail.arn}/*"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "osis_pipeline" {
  role       = aws_iam_role.osis_pipeline.name
  policy_arn = aws_iam_policy.osis_pipeline.arn
}

resource "aws_cloudwatch_log_group" "osis_pipeline" {
  name              = "/aws/vendedlogs/OpenSearchIngestion/cloudtrail-pipeline"
  retention_in_days = 30
}

resource "aws_osis_pipeline" "cloudtrail" {
  pipeline_name = "cloudtrail-ingestion"
  pipeline_configuration_body = <<-EOT
    version: "2"
    cloudtrail-pipeline:
      source:
        s3:
          notification_type: "sqs"
          codec:
            json:
          compression: "gzip"
          sqs:
            queue_url: "${var.cloudtrail_sqs_queue_url}"
          aws:
            sts_role_arn: "${aws_iam_role.osis_pipeline.arn}"
            region: "${data.aws_region.current.name}"
      processor:
        - date:
            from_time_received: true
            destination: "@timestamp"
      sink:
        - opensearch:
            hosts: ["https://${aws_opensearch_domain.cloudtrail.endpoint}"]
            index: "cloudtrail-%{yyyy.MM.dd}"
            aws:
              sts_role_arn: "${aws_iam_role.osis_pipeline.arn}"
              region: "${data.aws_region.current.name}"
  EOT
  min_units = 2
  max_units = 10
  log_publishing_options {
    is_logging_enabled = true
    cloudwatch_log_destination {
      log_group = aws_cloudwatch_log_group.osis_pipeline.name
    }
  }
  tags = {
    Environment = "production"
    Project     = "centralized-cloudtrail-monitoring"
    ManagedBy   = "terraform"
  }
}

The pipeline uses the S3 source plugin with SQS-based notifications. When new CloudTrail log files land in S3, an SQS message triggers the pipeline to fetch and parse them. The min_units and max_units parameters control auto scaling. The pipeline starts at 2 OCUs and scales up to 10 based on queue depth, handling ingestion spikes without manual intervention. For lower environments (development and testing), you can apply throttling on the Amazon SQS queue to prevent non-production spikes from starving production ingestion capacity.

Index template

Define index templates up front to avoid painful reindexing later. The following template sets explicit mappings for CloudTrail fields, optimizes for write throughput with async translog durability, and integrates with ISM for automatic rollover:

resource "opensearch_index_template" "cloudtrail" {
  name = "cloudtrail-template"
  body = jsonencode({
    index_patterns = ["cloudtrail-*"]
    priority       = 100
    template = {
      settings = {
        number_of_shards                                  = 6
        number_of_replicas                                = 0
        "index.refresh_interval"                          = "10s"
        "index.translog.durability"                       = "async"
        "index.translog.sync_interval"                    = "30s"
        "plugins.index_state_management.rollover_alias"   = "cloudtrail"
      }
      mappings = {
        properties = {
          "@timestamp"        = { type = "date" }
          eventSource         = { type = "keyword" }
          eventName           = { type = "keyword" }
          awsRegion           = { type = "keyword" }
          sourceIPAddress     = { type = "ip" }
          errorCode           = { type = "keyword" }
          errorMessage        = { type = "text" }
          recipientAccountId  = { type = "keyword" }
          userIdentity = {
            properties = {
              type      = { type = "keyword" }
              arn       = { type = "keyword" }
              accountId = { type = "keyword" }
              userName  = { type = "keyword" }
              sessionContext = {
                properties = {
                  sessionIssuer = {
                    properties = {
                      type     = { type = "keyword" }
                      arn      = { type = "keyword" }
                      userName = { type = "keyword" }
                    }
                  }
                }
              }
            }
          }
          requestParameters = { type = "object", enabled = true }
          responseElements  = { type = "object", enabled = true }
        }
      }
    }
  })
}

Defining mappings before ingestion prevents mapping conflicts and avoids the need to reindex data after the fact.

Lifecycle management (ISM policy)

The following ISM policy replaces custom Lambda functions with a single declarative policy. The rollover action uses two OR conditions: min_index_age and min_primary_shard_size. Whichever threshold is reached first triggers the rollover. This keeps shard sizes bounded while ensuring timely rotation even during low-volume periods:

resource "opensearch_ism_policy" "cloudtrail_lifecycle" {
  policy_id = "cloudtrail-lifecycle"
  body = jsonencode({
    policy = {
      description   = "CloudTrail lifecycle - rollover, retain 30d, delete"
      default_state = "hot"
      ism_template  = [{ index_patterns = ["cloudtrail-*"], priority = 100 }]
      states = [
        {
          name    = "hot"
          actions = [{ rollover = { min_primary_shard_size = "30gb", min_index_age = "1d" } }]
          transitions = [{ state_name = "delete", conditions = { min_index_age = "30d" } }]
        },
        {
          name        = "delete"
          actions     = [{ delete = {} }]
          transitions = []
        }
      ]
    }
  })
}

This approach reduces lifecycle management code by approximately 60% compared to per-environment Lambda functions, and changes deploy in a single terraform apply.

Note: For indexes ingesting more than 100 GB/day (such as CloudTrail at 200 GB/day in this deployment), you can override min_index_age to 12h to roll over more frequently. The two conditions are OR-based in OpenSearch ISM. If a shard reaches 30 GB before 1 day, it rolls over on size. If 1 day passes before 30 GB, it rolls over on age.

Multi-team access control

When multiple teams need different access levels to the same data, define all roles declaratively and use for_each to create them consistently. The following example defines 4 team roles with varying permissions:

locals {
  team_roles = {
    security_ops = {
      description         = "Security Operations - full read, alert management"
      cluster_permissions = ["cluster_monitor", "cluster:admin/opendistro/alerting/*"]
      index_permissions = [
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search", "get"] },
        { index_patterns = [".opendistro-alerting-*"], allowed_actions = ["read", "write", "search", "get", "delete"] }
      ]
    }
    incident_response = {
      description         = "Incident Response - full read for investigation"
      cluster_permissions = ["cluster_monitor"]
      index_permissions = [
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search", "get"] }
      ]
    }
    compliance_auditors = {
      description         = "Compliance - read-only"
      cluster_permissions = []
      index_permissions = [
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search"] }
      ]
    }
    devops = {
      description         = "DevOps - infra metrics and limited CloudTrail"
      cluster_permissions = ["cluster_monitor"]
      index_permissions = [
        { index_patterns = ["infra-metrics-*"], allowed_actions = ["read", "search", "get"] },
        { index_patterns = ["cloudtrail-*"], allowed_actions = ["read", "search"] }
      ]
    }
  }
}

resource "opensearch_role" "teams" {
  for_each            = local.team_roles
  role_name           = each.key
  description         = each.value.description
  cluster_permissions = each.value.cluster_permissions

  dynamic "index_permissions" {
    for_each = each.value.index_permissions
    content {
      index_patterns  = index_permissions.value.index_patterns
      allowed_actions = index_permissions.value.allowed_actions
    }
  }

  dynamic "tenant_permissions" {
    for_each = [each.key]
    content {
      tenant_patterns = [each.key]
      allowed_actions = ["kibana_all_write"]
    }
  }
}

resource "opensearch_roles_mapping" "teams" {
  for_each      = local.team_roles
  role_name     = opensearch_role.teams[each.key].role_name
  backend_roles = var.team_iam_roles[each.key]
}

resource "opensearch_tenant" "teams" {
  for_each    = local.team_roles
  tenant_name = each.key
  description = "Dashboard workspace for ${replace(each.key, "_", " ")}"
}

This approach maps IAM roles (not individual users) to OpenSearch roles. Adding a new team means adding one entry to the locals block and running terraform apply. Each team gets an isolated tenant in OpenSearch Dashboards, preventing cross-team interference with saved searches, visualizations, and dashboard configurations. We chose OpenSearch Dashboards because tenants, roles, visualizations, and saved objects can all be managed programmatically through the Terraform OpenSearch provider, keeping the entire stack under infrastructure-as-code governance. For teams building new visualizations outside of Terraform-managed workflows, we recommend OpenSearch UI. This next-generation analytics interface supports multiple data sources, provides workspaces for team isolation, and remains available during cluster upgrades.

Alerting

Define alerting monitors in Terraform to detect security-critical events automatically. The following monitor catches CloudTrail tampering attempts (StopLogging, DeleteTrail) and sends alerts through Amazon SNS:

resource "opensearch_monitor" "cloudtrail_tampering" {
  body = jsonencode({
    name     = "CloudTrail Tampering Detection"
    type     = "monitor"
    enabled  = true
    schedule = { period = { interval = 1, unit = "MINUTES" } }
    inputs = [{
      search = {
        indices = ["cloudtrail-*"]
        query = {
          size = 5
          query = {
            bool = {
              must = [{ terms = { eventName = ["StopLogging", "DeleteTrail",
                "UpdateTrail", "PutEventSelectors", "DeleteEventDataStore"] } }]
              filter = [{ range = { "@timestamp" = { gte = "now-1m" } } }]
            }
          }
        }
      }
    }]
    triggers = [{
      name     = "trail_tampering_detected"
      severity = "1"
      condition = { script = {
        source = "ctx.results[0].hits.total.value > 0"
        lang   = "painless"
      } }
      actions = [{
        name             = "notify_security"
        destination_id   = var.sns_destination_id
        message_template = { source = "CRITICAL: CloudTrail tampering detected." }
      }]
    }]
  })
}

Results and performance

After deploying the solution, we measured steady-state performance against the SLAs defined in the workload profile:

Metric Target Achieved
Index throughput 200 GB/day 200 GB/day sustained (~18,000 docs/sec)
Search latency (avg) < 100 ms 76 ms
Search availability 99.9% 99.95%+ (no unplanned downtime in 145 days)
Alert detection time < 2 minutes ~1 minute (monitor interval)
Compliance report generation < 5 minutes On-demand via saved searches

Key outcomes:

  • Threat detection dropped from hours to minutes. Automated alerting replaced manual log searching. The CloudTrail tampering monitor detects suspicious activity within one minute of the event.
  • Compliance reports generate on demand. With 600+ saved searches and 100+ dashboards, compliance teams produce SOC 2, PCI DSS, and HIPAA audit evidence in minutes rather than days.
  • Four teams operate independently. Each team has its own isolated tenant in OpenSearch Dashboards, preventing cross-team interference with saved searches and dashboard configurations.
  • Zero custom Lambda functions. ISM policies, index templates, and access control are all managed declaratively through Terraform, eliminating the previous fragmented code base.

Best practices

  • Define index templates before ingesting anything. Changing mappings on existing indices means reindexing. Get this right first.
  • Set rollover thresholds based on your actual ingestion rate. At 200 GB/day, rolling over at 30 GB keeps shard counts manageable while balancing query performance.
  • Test ISM transitions in a lower environment first. Warm and cold migrations on large indices take time.
  • Map IAM roles, not users. People change teams. Roles stay stable. This simplifies access management.
  • Put an Amazon SQS queue between S3 and the ingestion pipeline. This gives you per-environment throttling control without modifying pipeline configuration.
  • Use for_each aggressively. Roles, tenants, index patterns, and monitors all follow a pattern across teams or environments, so use for_each to eliminate copy-paste drift.
  • Consider OpenSearch UI for new visualization workflows. This solution uses OpenSearch Dashboards for Terraform-managed tenants and roles. OpenSearch UI is a next-generation interface that supports multiple data sources, stays available during cluster upgrades, and includes workspaces for team isolation. It is the recommended interface for creating new dashboards and visualizations going forward.

Optional: Extending with cold storage for longer retention

For organizations with compliance requirements mandating longer retention (for example, 7 years for PCI DSS or HIPAA), you can extend the ISM policy with warm and cold tiers. The following example adds tiered storage that moves data through hot, warm, cold, and delete states:

states = [
  {
    name    = "hot"
    actions = [{ rollover = { min_primary_shard_size = "30gb", min_index_age = "1d" } }]
    transitions = [{ state_name = "warm", conditions = { min_index_age = "30d" } }]
  },
  {
    name = "warm"
    actions = [
      { warm_migration = {} },
      { force_merge = { max_num_segments = 1 } }
    ]
    transitions = [{ state_name = "cold", conditions = { min_index_age = "365d" } }]
  },
  {
    name    = "cold"
    actions = [{ cold_migration = { timestamp_field = "@timestamp" } }]
    transitions = [{ state_name = "delete", conditions = { min_index_age = "2555d" } }]
  },
  {
    name        = "delete"
    actions     = [{ cold_delete = {} }]
    transitions = []
  }
]

Warm storage uses force-merge to reduce segment count (lowering query overhead), while cold storage moves data entirely to Amazon S3 for minimal cost. This tiered approach keeps hot-tier performance high while meeting long-term audit requirements.

Cleanup

To avoid incurring ongoing charges, remove the resources created in this post by running:

terraform destroy

This removes the OpenSearch domain, ingestion pipeline, IAM roles, SQS queues, and all associated configurations. Verify that you have exported any data or dashboards you want to retain before running destroy.

Conclusion

In this post, we showed you how to build a centralized CloudTrail monitoring solution on Amazon OpenSearch Service with Terraform managing the entire stack. The approach moves from fragile, manually configured systems with redundant Lambda code to a version-controlled, peer-reviewed, consistently deployed infrastructure.

Threat detection drops from hours to minutes with automated alerting. Compliance reports that took days now generate on demand. And your team spends time on security analysis instead of infrastructure maintenance.

To get started, use the AWS Terraform provider aws_opensearch_domain resource for the domain, then use the Terraform OpenSearch provider for index templates, ISM policies, roles, and monitors. Configure your ingestion pipeline to transform and enrich incoming CloudTrail logs before indexing, building a modern, scalable security foundation that grows with your organization.

The complete source code for this solution is available in the GitHub repository: GitHub repository

For more on the services used in this solution:


About the authors

Jagdish Komakula

Jagdish Komakula

Jagdish is a Senior Delivery Consultant at AWS Professional Services, focused on Amazon OpenSearch Service and Infrastructure automation. He has spent the last several years guiding financial services customers through building data platforms that scale.

Aditya Ambati

Aditya Ambati

Aditya is a Delivery Consultant at AWS Professional Services, focused on DevOps and infrastructure as code. He works with customers on automating cloud operations and implementing GitOps practices.

How a team at Epic Games tuned Amazon OpenSearch Service for Fortnite analytics

Post Syndicated from Jon Evans original https://aws.amazon.com/blogs/big-data/how-a-team-at-epic-games-tuned-amazon-opensearch-service-for-fortnite-analytics/

Since the launch of Fortnite in 2017, Epic Games has reached hundreds of millions of players worldwide. Fortnite runs on Amazon Web Services (AWS), and takes advantage of services such as Amazon OpenSearch Service to power certain internal analytics and drive decision making at scale.

Amazon OpenSearch Service has been helpful in understanding the game ecosystem. OpenSearch Service powers two types of use cases: search workloads and analytics workloads. A team at Epic Games had a use case for storing and analyzing a sliding window of game event data. This involves supporting complex queries and multilayered aggregations that feed analytical results into other internal systems, helping them power an evolving player experience. At the scale of a game like Fortnite with a large player base, these queries run against a significant volume of incoming data.

These insights help identify emerging gameplay trends, understand how players engage with new content, and reveal more about the Fortnite ecosystem. They inform live operation decisions and help surface relevant content to players based on aggregated activity across the community.

As Epic Games’ infrastructure handles billions of telemetry events, the team identified opportunities to optimize their OpenSearch Service cluster for better performance and cost efficiency. This post details how Epic Games partnered with AWS to transform their OpenSearch Service deployment, achieving significant improvements in query latency and resource utilization while reducing operational costs.

The challenge

Epic Games runs an OpenSearch Service domain that handles continuous high-volume writes alongside CPU-intensive batch aggregation jobs. Ideally, these aggregation jobs would run more frequently to keep analytics fresh. Shorter job intervals mean fresher data for identifying gameplay trends, detecting anomalies, and informing live operations decisions. But the existing configuration couldn’t support this without scaling the domain beyond what the workload justified, driving up costs. Epic Games worked with AWS to identify where improvements could be made, focusing on areas such as hardware utilization, sharding strategy, index mappings, and query behavior.

Observations

The cluster was running on r7g memory-optimized data nodes, with 48 vCPUs and 384 GiB of memory per node. Of each node’s available memory, only a fraction (32 GiB) was allocated to Java Virtual Machine (JVM) heap, set at the maximum recommended for compressed oops. The remainder (off-heap memory) was used for the filesystem cache and the operating system. System memory was not fully utilized across the data nodes (Figure 1).

Figure 1: System memory utilization across data nodes

As shown in the preceding figure, utilization stays well below 100% throughout the observation period, confirming that much of the off-heap memory allocated to these nodes goes unused. The excess capacity could be safely exchanged for additional compute resources.

JVM memory pressure is shown in Figure 2, and the correlating garbage collection metrics (both count and time) are shown in Figure 3.

Figure 2: JVM memory pressure

Figure 3: JVM garbage collection metrics, count (top) and time (bottom)

These charts show that JVM memory pressure remains below critical thresholds, and both garbage collection count and time are low and stable, indicating healthy JVM utilization across the domain.

While cluster-level CPU metrics appeared healthy at first glance (Figure 4), zooming into node-level metrics revealed clear node hotspots. The root cause of the node hotspots was the cluster’s sharding strategy.

Figure 4: Cluster-level CPU utilization

The cluster had data nodes distributed across multiple Availability Zones. Each index used a set number of primary shards with replicas, rolling over after shards reached a certain size. At first glance, the configuration appeared well-balanced, with shard copies distributed across Availability Zones and each node holding a manageable share of the data.

However, the primary shard count was lower than the total data node count. This meant that searches targeting the latest data, which is the most common access pattern, would only execute across a subset of available nodes. As a result, some nodes developed consistent CPU-based hotspots while the rest remained underutilized (Figure 5).

Figure 5: Node-level CPU utilization showing hotspots

As shown in the preceding figure, some nodes reach as high as 90 percent CPU utilization while several others remain under 20 percent, highlighting the uneven distribution of query execution across the cluster.

Recommendations and implementation

Based on these observations, AWS worked together with Epic Games on a set of targeted optimizations spanning hardware selection, sharding strategy, index mappings, and query behavior. The following sections detail each recommendation and how it was implemented.

Right-sizing the cluster

Because aggregation queries are CPU-intensive by nature and the cluster’s JVM memory pressure was well within acceptable ranges, AWS recommended migrating from memory-optimized r7g instances to compute-optimized c7g instances. The c7g family offers a higher ratio of vCPU to RAM, which is better suited for workloads where processing power rather than memory capacity is the binding constraint.

The proposed architecture called for a larger number of c7g nodes than the existing r7g count. This migration achieved approximately 33 percent more aggregate CPU capacity across the cluster while operating with two-thirds of the original memory. The net effect was a meaningful cost reduction of approximately 10 percent, delivering more processing power at lower cost by aligning the instance profile with the actual nature of the workload (Table 1).

 

R7g (Before) c7g (After) Net Impact
Instance Family Memory Optimized Compute Optimized Better CPU-to-RAM alignment for aggregation workloads
vCPUs per Node Same Same Same per-node CPU. More nodes = higher aggregate CPU
Memory per Node Higher Lower Reduced unused memory; JVM heap unchanged
Aggregate CPU Baseline +33% more total vCPUs Distributed more evenly across higher node count
Cost Baseline ~10% reduction More performance per dollar spent

Table 1: Instance migration comparison, r7g compared to c7g

Sharding strategy

To support the new cluster sizing, the Epic Games team changed the sharding strategy so that the number of primary shards matches the data node count, with 1 replica. This distributes both the write-heavy load and the batch aggregation search query load evenly on all the available data nodes.

The team employed ISM (Index State Management) policies to manage shard sizing through rollover, targeting shard sizes within recommended bounds using min_primary_shard_size. This kept shard counts bounded and predictable, providing a clear scaling pattern: adjust the node count, then update the ISM policy accordingly.

After implementation, node-level CPU utilization showed a much more even distribution (Figure 6).

Figure 6: Node-level CPU utilization after sharding optimization

As shown in Figure 6, all nodes in the domain are working at similar CPU utilization levels, confirming that data and traffic are well distributed across the cluster with no node hotspots.

Mapping optimization

The index mappings had both text and keyword field types enabled on many fields, even though access patterns showed those fields were only used for aggregation, sorting, or filter context, and never for full-text match queries. Removing the redundant text field type reduced storage overhead and improved query performance by eliminating unnecessary analysis at index time.

For high-cardinality string fields, the murmur3 field type does a compute-once-and-store optimization for cardinality aggregation. Instead of hashing keyword values at query time, murmur3 computes the hash once at index time and stores it as a numeric doc_value, so the aggregation can skip the expensive string hashing step at query time (the cardinality estimate itself is still computed at query time).

The following example illustrates the mapping changes:

Before: After:
"some_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},
"another_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},
"cardinality_field": {
  "type": "text",
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  }
},

"some_field": {
  "type": "keyword"
},
"another_field": {
  "type": "keyword"
},
"cardinality_field": {
  "type": "keyword",
  "fields": {
    "hash": {
      "type": "murmur3"
    }
  }
},

These mapping changes reduced overall storage, lowered shard count (which reduced CPU requirements), and reduced cluster manager node state size.

Index optimization

Additional index-level optimizations were applied to improve query performance and reduce overhead. Index sorting was configured to default to the primary date field, which improves performance for time-based access patterns by aligning the physical data layout with the most common query order. The ISM policy was updated to force merge indices down to 1 segment after rollover, reducing segment overhead on read-only indices. Finally, the refresh interval was tuned to balance indexing throughput with search freshness.

Upgrading from OpenSearch Service 2.17 to 3.1

The domain was upgraded from OpenSearch Service 2.17 to 3.1, which reduced error counts and improved throughput at the Amazon OpenSearch Ingestion pipeline level. The performance gains were notable: p99 latency on sum aggregations dropped by 40–50 percent after the upgrade alone, and large 96-hour cardinality aggregations saw p95 drop over 40 percent. General query performance improved across all query types, and thread pool pressure was reduced significantly, leading to far fewer 429 errors (Figure 7).

Figure 7: Query performance before and after the OpenSearch Service 3.1 upgrade

Upgrading from Graviton 3 to Graviton 4

The instance types were upgraded from c7g (Graviton 3) to c8g (Graviton 4). The performance gains were immediate:

  • p99 on all queries: 380 ms to 250 ms.
  • p95 on all queries: 245 ms to 230 ms.
  • p90 on all queries: 225 ms to 200 ms.
  • p50 on all queries: 100 ms to 70 ms.

Date-windowed cardinality queries saw their p99 halved from 220 ms to 98 ms, with sum-based aggregations experiencing similar gains. Overall throughput increased by 16 percent.

Tiered caching

With the upgrade to OpenSearch Service 3.1, the team enabled tiered caching. Tiered caching extends the default on-heap request cache with a disk-based tier. When items are evicted from the on-heap cache, they spill into a larger disk cache on the node’s local SSD rather than being discarded. This allows the cluster to retain cached results for a much larger set of queries without increasing JVM heap usage.

The batch aggregation jobs in Epic Games’ workload issue repeated queries over overlapping time windows. The on-heap cache alone was too small to retain results across successive job runs, so expensive aggregations were recomputed each time. With the disk tier enabled, results from longer time-window aggregations (such as the 96-hour cardinality queries) persisted between runs. This produced more consistent and faster results on some of the larger aggregation queries, particularly those spanning longer time windows.

Results summary

The following table summarizes the impact of each optimization.

Optimization Strategy Impact
Right-sizing (r7g to c7g) 33% more CPU, 10% cost reduction
Sharding rebalance Eliminated CPU hotspots across nodes
Mapping optimization Reduced storage, shard count, and cluster state size
OpenSearch Service 2.17 to 3.1 p99 sum aggs reduced 40-50%, fewer 429 errors
Graviton 3 to Graviton 4 p99 380 ms to 250 ms, 16% higher throughput
Tiered caching More consistent results on large aggregation queries

Table 2: Results summary

Conclusion

By optimizing their OpenSearch Service deployment, a team at Epic Games reduced p99 query latency from 380 ms to 250 ms, increased throughput by 16 percent, and lowered costs by 10 percent. These gains came from aligning instance types, sharding strategy, mappings, and engine versions with the workload’s actual demands.

To learn more about optimizing Amazon OpenSearch Service for your workloads, see Best practices for Amazon OpenSearch Service. For details on supported instance types, see Supported instance types in Amazon OpenSearch Service.


About the authors

Jon Evans

Jon is a Principal Software Engineer on the Epic Games Data Platform team. He builds and architects software solutions such as backend services, streaming pipelines and APIs to integrate analytics data to player facing products.

Aswath Srinivasan

Aswath Srinivasan

Aswath is a Senior Search Engine Architect at Amazon Web Services currently based in Munich, Germany. With over 18 years of experience in various search technologies, Aswath currently focuses on OpenSearch. He is a search and open-source enthusiast and helps customers and the search community with their search problems.

Gena Gizzi

Gena Gizzi

Gena is a Senior Games Solutions Architect at Amazon Web Services based in Southern California. She works with games customers to help optimize and scale their cloud infrastructure on AWS. She loves playing video games, especially Fortnite!

Rajani Guptan

Rajani Guptan

Rajani is a Senior Technical Account Manager at AWS Enterprise Support, where she helps large-scale gaming customers optimize their cloud infrastructure. She is passionate about building resilient, cost-efficient architectures and sharing operational best practices with the broader community. Outside of work, she enjoys gardening and spending time outdoors.

Amazon OpenSearch Service extends version lifecycle support timelines

Post Syndicated from Kuldeep Yadav original https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-extends-version-lifecycle-support-timelines/

In November 2024, we announced Standard and Extended Support dates for legacy Elasticsearch versions (1.5 through 7.8) and OpenSearch versions (1.0 through 1.2, and 2.3 through 2.9) running on Amazon OpenSearch Service. At that time, Extended Support for these versions was set to end on November 7, 2026 (except Elasticsearch 5.6, for which Extended Support ends on November 7, 2028), after which domains would no longer receive security fixes or operating system patches.

Since that announcement, many customers have upgraded to newer versions. However, some customers need more time to plan and complete their migrations. To provide this flexibility, we are continuing security and operating system patch coverage for these versions for an additional 12 months, through November 7, 2027, at an updated support rate.

Today, we’re announcing two updates: Extended Support extension for present versions and End of Standard Support and Extended Support for additional versions.

Extended Support extension for present versions

We are continuing security and operating system patch coverage for Elasticsearch versions 1.5 through 7.8, OpenSearch versions 1.0 through 1.2, and OpenSearch versions 2.3 through 2.9 for an additional 12 months. Coverage will now continue through November 7, 2027, giving customers additional time to plan and execute their migrations to the latest OpenSearch versions.

From November 7, 2026, the Extended Support surcharge for these versions will effectively double your instance pricing for the extension period. Storage costs are not affected. Elasticsearch 5.6, for which existing Extended Support rates end on November 7, 2028, will continue at the standard Extended Support cost of $0.0065 per Normalized Instance Hour (NIH). During this period, these versions will continue to receive critical security patches and operating system updates.

See the following table for the updated Extended Support end dates.

Software version End of Standard Support Original End of Extended Support date Updated End of Extended Support date
Elasticsearch versions 1.5 and 2.3 November 7, 2025 November 7, 2026 November 7, 2027
Elasticsearch versions 5.1 to 5.5 November 7, 2025 November 7, 2026 November 7, 2027
Elasticsearch version 5.6 November 7, 2025 November 7, 2028 No change
Elasticsearch versions 6.0 to 6.7 November 7, 2025 November 7, 2026 November 7, 2027
Elasticsearch versions 7.1 to 7.8 November 7, 2025 November 7, 2026 November 7, 2027
OpenSearch versions 1.0 to 1.2 November 7, 2025 November 7, 2026 November 7, 2027
OpenSearch versions 2.3 to 2.9 November 7, 2025 November 7, 2026 November 7, 2027

We recommend that you upgrade to the latest available OpenSearch version.

End of Standard Support and Extended Support for additional versions

Today we are announcing end of Standard and Extended Support dates for Elasticsearch versions 6.8, 7.9, and 7.10, OpenSearch version 1.3, and OpenSearch versions 2.11 to 2.19. For future updates on versions in Standard Support and Extended Support, follow supported versions.

For OpenSearch versions running on Amazon OpenSearch Service, we provide at least 12 months of Standard Support after the end-of-support date for the corresponding upstream open source OpenSearch version. Alternatively, we provide 12 months of Standard Support after the release of the next minor version on Amazon OpenSearch Service, whichever is longer. This aligns with the open source OpenSearch maintenance policy.

We categorize these versions into two groups:

  • The last versions of each major version family (ES 6.8, ES 7.10, OS 1.3, OS 2.19) will receive 3 years of Extended Support at the standard Extended Support charge of $0.0065 per NIH.
  • All other minor versions with clear upgrade paths within the same major family (ES 7.9, OS 2.11, OS 2.13, OS 2.15, OS 2.17) will receive 1 year of Extended Support at the same standard rate of $0.0065 per NIH.

After Extended Support ends for a version, domains running that version will not receive bug fixes or security updates. The following table shows the end of Standard Support and Extended Support dates for Elasticsearch and OpenSearch versions.

Elasticsearch versions

Software version End of Standard Support End of Extended Support
Elasticsearch version 6.8 November 7, 2027 November 7, 2030
Elasticsearch version 7.9 November 7, 2027 November 7, 2028
Elasticsearch version 7.10 November 7, 2027 November 7, 2030

OpenSearch versions

Software version End of Standard Support End of Extended Support
OpenSearch version 1.3 November 7, 2027 November 7, 2030
OpenSearch version 2.11 November 7, 2027 November 7, 2028
OpenSearch version 2.13 November 7, 2027 November 7, 2028
OpenSearch version 2.15 November 7, 2027 November 7, 2028
OpenSearch version 2.17 November 7, 2027 November 7, 2028
OpenSearch version 2.19 November 7, 2027 November 7, 2030
OpenSearch version 3.1 and above Not announced Not announced

Upgrading OpenSearch Service domains: We recommend that you upgrade your domains to the latest available OpenSearch version to derive maximum value out of Amazon OpenSearch Service. Minor version upgrades on OpenSearch don’t contain breaking changes. These version upgrades are typically non-disruptive. We recommend moving to the latest minor version. See Upgrading OpenSearch Service domains for detailed instructions. You can also use the Migration Assistant for Amazon OpenSearch Service for upgrading to newer versions.

New domain creation: New domain creation will be blocked after Extended Support ends for each version.

Calculating Extended Support charges

Amazon OpenSearch Service domains running versions under Extended Support will be charged a flat additional fee per NIH. NIH is computed as a factor of the instance size (for example, medium or large), and the number of instance hours.

Depending on which version you are on, the Extended Support charges are as follows:

  • Versions that have been on Extended Support (ES 1.5–7.8 (other than ES 5.6), OS 1.0–1.2, OS 2.3–2.9) from November 7, 2026: Extended Support cost will be equal to your instance price. This will be in addition to the standard instance pricing, effectively doubling the instance cost. Storage costs are not affected. Example (for versions that have been on Extended Support from November 7, 2026): If you are running an m7g.medium.search instance priced at $0.068/hr (on-demand) in US East (N. Virginia) for 24 hours, your standard instance cost is $1.632/day ($0.068×24). The Extended Support surcharge for the extension period will be equal to your instance cost ($1.632/day), doubling your instance pricing to ~$3.264/day. Storage costs remain unchanged.
  • New versions coming under Extended Support (ES 6.8, 7.9, 7.10, OS 1.3, OS 2.11–2.19): $0.0065 per NIH (standard Extended Support rate). See the pricing page for exact pricing by Region. Example (new versions — standard rate): If you are running an m7g.medium.search instance for 24 hours in the US East (N. Virginia) Region, priced at $0.068 per instance hour (on-demand), you will typically pay $1.632 ($0.068×24). If you are running a version that is in Extended Support, you will pay an additional $0.0065 per NIH. This is computed as $0.0065 × 24 (instance hours) × 2 (normalization factor for medium) = $0.312 for Extended Support for 24 hours. The total amount you will pay for 24 hours is $1.944 ($1.632 + $0.312, excluding storage cost).

The following table shows the normalization factor for various instance sizes in OpenSearch Service.

Instance size Normalization Factor
nano 0.25
micro 0.5
small 1
medium 2
large 4
xlarge 8
2xlarge 16
4xlarge 32
8xlarge 64
9xlarge 72
10xlarge 80
12xlarge 96
16xlarge 128
18xlarge 144
24xlarge 192
32xlarge 256

Summary

The latest OpenSearch versions include new features, performance and resiliency improvements, and security enhancements. With today’s announcement, we are:

  • Continuing security and operating system patch coverage for present versions through November 7, 2027, giving customers additional time to complete their upgrades at a new Extended Support rate.
  • Announcing Standard and Extended Support timelines for the next set of versions (ES 6.8, 7.9, 7.10, OS 1.3, 2.11–2.19) with predictable cost visibility.

We recommend that you upgrade to the latest OpenSearch versions to get the most benefit out of OpenSearch Service. For any questions on Standard and Extended Support options, see the FAQs. For further questions, contact AWS Support.


About the authors

Kuldeep Yadav

Kuldeep Yadav

Kuldeep is a Principal Technical Program Manager at AWS. He’s passionate about driving innovation and complex problem solving. He works closely with teams and customers to ensure operational excellence and achieve more with less.

Arvind Mahesh

Arvind Mahesh

Arvind is a Senior Manager-Product at AWS (Amazon OpenSearch Service). With close to two decades of technology experience, he brings deep expertise across Analytics, Search, Cloud, Network Security, and Telecom.

Jon Handler

Jon Handler

Jon is a Senior Principal Solutions Architect at AWS. He works closely with OpenSearch and Amazon OpenSearch Service, guiding a broad range of customers looking to move their search and log analytics workloads to the AWS Cloud.

Transforming search at Delivery Hero: A migration journey to OpenSearch Service with radial search

Post Syndicated from Sayan Das original https://aws.amazon.com/blogs/big-data/transforming-search-at-delivery-hero-a-migration-journey-to-opensearch-service-with-radial-search/

Have you ever searched for something like “low fat yogurt” at any online grocery store and noticed how the results seem to understand what you mean? Instead of only showing items with an exact match, the top-ranked products are often semantically related. You might see items like “Greek yogurt” or “yogurt with 0.5% fat,” even when only one word matches lexically. This is the power of semantic search, and when combined with traditional lexical search, it creates a hybrid search experience that delivers both precision and recall.

Semantic search returning products semantically related to a low fat yogurt query

At Delivery Hero, one of the world’s leading online food delivery platforms, the search team has been using semantic search for grocery verticals since 2024. What started as a proof-of-concept has evolved into a production-grade hybrid search system powered by Amazon OpenSearch Service. This system combines radial vector search with lexical retrieval to deliver highly relevant product results at scale.

In this post, we walk through how Delivery Hero migrated their semantic search infrastructure to Amazon OpenSearch Service, why they chose radial search over traditional k-nearest neighbor (k-NN) search, and the optimizations that made the system fast, cost-effective, and flexible for experimentation.

Legacy system overview

The original semantic search system was built as a standalone service using SpringBoot and Apache Lucene 9.9, deployed on Kubernetes. The retrieval flow worked as follows:

  1. A user starts a search on the application.
  2. The semantic search system retrieves the top 50 nearest-neighbor candidates from a static in-memory Lucene index.
  3. These candidates passed through a filtering layer to remove out-of-stock items.
  4. The filtered semantic results were merged with a parallel set of lexical search results.
  5. A final ranking step combined both candidate sets to produce the response.

The team iterated on this system over seven versions and conducted multiple A/B tests to refine the approach. The initial system performed well, however as the business scaled, several pain points emerged:

  • Scalability limitations: Running vector indices as static, in-memory structures inside Kubernetes pods meant that scaling required provisioning larger pods or adding replicas. Both options were expensive and operationally complex.
  • Multi-model experimentation was difficult: Running A/B/C tests with three different product embedding model variants required fitting all models within a Kubernetes stateless workload. This created memory pressure and complicated deployment pipelines.
  • Operational overhead: Managing index builds, deployments, and version rollouts for a custom Lucene-based service required significant engineering effort compared to a managed service.

Architecture modernization with OpenSearch Service

By the end of 2025, Delivery Hero had migrated their entire search infrastructure from self-managed Elasticsearch 7.x on Google Kubernetes Engine (GKE) to the fully managed Amazon OpenSearch Service 3.x. This migration created a natural opportunity to consolidate the legacy semantic search service into OpenSearch as well.

The new architecture separates concerns into two distinct pipelines: an ingestion pipeline for indexing product embeddings, and an inference pipeline for real-time hybrid retrieval.

Ingestion pipeline

For the ingestion pipeline, Delivery Hero chose Amazon OpenSearch Ingestion (OSIS) to sync product embedding data from Amazon Simple Storage Service (Amazon S3) to the OpenSearch domain.

Ingestion pipeline syncing product embeddings from Amazon S3 to Amazon OpenSearch Service through OpenSearch Ingestion

The flow works as follows:

  1. ML model
  2. Airflow job: An existing Apache Airflow job periodically generates product embeddings using an external machine learning (ML) model and periodically dumps the results (product parent ID + embedding vector) to an S3 bucket.
  3. OpenSearch Ingestion pipeline: An OpenSearch Ingestion pipeline is configured with a scheduled S3 scan that performs a nightly scan from S3 and updates the new k-NN index in OpenSearch Service.
version: '2'
embedding-pipeline:
  source:
    s3:
      acknowledgments: true
      scan:
        buckets:
          - bucket:
              name: my-bucket-name
              filter:
                include_prefix:
                  - vector-search/json-index/latest
        range: PT24H
        scheduling:
          interval: PT24H
      aws:
        region: eu-central-1
        sts_role_arn: arn:aws:iam::<aws-account-id>:role/osis-pipeline-role
      codec:
        ndjson: {}
      compression: none
  workers: '1'
  sink:
    - opensearch:
        hosts:
          - "https://<search-domain>.<aws-region>.es.amazonaws.com"
        aws:
          serverless: false
          region: eu-central-1
          sts_role_arn: arn:aws:iam::<aws-account-id>:role/search-xxx
        index_type: custom
        index: emb_products_v1
        template_content: ...
        template_type: index-template
        routing: '${global_entity_id}'
        document_id: '${global_entity_id}:${master_code}'
        max_retries: '3'

Because the index stores product parent IDs and embeddings are regenerated in batch, there is no need for real-time updates. This allows the team to refresh and force-merge the index once per day, resulting in highly optimized segment structures and fast retrieval speeds (p99 < 35 ms during peak hours).

Setting up the OSIS pipeline required only a few lines of Terraform, making it straightforward to provision and maintain as infrastructure-as-code.

Inference pipeline

On the retrieval side, the system runs a hybrid search strategy that combines radial vector search with lexical search in parallel:

Hybrid inference pipeline running radial vector search and lexical search in parallel before merging and re-ranking results

  1. Query embedding: A user’s search query first reaches the Query Understanding (QU) service, where it is encoded into an embedding using the same live ML model employed for product embeddings. To optimize performance, embeddings for top queries are cached.
  2. Parallel lexical and semantic retrieval:
    • A radial k-NN search runs against the product embeddings index using min_score to retrieve all semantically similar products above a similarity threshold.
    • A lexical BM25 search runs against the product catalog index.

      Chart comparing p95 OpenSearch take-time for lexical and semantic search

      Comparing p95 OpenSearch time for both lexical and semantic search.

  1. ID resolution and inventory filter: Because the k-NN index stores product parent IDs, a resolution step maps these to individual product IDs via a secondary index that maintains near real-time inventory updates. This approach satisfies two key business requirements within a single retrieval call: product-id resolution and real-time availability filtering.
  2. Merge and re-rank: A custom post-processing step combines results from both lexical and radial search, applies re-ranking logic, and returns the final result set.

Traditional k-NN search in OpenSearch uses a top-k approach: you ask for the k nearest neighbors, and you get exactly k results regardless of how similar they actually are. This works well for many use cases, but it has a fundamental limitation for product search. It always returns a fixed number of results, even when some of those results are not semantically relevant.

Radial search solves this by flipping the paradigm. Instead of asking “give me the 50 closest items,” you ask “give me all items that are at least this similar.” This is done using the min_score parameter in the k-NN query:

GET product-embeddings/_search
{
  "query": {
    "knn": {
      "embedding": {
        "vector": [0.12, 0.45, 0.78, ...],
        "min_score": 0.72
      }
    }
  }
}

When using radial search with cosine similarity as the space type, OpenSearch normalizes scores using the related formula (score = (1 + cosine_similarity) / 2), as documented in the OpenSearch knn-spaces reference.

This means a min_score of 0.72 in the query example, does not directly correspond to cosine similarity. Instead, 0.72 is the normalized OpenSearch score which translates to 44% cosine similarity (that is, cosine_similarity = 2 × 0.72 – 1 = 0.44).

If you need results with at least 90% cosine similarity, apply the formula:

min_score = (1 + 0.90) / 2 = 0.95. So, you would set “min_score”: 0.95 in your query.

This approach offers several advantages for product search:

  • Quality over quantity: Low-relevance results are excluded at the retrieval stage rather than relying on downstream re-ranking to filter them out.
  • Variable result set size: The system naturally adapts to query specificity. Niche queries return fewer, more precise results. Broad queries return more candidates for the re-ranker to work with. For example, a highly specific query like “Oatly oat milk barista edition” might return 5 results, while a broader query like “milk” might return 200.
  • Better recall-precision trade-off: By tuning the min_score threshold, the team can directly control the balance between returning too many irrelevant results and missing relevant ones.

Choosing the right min_score threshold is important. Set it too high and you miss relevant products. Set it too low and you flood the re-ranker with noise.

Delivery Hero approaches threshold selection through systematic experimentation. To achieve optimal precision across diverse markets, a tailored min_score threshold is assigned to each country and query type. These thresholds are meticulously determined through rigorous offline evaluations, which use historical user interaction and manually labeled data to establish a rough estimate. This initial estimate is then further refined and validated through a series of live A/B experiments.

Evaluation of the new search system

One of the key advantages of the new architecture is how naturally it supports experimentation. At Delivery Hero, we store three variants of product embeddings within a single document:

PUT product-embeddings/_doc/1?routing=FP_DE
{
  "master_product_code": "abc123",
  "embedding_variant_1": [0.12, 0.45, 0.78, ...],
  "embedding_variant_2": [0.21, 0.4, 0.98, ...],
  "embedding_variant_3": [0.13, 0.65, 0.58, ...],
  "global_entity_id": "FP_DE"
}

In this example, embedding_variant_1, embedding_variant_2, and embedding_variant_3 are generated from three different models for A/B/C testing. After each test, the winning variant is designated as the control, while the other two are replaced with new models for further experimentation. With this approach, the team can iterate continuously while maintaining constant space complexity.

Optimizations of large scale production system

Engine upgrade: OpenSearch 2.17 to 3.3

Production k-NN query latency metrics from one of the busiest countries after the OpenSearch 3.3 upgrade

Production metrics from one of the busiest countries.

OpenSearch 3.x introduced significant performance improvements for vector search workloads. Post-upgrade to OpenSearch 3.3, we observed a ~18% reduction in p95 latency for k-NN queries.

For Delivery Hero’s use case, the k-NN search latency was already very low on OpenSearch 2.17 (p99 of 20–30 ms), which meant the upgrade to 3.3 was not strictly necessary for all clusters. The cluster serving the control group in A/B tests still runs on OpenSearch 2.17.

Shard routing

To minimize cross-shard overhead during k-NN queries, Delivery Hero implemented custom shard routing based on geographic market. Because each market (for example, Germany, Sweden, and Finland) has its own product catalog, routing queries to market-specific shards avoids unnecessary fan-out across the entire index.

This is an example of how to configure routing at index time and search time using the _routing field:

PUT product-embeddings/_doc/1?routing=FP_DE
{
  "master_product_code": "abc123",
  "embedding_variant_1": [0.12, 0.45, 0.78, ...],
  "embedding_variant_2": [0.21, 0.4, 0.98, ...],
  "embedding_variant_3": [0.13, 0.65, 0.58, ...],
  "global_entity_id": "FP_DE"
}

And at query time:

GET product-embeddings/_search?routing=FP_DE
{
  "query": {
    "knn": {
      "embedding_variant_2": {
        "vector": [0.12, 0.45, 0.78, ...],
        "min_score": 0.72
      }
    }
  }
}

This ensures that a query for the German market only hits shards containing German products, reducing latency and compute overhead.

Refresh interval

Because the product embedding index is updated only once per day via the OSIS batch pipeline, there is no need for the default 1-second refresh interval. Delivery Hero configured the index with a longer refresh interval during ingestion and triggers a manual refresh + force merge after the nightly batch completes.

Impact on the business

The migration from self-managed Lucene on Kubernetes to Amazon OpenSearch Service achieved a ~50% reduction in p95 latency, dropping response times from a variable 200ms+ to a stable 100ms baseline. This transition significantly improved system consistency by eliminating the high variance and rhythmic latency spikes seen in the previous architecture.

End-to-end service latency dropping to a stable 100 ms baseline after rolling out semantic search on OpenSearch for foodpanda and yemeksepeti

End service latency after rolling out semantic search with OpenSearch for foodpanda and yemeksepeti.

Beyond raw latency, the operational benefits were significant:

  • Reduced infrastructure complexity: Eliminating the standalone Lucene service removed an entire deployment pipeline, monitoring stack, and on-call rotation.
  • Faster experimentation: New embedding models can be tested by creating a new index and adjusting query routing, without requiring code deployments.
  • Cost efficiency: Using OpenSearch’s managed infrastructure and the batch ingestion pattern (refresh once per day) reduced compute costs compared to running always-on Kubernetes pods with in-memory indices.

Conclusion

By combining radial search with lexical retrieval, Delivery Hero’s team built a system that adapts dynamically to query intent. It returns precise results for specific queries and broader candidate sets for general ones.

The migration to Amazon OpenSearch Service demonstrates how a managed search platform can simplify the operational complexity of vector search while improving performance.

To get started with vector search on Amazon OpenSearch Service, see the AI search documentation and the OpenSearch radial search guide.


About the authors

Sayan Das

Sayan Das

Sayan is Staff Software Engineer at Delivery Hero specializing in high-performance search infrastructure and large-scale distributed systems. With a deep background in Big Data engineering and core search internals (Solr, Lucene, OpenSearch)

Hajer Bouafif

Hajer Bouafif

Hajer is a senior solutions architect in Data Analytics and ML search with a background in Big Data engineering. Hajer provides organizations with best practices and well-architected reviews to build large-scale Machine Learning search solutions

Build a contract compliance search system with Amazon OpenSearch

Post Syndicated from Durga Prasad original https://aws.amazon.com/blogs/big-data/build-a-contract-compliance-search-system-with-amazon-opensearch/

For legal and compliance teams, auditing a repository of thousands of contracts for a single regulatory obligation shouldn’t take weeks. But with keyword search, it often does. A search for “inadvertent access notification” returns exact matches while missing functionally equivalent clauses such as “security incident disclosure” or “unauthorized access reporting.” This creates two problems:

Discovery gap: Critical risk exposure goes undetected because keyword search cannot match semantically equivalent terms across different contracts.

Review latency: After finding relevant contracts, legal counsel must manually scan lengthy documents to locate the specific clauses that matter. This process can stretch from minutes to hours per document.

Amazon OpenSearch Service is a fully managed search and analytics service that configures, manages, and scales OpenSearch clusters in the AWS Cloud. It supports use cases from log analytics and application monitoring to full-text search and real-time security analytics. It also supports AI-powered semantic search.

Amazon OpenSearch Service addresses both problems through two capabilities:

  • Semantic search retrieves contracts based on meaning rather than exact keyword matches, closing the discovery gap.
  • Semantic highlighting pinpoints the exact clauses within retrieved contracts that answer the query, reducing review time from hours of manual scanning to seconds of targeted reading.

In this post, you build a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. You deploy the solution using two AWS CloudFormation stacks, test it with synthetic contract documents, and see how a single query surfaces both the right contracts and the right clauses within them.

Solution overview

The solution uses a two-stage retrieval and extraction pipeline. First, semantic search identifies relevant contracts across the repository. Then, semantic highlighting marks the specific clauses within those contracts that match the query intent.

The following diagram illustrates the solution architecture:

Solution architecture showing contracts flowing from Amazon S3 through OpenSearch Ingestion and Amazon Bedrock embeddings to semantic search and Amazon SageMaker AI highlighting

  1. Upload contracts to Amazon Simple Storage Service (Amazon S3) – Contract documents (JSON format) are uploaded to an Amazon S3 bucket, which serves as the centralized document repository.
  2. Amazon OpenSearch Ingestion (OSI) reads from S3 – A serverless OSI pipeline detects new documents in the S3 bucket and reads them for processing.
  3. OpenSearch ingest pipeline generates embeddings through Amazon Bedrock – As documents arrive, the ingest pipeline’s text_embedding processor invokes Amazon Titan Text Embeddings V2 through an ML Commons Bedrock connector. This converts contract text into 1024-dimension vector representations, stored in a k-NN index that uses the faiss engine.
  4. User submits a search query – A user queries the system with natural language (for example, “data protection regulations”) through a test AWS Lambda function that forwards the request to OpenSearch using the neural query type.
  5. OpenSearch generates the query embedding – OpenSearch converts the user’s natural language query into a vector embedding using the same machine learning (ML) Commons Amazon Bedrock connector and Amazon Titan V2 model.
  6. Amazon OpenSearch Service performs semantic search – OpenSearch uses k-NN vector similarity to retrieve contracts that are semantically relevant to the query, even when exact terminology differs.
  7. Amazon SageMaker AI performs semantic highlighting – The opensearch-semantic-highlighter-v1 model, hosted on an Amazon SageMaker AI GPU endpoint, scores sentence relevance using cross-encoder inference and wraps the matching clauses in <em> tags for targeted reading.

How semantic search and semantic highlighting work together

The system processes queries in two steps:

Step 1 – Semantic search (document discovery): You query the contract corpus using natural language. The system retrieves contracts with semantically similar concepts, even when exact terminology differs. For example, searching for “force majeure” returns contracts discussing “natural disasters” or “unforeseeable circumstances” because the system understands these concepts are related.

Step 2 – Semantic highlighting (clause identification): After relevant contracts are retrieved, semantic highlighting automatically marks the clauses that semantically match your search intent. Instead of scanning pages of legal text, you immediately see the specific paragraphs that answer your question.

The difference between standard keyword highlighting and semantic highlighting is significant:

  • Keyword highlighting wraps individual matching words: <em>termination</em> and <em>rights</em>.
  • Semantic highlighting wraps entire relevant clauses: <em>Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days.</em>.

This reduces false positives, cuts review time, and provides explainability for why each document was retrieved.

Semantic highlighting model deployment

Before the system can highlight clauses based on meaning, the opensearch-semantic-highlighter-v1 model must be deployed to an Amazon SageMaker AI GPU endpoint and registered with the OpenSearch ML Commons plugin through a remote connector.

Stack 2 of the CloudFormation deployment automates this process. It performs the following steps:

  1. Downloads the model artifact from an AWS-managed source and deploys it to an Amazon SageMaker AI endpoint (ml.g5.xlarge).
  2. Creates a remote ML Commons connector in OpenSearch that points to the SageMaker endpoint.
  3. Registers the model with the QUESTION_ANSWERING function so that OpenSearch can use the model’s cross-encoder capabilities to score sentence relevance at query time.

The equivalent manual registration call (handled automatically by the stack) is:

POST /_plugins/_ml/models/_register?deploy=true
{
  "name": "amazon/sentence-highlighting/opensearch-semantic-highlighter-v1",
  "version": "1.0.0",
  "model_format": "TORCH_SCRIPT",
  "function_name": "QUESTION_ANSWERING"
}

You don’t need to run this manually. The deployment script and CloudFormation stack handle model registration end-to-end. The resulting model ID is automatically passed to the query Lambda function for use in semantic highlighting requests.

Index configuration

The index uses a k-NN vector field with 1024 dimensions (matching the Amazon Titan V2 output) and the faiss engine with HNSW method. The mapping includes both a knn_vector field for semantic retrieval and a standard text field for keyword matching and highlighting. When you search for “liability limits,” OpenSearch first retrieves documents through vector similarity, then uses the Amazon SageMaker AI model to identify and wrap the specific relevant sentences in <em> tags.

PUT /legal-contracts-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "clause_text": { "type": "text" },
      "clause_embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "l2"
        }
      }
    }
  }
}

Implementation steps

This section walks you through deploying the solution using two AWS CloudFormation stacks and two shell scripts. You first set up the core infrastructure (OpenSearch, ingestion pipeline, and ML Commons Bedrock connector), then deploy the semantic highlighting model on Amazon SageMaker AI.

Prerequisites

To deploy this solution, you need:

  • An active AWS account with permissions to create Amazon S3 buckets, AWS Lambda functions, Amazon SageMaker AI endpoints, Amazon Bedrock model access, Amazon OpenSearch Ingestion pipelines, Amazon OpenSearch Service domains, and AWS Identity and Access Management (IAM) roles (including iam:PassRole and sts:AssumeRole). For the exact least-privilege policy, see iam-deployer-policy.json in the repository. Both CloudFormation stacks require the CAPABILITY_NAMED_IAM acknowledgement.
  • Amazon Bedrock model access enabled for Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0).
  • Familiarity with AWS CloudFormation.
  • Estimated deployment time: approximately 35 minutes.
  • Estimated cost: approximately USD $ 2.00–3.00 for a quick demo. Delete the stacks promptly after testing.
  • This post uses US East (N. Virginia) as the deployment AWS Region. Verify service availability in your preferred Region before deploying.

Deploy the solution

The solution deploys using two AWS CloudFormation stacks and two shell scripts. The demo includes synthetic contract documents covering common contract types including software licenses, data processing agreements, managed services, and software as a service (SaaS) subscriptions.

Clone the repository and run the deployment script:

git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./deploy.sh

The deployment script creates the following resources across two stacks:

Stack 1:

  • An Amazon OpenSearch Service domain with fine-grained access control.
  • An Amazon OpenSearch Ingestion (OSI) pipeline that reads contracts from S3 and sends them to OpenSearch for indexing.
  • An ML Commons Bedrock connector and ingest pipeline that automatically generates 1024-dimension vector embeddings through Amazon Titan Text Embeddings V2 during document indexing.
  • A test Lambda function for querying the OpenSearch index using keyword, neural, or hybrid search with semantic highlighting support.
  • An S3 bucket for storing contract documents.
  • IAM roles for Lambda functions, the OSI pipeline, and OpenSearch access.

Stack 2:

  • An Amazon SageMaker AI endpoint hosting the semantic highlighting model.
  • A Lambda function that creates an ML Commons remote connector in OpenSearch and registers the highlighting model.

After both stacks deploy, the script automatically configures OpenSearch (role mappings, Amazon Bedrock connector, embedding model, k-NN index), ingests the sample contract data, and registers the semantic highlighting model.

The total deployment takes approximately 35 minutes to complete.

(Optional) Automated deployment with Claude Code CLI

If you have Claude Code CLI installed, you can deploy the solution using an AI-assisted workflow that creates a least-privilege IAM role scoped to this demo before deploying:

git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./scripts/create-deployer-role.sh
export OS_DEMO_DEPLOYER_ROLE=arn:aws:iam::<ACCOUNT_ID>:role/os-demo-deployer-role
export AWS_DEFAULT_REGION=us-east-1
claude "Deploy the OpenSearch semantic search demo following README.md"

Claude Code reads the repository instructions, assumes the deployer role, deploys both CloudFormation stacks in order, runs the setup scripts, and verifies the deployment end-to-end. The deployer role restricts actions to resources prefixed with os-demo-*, following the principle of least privilege.

Test the solution

After the deployment succeeds, follow these steps to test the solution.

  1. On the Lambda console, choose Functions in the navigation pane.
  2. Choose the function that has os-demo-query in its name.
  3. On the Test tab, in the Event JSON paste this keyword search query {"query": "data protection regulations?", "type": "keyword", "k": 3}
  4. Choose Test to run the Lambda function.

The following screenshot shows the Lambda function test configuration on the AWS Management Console with the keyword search query.

Lambda console Test tab with the keyword search query entered in the Event JSON field

The function processes the query in two ways depending on the search type:

For keyword search (enter: keyword): The function sends a standard match query to OpenSearch, which returns documents containing the exact query terms. The highlight fragments wrap individual matching words like <em>termination</em> and <em>rights</em>.

For neural search (enter: neural): The function sends a hybrid query to OpenSearch combining k-NN (semantic similarity) with keyword matching. OpenSearch automatically generates the query embedding through the ML Commons Amazon Bedrock connector using the same Amazon Titan V2 model. This returns semantically related documents even if they don’t contain the exact query terms. The SageMaker endpoint powers the semantic highlighting, identifying the most relevant clauses within each retrieved document. It wraps entire passages like <em>Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days.</em>.

  1. Download the highlight viewer HTML file and open it in the browser. This file helps you view the highlighted text.
  2. Copy the entire execution output of the Lambda execution, paste it into the placeholder in the HTML file, and then choose Load Results.
  3. The following screenshot shows that only the matching keywords are highlighted.

Highlight viewer showing only individual keywords highlighted in the keyword search results

  1. Next, paste the neural search query as input to the Lambda function to see how semantic highlighting works: {"query": "data protection regulations", "type": "neural", "k": 1}
  2. Choose Test to run, and then paste the entire output into the HTML viewer.

The viewer now displays entire sentences highlighted instead of individual keywords.

Highlight viewer showing entire relevant clauses highlighted in the neural search results

Optimizing for scale: batch semantic highlighting

In a standard search, a query might return dozens of relevant contracts. Using the default single inference mode, OpenSearch makes a separate ML call for every document in the result set. For a compliance officer reviewing 50 contracts, this sequential processing introduces noticeable latency.

OpenSearch 3.3 introduced batch inference mode to address this. Batch inference collects the matching documents and processes them in a single ML inference call. In the contract compliance use case, this shifts the performance characteristic from multiple sequential roundtrips to a single parallel execution on the Amazon SageMaker AI GPU.

To enable batch inference, first configure the cluster setting:

PUT _cluster/settings
{
  "persistent": {
    "search.pipeline.enabled_system_generated_factories": ["semantic-highlighter"]
  }
}

Then add batch_inference: true to your highlight options. The following query searches for data privacy clauses across the contracts and highlights the top 10 results using a single batch call:

POST /legal-contracts-index/_search
{
  "query": {
    "neural": {
      "clause_embedding": {
        "query_text": "standard for inadvertent access notification",
        "model_id": "<TEXT_EMBEDDING_MODEL_ID>",
        "k": 10
      }
    }
  },
  "highlight": {
    "fields": {
      "clause_text": { "type": "semantic" }
    },
    "options": {
      "model_id": "<REMOTE_HIGHLIGHTER_MODEL_ID>",
      "batch_inference": true,
      "max_inference_batch_size": 50
    }
  }
}

Best practices

Follow these recommendations to optimize performance, security, and cost-efficiency when deploying the contract compliance search system in production.

  • Experiment with overlapping chunk sizes (for example, 500 characters with a 10 percent overlap) in your OSI pipeline to verify that context is preserved for long indemnification or liability clauses.
  • Verify that your Amazon S3 buckets and OpenSearch domains are encrypted using AWS Key Management Service (AWS KMS). For production workloads containing sensitive data, make sure that all traffic stays within your virtual private cloud (VPC) through interface endpoints.

This demo uses simplified configurations for learning purposes. For production deployments, implement VPC isolation, AWS KMS encryption with customer-managed keys, and multi-AZ OpenSearch clusters.

Clean up resources

To avoid ongoing charges, delete the AWS CloudFormation stacks and associated resources:

  1. On the AWS CloudFormation console, choose Stacks in the navigation pane.
  2. Select the os-demo-highlighting stack (Stack 2) and choose Delete. Wait for deletion to complete.
  3. Select the os-demo-search stack (Stack 1) and choose Delete. Stack deletion takes approximately 10–15 minutes to complete.

The stack deletion will automatically remove:

  • OpenSearch domain.
  • SageMaker model and endpoint.
  • Lambda functions.
  • IAM roles and policies.
  1. After both stacks are deleted, manually delete the S3 bucket (opensearch-cfn-semantic-highlighting-us-east-1-<ACCOUNT_ID>) created for model artifacts. This bucket is provisioned at deploy time and is not managed by CloudFormation. Replace <ACCOUNT_ID> with your AWS account ID in the bucket name.

Conclusion

In this post, you built a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. The system helps close the discovery gap by retrieving contracts based on meaning rather than exact keywords, and it reduces review latency by highlighting the specific clauses that answer your query.

While we focused on legal agreements, the architecture described here is a blueprint for domains requiring high-stakes document discovery, including:

  • Regulatory filings: Identifying specific compliance mandates in financial reports.
  • Technical documentation: Pinpointing troubleshooting steps across massive product manuals.
  • Research and academia: Isolating specific methodologies within thousands of scientific papers.
  • Internal knowledge bases: Empowering employees to find exact policy language instantly.

To get started, deploy the solution from the sample repository on GitHub and try semantic search in the Amazon OpenSearch Service console. For more information about semantic search, see Semantic search in the Amazon OpenSearch Service Developer Guide.


About the authors

Durga Prasad

Durga Prasad

Durga is a Senior Consultant at AWS, specializing in the Data and AI/ML. He has over 18 years of industry experience and is passionate about helping customers design, prototype, and scale Big Data and Generative AI applications using AWS native and open-source tech stacks.

Chanpreet Singh

Chanpreet Singh

Chanpreet is a Senior Consultant at AWS with 19 years of industry experience, specializing in Data Analytics and AI/ML solutions. He partners with enterprise customers to architect and implement cutting-edge solutions in Big Data, Machine Learning, and Generative AI using AWS native services, partner solutions and open-source technologies. A passionate technologist and problem solver, he balances his professional life with nature exploration, reading, and quality family time.

Efficient log management with Amazon OpenSearch Service data streams

Post Syndicated from Praveen Krishnamoorthy Ravikumar original https://aws.amazon.com/blogs/big-data/efficient-log-management-with-amazon-opensearch-service-data-streams/

Time series data workloads in Amazon OpenSearch Service can present unique challenges for organizations, especially when dealing with continuously growing datasets. Many customers struggle with heavily loaded single indices that lead to high query latency, degraded performance, and unnecessary costs. In this post, we show you how to implement data streams with Index State Management (ISM) in Amazon OpenSearch Service. This approach automatically manages your time series data lifecycle and optimizes both performance and costs. Data streams distribute incoming data across multiple backing indices, helping to reduce single-index bottlenecks, while ISM policies automate rollover, retention, and storage tiering to help manage costs.

The challenge

While Amazon OpenSearch Service has long provided tools like Index State Management (ISM) for time series data management, many organizations still struggle with implementing optimal patterns for their continuously growing datasets. Common challenges include:

  • Performance degradation from index growth: As single indices grow unbounded, query latency increases, you might find shard sizes more difficult to manage, and you might experience strain on your cluster resources.
  • Manual index management overhead: Without automation, you must invest significant operational effort to manage index lifecycles, rollover, and retention.
  • Complex setup: Coordinating index templates, aliases, and ISM policies manually can be error-prone.
  • Inefficient resource utilization: All data residing in hot storage regardless of access patterns, leading to unnecessarily high costs.

Solution overview

As illustrated in Figure 1, our solution uses Amazon OpenSearch Service data streams combined with Index State Management to automatically distribute data across multiple indices and manage the data lifecycle. A data stream is an abstraction layer that simplifies time series data ingestion. It provides a single, consistent endpoint for writes while automatically managing multiple backing indices behind the scenes. Instead of writing directly to individual indices, applications write to the data stream, which routes data to the appropriate backing index.

Here’s how it works:

  • Data streams provide a single write index when data is first ingested, which can help streamline time series data ingestion.
  • When the backing index ages or grows to meet your defined criteria, ISM automatically performs the rollover operation.
  • You can use ISM policies to automatically transition your aged data to different storage tiers based on rules you define and configure.
  • You can automate the entire process through rules you define in the index template and ISM policies.

Architecture diagram showing time series data flowing into an Amazon OpenSearch Service data stream, which routes writes to multiple backing indices while ISM transitions aged indices from hot to UltraWarm storage

Figure 1 — Time series data workflow using Amazon OpenSearch Service data streams

Data ingests into an index according to your index template configuration. Over time, a data stream creates new indices automatically. Amazon OpenSearch Service manages the lifecycle, transitioning data from hot to warm storage according to the ISM policy configuration you define.

When to use this solution

This approach is ideal when:

Implementation steps

Prerequisites

Before you begin, make sure that you have the following:

The following steps walk through implementing a time series data solution, using a web server logs database as an example. You can run these steps using any of the following:

For this post, we use the Dev Tools Console in OpenSearch Dashboards. To access it:

  • Log in to OpenSearch Dashboards.
  • Navigate to Dev Tools (usually found on the menu under Management).
  • Use the interactive console to run the commands.

Section 1: Create data stream

  1. Create an ISM policy

First, create an ISM policy that defines the rules for index rollover and storage tier transitions. The following policy defines two states (hot and warm) and sets rules for when indices transition between them. The policy triggers a rollover when the document count reaches 1,000 and moves indices to warm storage after 2 minutes.

Note: The rollover and transitions configurations are only for demo purposes.

PUT _plugins/_ism/policies/ds-ism-policy
{
"policy": {
"description": "rollover policy when index is large",
"default_state": "hot",
"ism_template": [
{
"index_patterns": ["webserver-logs-data-stream*"],
"priority": 300
}
],
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_doc_count": 1000
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": {
"min_index_age": "2m"
}
}
]
},
{
"name": "warm",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"warm_migration": {}
}
]
}
]
}
}
  1. Create an index template for the data stream

An index template matches indices by a regex pattern and applies predefined settings and schema at index creation. The following template maps the required timestamp field for the data stream and associates matching indices with the ISM policy we created.

PUT _index_template/webserver-logs-data-stream-template
{
"index_patterns": ["webserver-logs-data-stream*"],
"data_stream": {
"timestamp_field": {
"name": "timestamp"
}
},
"template": {
"settings": {
"plugins.index_state_management.policy_id": "ds-ism-policy"
},
"mappings": {
"properties": {
"timestamp": {
"type": "date"
}
}
}
}
}
  1. Create data stream

In this step, you create the data stream that handles the time series data. The data stream provides a single, unified write target for ingesting data while managing multiple backing indices behind the scenes.

PUT _data_stream/webserver-logs-data-stream
  1. Validate ISM policy mapping

Verify that the ISM policy is correctly associated with the data stream that was created in the earlier step. This validation step confirms that automatic lifecycle management works as expected.

GET _plugins/_ism/explain/webserver-logs-data-stream

Output

Confirm that policy_id matches the ISM policy name you created earlier and that enabled is set to true.

OpenSearch explain output showing the ISM policy_id mapped to the data stream with enabled set to true

Section 2: Ingesting data to data stream

In real-world scenarios, log data is typically collected and streamed directly to Amazon OpenSearch Service data streams. However, to demonstrate rollover and migration scenarios in this post, we take a different approach. We first load sample log data into a standard OpenSearch Service index, then reindex and migrate that data to a data stream.

To get started, run the commands in the Dev Tools console to create an index and populate it with sample log data.

  1. Reindex existing data

This step shows how to migrate existing data in a traditional index to the new data stream. The reindex operation includes a script that confirms each document has a valid timestamp field (timestamp). Documents without a timestamp field are skipped by the reindex operation to maintain data integrity.

POST _reindex
{
"source": {
"index": "webserver-logs"
},
"dest": {
"index": "webserver-logs-data-stream",
"op_type": "create"
},
"script": {
"source": """
try {
// Validate timestamp field exists and has a value
if (ctx._source.timestamp == null || ctx._source.timestamp.empty) {
ctx.op = 'noop';
}
} catch (Exception e) {
// Skip this document on any error
ctx.op = 'noop';
}
"""
},
"conflicts": "proceed"
}
  1. Monitor index rollover

As shown in Figure 2, after reindexing, monitor the creation of backing indices. The ISM policy evaluates indices every 5 minutes by default. Rollover occurs based on the defined conditions (1,000 documents or 2 minutes of age). Verify this using the following command.

GET _cat/indices/.ds-*?v&h=index,status,health,pri,rep,docs.count,store.size,creation.date&s=index

The output should look like the following:

_cat/indices output listing the .ds backing indices with their status, health, and document counts

You can also validate this from OpenSearch Dashboards by navigating to Index Management, Data streams, webserver-logs-data-stream.

OpenSearch Dashboards Index Management page showing the webserver-logs-data-stream and its backing indices

*Figure 2 — Combined view of the _cat/indices CLI output and the OpenSearch Dashboards data stream details, showing a successful index rollover across multiple backing indices*

  1. Validate warm transition

Verify that indices are correctly transitioning from hot to warm storage based on the ISM policy conditions. You can monitor this through OpenSearch Dashboards or API queries in the Dev Tools console.

GET _plugins/_ism/explain/webserver-logs-data-stream

OpenSearch explain output showing backing indices transitioning from the hot state to the warm state

  1. Verify ingested data

Run a search against the data stream to confirm your documents were successfully indexed:

GET webserver-logs-data-stream/_search
{
"size": 1
}

Output

You should see your ingested documents returned in the hits.hits array, with the timestamp field and other fields you defined in the index template. A non-zero hits.total.value confirms data is flowing correctly through the data stream.

Search results showing an ingested document with the @timestamp field and a non-zero hits.total.value

  1. Clean up

If needed, these commands remove the data stream and its template.

DELETE _data_stream/webserver-logs-data-stream
DELETE _index_template/ webserver-logs-data-stream-template

Delete the sample data.

Conclusion

OpenSearch data streams with ISM offer capabilities for managing time series data at scale. Organizations that implement this approach can see improved query performance through distributed load and smaller, time-based backing indices that support efficient time-range queries. Automated index management reduces operational overhead. Storage tiering automatically moves aged data to UltraWarm storage, which significantly lowers costs without sacrificing access to historical data. Combined with better scalability for growing datasets, this solution simplifies index management while delivering improved performance and a more cost-effective, maintainable infrastructure.


About the authors

Praveen Krishnamoorthy Ravikumar

Praveen Krishnamoorthy Ravikumar

Praveen is an Analytics Specialist Solutions Architect at AWS. He helps customers design and implement modern data and analytics platforms that leverage the scalability, flexibility, and innovation of the cloud. He is passionate about solving complex data challenges and enabling organizations to unlock actionable insights from their data.

JP Boreddy

JP Boreddy

JP is a Senior Solutions Architect at Amazon Web Services, based in San Diego, California. He works with ISV customers in the security segment, helping them architect and optimize their workloads on AWS. JP specializes in AI/ML, containers, and cloud infrastructure, with a focus on enabling customers to build scalable, cost-effective solutions. He has been with AWS for over four years.

Aswin Vasudevan

Aswin Vasudevan

Aswin is a Senior Solutions Architect for Security, ISV at AWS. He is a big fan of generative AI and serverless architecture and enjoys collaborating and working with customers to build solutions that drive business value.

Kevin Fallis

Kevin is seasoned leader, architect, and developer with experience across many industry verticals and disciplines such as agriculture, ad tech, financial services, networking, security, telecommunications and of course search technologies. His passion helps others leverage the correct mix of AWS services and open-source solutions to achieve success for their business goals. His after-work activities include family, DIY projects, carpentry, horses, playing drums, and all things music.

How Alight Solutions achieved 55% cost savings with Amazon OpenSearch Service

Post Syndicated from Mark Larson original https://aws.amazon.com/blogs/big-data/how-alight-solutions-achieved-55-cost-savings-with-amazon-opensearch-service/

This is a guest post by Mark Larson, Andrew Kummerow, and Tim Razik at Alight Solutions, in partnership with AWS.

Alight Solutions is a leading cloud-based human capital technology and services provider focused on integrated benefits administration, healthcare navigation, and employee experience solutions. The company serves hundreds of enterprise customers globally, with services that support millions of people worldwide.

Alight’s technology stack generates over 1 billion log records per day across their containerized microservices architecture, with peaks reaching 100,000 records per second during Annual Enrollment periods. Previously, Alight relied on a self-managed Elastic Stack (Elasticsearch, Logstash, Kibana) deployment that had been in production since 2018. As their logging volumes grew and Elasticsearch 7.x approached end of support, the operational burden of maintaining this infrastructure consumed their entire operational budget, leaving no capacity for innovation.

In this post, we share how Alight Solutions migrated from self-managed Elasticsearch to Amazon OpenSearch Service. The migration achieved a 55% cost reduction, alleviated approximately 2,000 hours per year of operational overhead, and gave Alight access to advanced observability features they could not prioritize before.

Challenges with self-managed Elasticsearch

Alight’s self-managed Elastic Stack infrastructure presented compounding technical and operational challenges. Their production environment consisted of 15 Elasticsearch nodes with 168 TB of EBS storage, handling log ingestion from their flagship Alight Worklife system and supporting applications. The infrastructure required an Elastic Platinum subscription, though the team’s operational bandwidth was fully consumed by maintenance, leaving limited capacity to adopt advanced features included in the license.

The operational pain points included:

  • Security vulnerability patching required working over Christmas holidays to address critical fixes, with no flexibility on timing.
  • Elastic upgrades were time-consuming and required depth of knowledge to manage at scale.
  • Logstash using TCP-socket shipping was unreliable, experiencing log loss at high volumes.
  • Backpressure from Logstash caused two P1 incidents over two years, where the logging subsystem directly impacted microservice tasks.
  • Elasticsearch 7.x approaching end of support created urgency to act before the next Annual Enrollment period (September through January).

Alight was spending more than $100,000 per month on self-managed infrastructure and Elastic licensing across all environments. All operational budget was consumed by cluster maintenance, leaving zero capacity for innovation.

Evaluating alternatives

Alight evaluated several alternatives before selecting OpenSearch Service:

  • New Relic and Dynatrace were evaluated for log aggregation but proved prohibitively expensive at Alight’s volume.
  • Amazon CloudWatch was evaluated but did not meet requirements for complex log research at their volume and visualization complexity.

Amazon OpenSearch Service is a managed service that makes it straightforward to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. You can use it for use cases such as log analytics and real-time application monitoring. It provisions cluster resources, automatically detects and replaces failed nodes, and scales with a single API call or a few clicks, reducing the operational overhead associated with self-managed infrastructure. It won the evaluation based on five factors:

  1. Cost: significantly cheaper than self-managed Elastic Stack and competing solutions.
  2. Minimal change management: as a fork of Elasticsearch 7.10, engineers were already familiar with the query syntax and dashboards.
  3. Compliance: using a native AWS service avoided hundreds of hours of vendor compliance, audit, and regulatory work. The team spent a few hours getting approval compared to potentially weeks for an external vendor.
  4. Cloud-native strategy: aligned with Alight’s overarching strategy to use cloud-native services.
  5. Security and data privacy: keeping everything within their AWS landing zone alleviated data egress concerns.

Solution overview

Alight partnered with AWS to design a cloud-native log aggregation architecture that replaced self-managed Elasticsearch and Logstash with Amazon OpenSearch Service and Amazon OpenSearch Ingestion (OSIS), alleviating the operational burden, including the Logstash backpressure that had caused two P1 incidents.

The architecture uses a cross-account model with two primary account types:

The following diagram illustrates the solution architecture.

Cross-account architecture showing Amazon ECS and Amazon EC2 workloads sending logs through OpenSearch Ingestion to Amazon OpenSearch Service

Alight OpenSearch Service architecture showing cross-account log ingestion from Amazon ECS and Amazon EC2 workloads through OpenSearch Ingestion to Amazon OpenSearch Service

Ingestion paths

The solution supports multiple ingestion paths depending on the application hosting model:

  • ECS applications: FireLens/Fluent Bit sidecar containers capture stdout/stderr through the awsfirelens log driver, then ship logs over HTTPS directly to OSIS in the shared services account. ECS task roles assume a cross-account OSIS Ingest Role for authentication.
  • EC2 applications: Open-source Fluent Bit (RPM-based, non-containerized) uses tail input to read log files, then ships to OSIS through an EC2 IAM Role with cross-account trust.
  • S3-based ingestion (planned): Some applications write to Amazon Simple Storage Service (Amazon S3) with Amazon Simple Queue Service (Amazon SQS) notifications triggering OSIS pipelines.

Spring Boot microservices use a custom logging framework built on Logback (not Log4j) that formats logs as JSON and flushes to console, which FireLens picks up.

Security model

Traffic flows over HTTPS. The security model uses role separation with least privilege:

  • OSIS Ingest Role: write-only access to OSIS pipelines, assumed by application account roles via cross-account trust.
  • OSIS Sink Role: used by OSIS to write into the OpenSearch domain, with full index access scoped to the ingestion pipeline.
  • Security groups: restrict OSIS traffic to known CIDRs and VPCs.

Each application has its own indices, and access is governed by application-specific roles.

Persistent buffering

Amazon Elastic File System (Amazon EFS) provides persistent filesystem buffering for the Fluent Bit sidecar, helping prevent log loss during transient failures or backpressure events. This directly addresses the P1 incidents Alight experienced with Logstash. For the next Annual Enrollment period, Alight plans to also enable persistent buffering at the OSIS layer to handle burst ingestion without log loss.

User access

End-user access to OpenSearch Dashboards is managed through AWS IAM Identity Center with System for Cross-domain Identity Management (SCIM) synchronization from Alight’s enterprise Identity Provider. Users navigate to the Applications tab in Identity Center to access OpenSearch Dashboards over SAML/HTTPS.

At Alight, IAM Identity Center and SCIM are configured in the payer account. They use the same synchronization and entitlement request and approval process that governs Alight’s user and entitlement provisioning into AWS. With this setup, the team uses the same single sign-on (SSO) and entitlement workflow for OpenSearch Dashboards access as for the AWS Management Console, in conjunction with fine-grained access control (FGAC) defined within the OpenSearch domains.

OpenSearch domain configuration

For their production workload, Alight deployed:

Component Configuration
Data nodes 18 im4gn.2xlarge.search
UltraWarm nodes 9
Dedicated leader nodes 3
Hot tier storage 25 TB
UltraWarm storage 180 TB
Primary logical data 80 TB
Total with replicas 100-105 TB

Additional environments include a secondary production cluster (12 hot nodes, 3 UltraWarm, 3 dedicated leader nodes), plus client test and engineering clusters with 3 hot nodes each.

Migration process

The migration was completed over seven months (February through August 2025), with five applications migrated including the flagship Alight Worklife application.

Infrastructure as code

The team built new Terraform modules to manage deployment of OSIS pipelines, OpenSearch domains, and FireLens sidecar additions to ECS applications. Onboarding new applications is now templatized, resulting in significant time savings compared to adding new indices in Elasticsearch. Onboarding a new application now takes between 4-8 hours, whereas before we would spend 80-120 hours per application.

Migration timeline

Alight first enabled Amazon OpenSearch Service in production for two smaller applications, to make sure operational processes were up and running before migrating the highest volume log producers. For each application, logging to OpenSearch was enabled while continuing to write logs to the existing logging infrastructure. This parallel run allowed fine-tuning of OSIS pipeline configuration, OpenSearch cluster size and configuration before doing a full cutover. This approach also validated that logs were being ingested properly into OpenSearch. It confirmed that the performance of OpenSearch Dashboards and queries was as good as or better than the existing self-managed Elasticsearch cluster.

For historical data, Alight migrated the most recent 30 days of live data from Elasticsearch into OpenSearch just prior to cutover. They also retained a full archive of older log data in an Amazon S3 bucket, so that data older than 30 days could be loaded into OpenSearch on request if a user needs it.

AWS partnership

Alight engaged the AWS team during the evaluation phase. Through AWS Enterprise Support, their Technical Account Manager (TAM) served as the dedicated point of contact throughout the journey. The TAM coordinated sessions with OpenSearch Service subject matter experts to address specific service capabilities, help with design, troubleshoot issues, and provide performance guidance.

Results

The migration to Amazon OpenSearch Service delivered results across cost, operations, and capability dimensions.

“Alight’s mission critical applications are built on hundreds of interdependent microservices, so effective application logging is critical for analyzing system behaviors, performance tuning, and troubleshooting. Amazon OpenSearch Service provides us with great log analytics, very cost effectively at scale, and integrates seamlessly with our IAM strategy for granular access control and authorization. The ability to reconfigure, resize, and upgrade OpenSearch domains with a few clicks and zero downtime is a game changer for us.”

— Mark Larson, Enterprise Architect

Cost and licensing

Metric Before After Improvement
Monthly infrastructure + licensing cost Self-managed EC2/EBS + Elastic Platinum licensing Fully managed OpenSearch Service, no separate licensing ~55% cost reduction
Licensing model Elastic Platinum (fixed) Zero licensing cost No longer needed

Not all Elasticsearch clusters are decommissioned yet. Once decommissioning is complete, savings will reach approximately 65%. Additionally, more applications have been added to OpenSearch than were originally on Elasticsearch, making the per-application cost even more favorable. Beyond compute and licensing, the migration also reduced data transfer costs previously incurred across the self-managed cross-account architecture, adding further to the overall savings.

Operational improvements

Metric Before After
Engineering hours on cluster management 2,000 hours/year (≈1 FTE) Near zero (managed service)
Security vulnerability patching Manual, including holiday work Handled by AWS
Application onboarding Manual index creation and configuration Templatized via Terraform
P1 incidents from logging subsystem 2 in past 2 years Zero since migration

Performance and scale

Metric Value
Daily log volume 1 billion records
Peak ingestion rate 100,000 records/second
Applications migrated 5 (including Alight Worklife)
Total data under management 100–105 TB with replicas

Lessons learned and best practices

Through their migration journey, Alight gained the following insights:

  • Use your account team relationship to advocate: When Fluent Bit had a blocking issue, the AWS account team relationship helped push for the fix and provided workaround guidance.
  • Separate concerns for data durability: Do not put 100% delivery guarantees on logging infrastructure. Use a separate event stream (such as Amazon SQS) for critical data that cannot tolerate loss.
  • Templatize everything: Terraform modules for OSIS, OpenSearch domains, and FireLens sidecars reduce the time to onboard new applications.
  • Security architecture matters: Separating ingest roles from sync roles (least privilege) and using cross-account trust provides strong security without complexity.
  • Plan around business-critical periods: Pausing the production rollout during Annual Enrollment was the right call. The risk of introducing changes during peak was not worth the schedule pressure.

What’s next

Alight has several initiatives planned to expand their OpenSearch Service usage:

  • Anomaly detection: top priority, a feature they paid for with Elastic Platinum but never had capacity to implement.
  • Amazon OpenSearch Serverless: evaluating for new log sources, particularly interested in zero-OCU baseline for cost optimization.
  • OSIS persistent buffer: planned for next Annual Enrollment to handle burst ingestion without log loss.
  • Amazon Bedrock AgentCore logging: new artificial intelligence (AI) workloads will send logs to OpenSearch.
  • AI-assisted log analytics: adopting the agentic AI capabilities now built into Amazon OpenSearch Service. These include the Investigation Agent for autonomous, hypothesis-driven root cause analysis, which helps site reliability engineering (SRE) and engineering teams gain deeper insights from application logs.
  • Vector database: already using OpenSearch as a vector store for a conversational AI assistant (separate team).
  • Migration progress: All workloads previously logging to Elasticsearch have been migrated to OpenSearch, plus an additional eight applications.
  • Enterprise Logging Service: All new applications will now log to Amazon OpenSearch Service by default using the templatized approach.
  • Decommission: All existing Elasticsearch instances will be decommissioned by July 2026.

Conclusion

Alight’s migration from self-managed Elasticsearch to Amazon OpenSearch Service demonstrates how enterprises can alleviate operational burden while achieving significant cost savings. By using Amazon OpenSearch Ingestion and FireLens, Alight built a scalable log aggregation system that handles 1 billion records per day with zero P1 incidents since deployment.

The 55% cost reduction and approximately 2,000 hours per year of recovered engineering time have freed Alight to pursue advanced observability capabilities like anomaly detection and AI-powered log analytics, features they paid for but could never use under the operational weight of self-managed infrastructure.

To learn more, see the Amazon OpenSearch Service documentation. To get started with ingestion pipelines, see Amazon OpenSearch Ingestion. For migration guidance, see Migrating to Amazon OpenSearch Service.


About the authors

Mark Larson

Mark is an Enterprise Architect at Alight. This team is responsible for translating business and product strategy into secure, scalable, and sustainable technology outcomes through clear architectural guidance, governance, and partnership with business and engineering leaders.

Andrew Kummerow

Andrew is the Head of Enterprise Architecture at Alight, where he leads the EA organization. This team is responsible for translating business and product strategy into secure, scalable, and sustainable technology outcomes through clear architectural guidance, governance, and partnership with business and engineering leaders.

Tim Razik

Tim is a Senior IT Application Architect at Alight with over 25 years of experience in Site Reliability Engineering (SRE) and DevSecOps. He specializes in building scalable, secure, and highly observable cloud platforms, with deep expertise in log and telemetry pipeline design using AWS services such as Amazon OpenSearch. Tim is currently leading observability efforts for AI platforms like Amazon Bedrock, working closely with engineering teams to improve system reliability, operational visibility, and production performance.

Puneeth Ranjan Komaragiri

Puneeth Ranjan Komaragiri

Puneeth is a Principal Technical Account Manager at AWS. He is particularly passionate about monitoring and observability, cloud financial management, and generative AI domains. In his current role, Puneeth enjoys collaborating closely with customers, using his expertise to help them design and architect their cloud workloads for optimal scale and resilience.

Praful Kava

Praful Kava

Praful is a Sr. Specialist Solutions Architect at AWS. He guides customers to design and engineer cloud-scale analytics pipelines on AWS. Outside work, he enjoys traveling with his family and exploring new hiking trails.

Jagadish Kumar (Jag)

Jagadish Kumar (Jag)

Jagadish is a Senior Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service. He is deeply passionate about data architecture and helps customers build analytics solutions at scale on AWS.

Migrate from Apache Solr to Amazon OpenSearch Serverless

Post Syndicated from Jon Handler original https://aws.amazon.com/blogs/big-data/migrate-from-apache-solr-to-amazon-opensearch-serverless/

If you’re running Apache Solr for search, now is a great time to migrate to Amazon OpenSearch Service. Amazon OpenSearch Serverless offers a modern, managed destination that greatly reduces operational overhead. Migration Assistant for Amazon OpenSearch Service now supports Apache Solr sources from versions 6.x through 9.x. Migration Assistant now includes an AI assistant that you can drive from your preferred AI tools. The assistant walks you through the migration, providing a detailed report with timelines, blockers, schema and query translation.

In this post, you will learn why now is the time to take advantage of the ease of operations and native AI capabilities of OpenSearch Serverless, and migrate from Solr.

The challenge of managing Solr

Many organizations have Solr deployments that have run for years, carrying accumulated technical debt: older versions, custom patches, and operational processes originating with engineers who have left. Running Solr in production requires ongoing investment in upgrades, security patches, monitoring, failure recovery, and capacity planning. The operational burden compounds as your deployment ages and the engineers who built it move on, leaving the remaining team with a system that is difficult to modify safely.

Meanwhile, search has evolved. Users interact through chat interfaces and AI agents that synthesize information on their behalf. These patterns require semantic understanding, hybrid retrieval, and agentic capabilities like memory and Model Context Protocol (MCP) support. OpenSearch is an open source software suite for search, analytics, and observability, licensed under the Apache License V2.0 and based on Apache Lucene. OpenSearch provides a broad and deep set of vector capabilities for AI workloads: multiple engines (Facebook AI Similarity Search (FAISS) and Lucene), multiple algorithms (Hierarchical Navigable Small World (HNSW) and Inverted File (IVF)), quantization for cost management, hybrid search with score normalization, neural sparse search, and a connector framework. You can use the OpenSearch connector framework to connect to your own models hosted in services such as Amazon Bedrock and Amazon SageMaker. With these capabilities, you can build chat-based search, power AI agents, and implement Retrieval Augmented Generation (RAG) workflows on your search engine.

Technical debt in custom patches and undocumented procedures can make shipping new features on Solr slow and risky. OpenSearch Serverless provides a modern engine with all the capabilities just mentioned, along with automatic scaling, and minimal infrastructure to maintain. Even if you remain focused on traditional search, OpenSearch Serverless delivers relevant results with less operational effort.

Amazon OpenSearch Serverless alleviates infrastructure provisioning, capacity planning, and data lifecycle management. You create a collection, send data, and run queries. OpenSearch Serverless automatically matches compute to your workload, independently scaling up indexing and search compute during spikes, and scaling compute to zero when idle. You pay only for storage when no requests are being processed. For variable traffic patterns, OpenSearch Serverless can cost up to 60% less than an OpenSearch Service domain provisioned for peak.

The vector engine supports HNSW and IVF with FAISS and Lucene, plus quantization techniques to manage cost as your collection grows. With Automatic Semantic Enrichment, you can enable a sparse model with a single setting to augment text with vectors and improve relevance without building an embedding pipeline. GPU acceleration reduces HNSW index build times from hours to minutes. OpenSearch Serverless also supports agents and RAG workflows.

If your workload requires tight control over infrastructure: specific instance types, custom plugin configurations, or extreme scale beyond what serverless deployments target, Amazon OpenSearch Service domains are the alternative destination. Domains give you full control over the cluster with one-click provisioning, patching, and backups. Migration Assistant supports both destinations, so you can pick the deployment model that fits your workload without changing your migration approach.

Migration Assistant for Amazon OpenSearch Service

Migration Assistant has helped customers move from self-managed Elasticsearch and OpenSearch to OpenSearch Service since December 2023. It now supports Solr sources. Migration Assistant now includes an AI-assisted experience that you can drive from your preferred AI tools, like Kiro, Claude Code, and others, to plan a migration, deploy the necessary infrastructure, and execute both historical and live traffic migration.

Historically, migrations required weeks of planning and assessments before any data movement could begin, and the process was often error-prone. The AI-assisted experience provides an agent-guided workflow that helps you structure, execute, and validate your migration faster and more reliably. You can use Migration Assistant itself to do your assessment, or connect with the skills in the AWS Model Context Protocol (MCP) server. Migration Assistant also now supports live traffic capture and replay for Solr, so you can validate the new environment with real workloads before cutting over.

Migration Assistant supports migrations to OpenSearch Serverless and OpenSearch Service domains from a range of Solr, Elasticsearch, and OpenSearch versions. It’s open source, so you can use it for migrating to OpenSearch Serverless in all commercial AWS Regions and AWS GovCloud (US) Regions where OpenSearch Serverless is available.

Migrate now

OpenSearch Serverless gives you vector, hybrid, and semantic search, automatic scaling, scale-to-zero compute, and zero operational overhead. Migration Assistant for Amazon OpenSearch Service provides the AI-assisted tools to get there: plan with the agent, deploy the infrastructure, run a backfill from your Solr backups, and validate with live traffic capture and replay before you cut over. Start by pointing your AI tool of choice to Migration Assistant to get a plan, timeline, and cost estimate.

For more information, see the Amazon OpenSearch Service documentation and the Migration Assistant for Amazon OpenSearch Service documentation.


About the author

Jon Handler

Jon Handler

Jon is a Senior Principal Solutions Architect for Search Services at Amazon Web Services. Jon works closely with OpenSearch and Amazon OpenSearch Service, providing help and guidance to a broad range of customers who have search and log analytics workloads. Prior to joining AWS, Jon’s career as a software developer included four years of coding a large-scale, eCommerce search engine.

Specification-driven composition for flexible data workflows

Post Syndicated from Rostislav Markov original https://aws.amazon.com/blogs/architecture/specification-driven-composition-for-flexible-data-workflows/

Specification-driven composition addresses a common scalability bottleneck in data pipelines. Data pipelines often start as simple scripts, but as they grow, you duplicate transformation logic and small changes cascade across multiple workflows. Copying and modifying data transformation logic across scripts leads to workflows that become difficult to manage at scale. Tracking what each pipeline does becomes harder because workflow intent is embedded in code. This lack of visibility complicates governance, especially in regulated environments such as healthcare, finance, and life sciences.

Many implementations combine orchestration, transformation logic, and validation rules in the same scripts. Supporting new datasets requires modifying and redeploying code, while validation often happens only during processing. As a result, issues surface late in the lifecycle which increases the operational risk. Specification-driven composition separates workflow intent from implementation so you can build flexible data workflows.

In this post, I show how to apply specification-driven composition to data transformation workflows. I explain the challenges with script-based pipelines, introduce the pattern and its core components, and walk through a serverless implementation using AWS Lambda, AWS Step Functions, Amazon Simple Storage Service (Amazon S3), and Amazon OpenSearch Service.

Solution overview

You can separate workflow intent from processing logic with specification-driven composition. This approach reduces duplication, shortens the time required to onboard new datasets, and improves consistency across workflows. Instead of embedding logic in scripts, the system describes workflow intent in a structured specification, validates the specification before processing, and dynamically assembles a processing pipeline.

This approach moves pipeline configuration outside application code and composes pipelines from reusable processing components. To separate concerns, it organizes a workflow into three layers, as shown in Figure 1. The intent layer defines workflow behavior using specifications. The composition layer validates specifications and assembles pipelines. The processing layer runs the pipeline of transformation steps.

A diagram showing the three layers of specification-driven composition. The intent layer holds the specification, the composition layer contains the composer and capability registry, and the processing layer runs the capability pipeline.

Figure 1. Specification-Driven Composition design pattern.

Benefits

Specification-driven composition provides several practical benefits when you manage multiple pipelines. First, it improves governance because specifications provide a clear, traceable description of workflow behavior that you can review and validate before invocation. In regulated industries, this traceability shortens audit preparation time and reduces the review burden for new dataset submissions.

Second, the pattern supports reusable transformations. You implement transformation logic once and reuse it across multiple workflows. This reduces duplication and improves consistency. In practice, teams adopting this pattern report being able to onboard new datasets in days rather than weeks because most required capabilities already exist in the registry.

Third, specification-driven composition enables flexible pipeline design. You define new specifications to create pipelines supporting new datasets and use cases without modifying application code and registering new system release.

Fourth, the pattern separates business intent from execution artifacts. The specification expresses what the workflow should do in domain terms, while the generated state machine remains a system artifact. This separation matters in regulated environments (for example, GxP) where business users author intent but are not allowed to author or modify execution code directly.

Finally, the declarative representation of workflows lets AI tools assist with capability discovery, specification authoring, and pipeline analysis, while runtime behavior stays predictable as it relies on validated capabilities.

Core components

You work with four key components to define, validate, and run workflows.

  1. Specification

A specification is a structured document, typically JSON or YAML, that describes datasets, mappings, and transformations. It defines what the workflow should do without including processing logic. Because specifications are explicit and versioned, they provide a clear record of workflow intent.

  1. Composer

The composer converts specifications into runnable steps, so workflows run consistently without embedding transformation logic in application code. It checks that referenced capabilities exist, retrieves metadata, and builds a workflow that can run on the processing layer. The composer does not perform transformations. It only assembles the workflow. In practice, the composer compiles business intent expressed in the specification into a code artifact such as an Amazon States Language (ASL) definition. This abstraction lets domain users author specifications without producing runnable code, which is important in environments with strict separation of duties.

  1. Capability registry

The registry stores metadata about reusable transformation functions. This includes identifiers, input/output formats, invocation details, and permission boundaries. The composer uses the registry to validate specifications and locate capabilities. Treat this registry as a governed artifact rather than a manually edited lookup table. Capability definitions live in version control, and your CI/CD pipeline validates metadata and runs tests before you publish new versions. Specification authors include explicit capability version references in the specifications to support reproducible workflow runs. In regulated systems, you must validate new capabilities and obtain approval through a separate workflow before you register them.

  1. Capability pipeline

Once assembled, the pipeline runs a sequence of transformation steps. Each step performs a specific operation such as formatting, validation, or enrichment. Because these steps are reusable, you can apply them across many workflows.

Technical implementation

Let’s walk through a technical implementation of this pattern using serverless AWS services. In this example, workflow specifications are uploaded to an S3 bucket. An AWS Lambda composer retrieves and validates the specification, looking up capability metadata in Amazon OpenSearch Service. The composer then assembles the workflow in AWS Step Functions, which orchestrates the workflow and invokes AWS Lambda capability processors. Each processor emits traces to Amazon CloudWatch Logs.

Figure 2 shows the architecture.

Users upload specifications to Amazon S3, which invokes the Lambda composer. The composer queries the OpenSearch registry and assembles a Step Functions workflow of Lambda capability processors, which emit traces to CloudWatch.

Figure 2. AWS implementation of Specification-Driven Composition

Interpreting the workflow specification

The workflow specification defines datasets and transformation logic using a structured format (in this example, JSON). A specification is a declarative document that describes what the pipeline should produce rather than how to produce it. Your composer reads the specification, validates it against a schema, and uses it to construct the pipeline.

The following example maps source fields to target fields using reusable capabilities (Figure 3). It takes data from a source dataset (‘raw_orders’), maps specific fields (‘order_date’, ‘amount’) and applies reusable transformation capabilities such as ‘format_date’ and ‘normalize_currency’. Each mapping explicitly links a source field to a target field and references a capability that performs the transformation. Source dataset(s) are listed under the ‘source’ section of the JSON document, target datasets under the ‘target’ section, and mappings ‘mappings’. You can define your own specification structure and build custom validation logic in your composer to make sure specifications are valid.

{
  "source": {
    "dataset": "raw_orders"
  },
  "target": {
    "dataset": "orders_clean"
  },
  "mappings": [
    {
      "source_field": "order_date",
      "target_field": "order_date",
      "transformation": {
        "capability": "format_date"
      }
    },
    {
      "source_field": "amount",
      "target_field": "amount_normalized",
      "transformation": {
        "capability": "normalize_currency"
      }
    }
  ]
}

You can model preprocessing steps such as column standardization, numeric casting, or unit conversion as first-class capabilities and reference them earlier in the specification (for example, in a dedicated ‘preprocessing’ section) before downstream mappings run. This way, preparation logic uses the same metadata, versioning, validation, and observability model as the rest of the workflow, which simplifies lineage and review.

In this example, an S3 event notification invokes the composer Lambda function when you upload a specification. In practice, the same composer can be invoked through several mechanisms depending on the use case: S3 events for new specifications, an Amazon EventBridge schedule for recurring runs, an API or UI action for on-demand invocation, a direct Step Functions StartExecution call, or an upstream pipeline. This flexibility lets you re-run an approved specification against refreshed data without re-authoring or re-approving the workflow.

The composer parses the specification and validates the referenced capabilities by querying Amazon OpenSearch Service to retrieve capability metadata such as Amazon Resource Names (ARNs). OpenSearch Service is used here because the composer does more than direct-key lookups. It supports capability discovery through full-text and semantic search over capability metadata such as capability descriptions, input/output schemas, and tags. This lets data authors and AI tools find reusable capabilities by intent rather than by exact identifier. After validation, the composer assembles and starts an AWS Step Functions state machine which invokes each capability in sequence. Each capability runs independently, making the pipeline modular and reusable.

Securing sensitive data flows

This pattern suits regulated workloads, so handle security as part of the design. Use SSE-KMS with a customer managed key on the specification and data S3 buckets and enable encryption at rest on the Amazon OpenSearch Service domain. Enforce HTTPS (TLS) access with an S3 bucket policy (aws:SecureTransport) and enable node-to-node encryption on Amazon OpenSearch Service. AWS Step Functions and Lambda calls use TLS by default. Each capability processor receives only the fields its mapping references, and you can apply IAM policy at the source bucket to restrict access. Processors emit traces to Amazon CloudWatch Logs.

For data classification, you can tag sensitive fields in the specification (example: "sensitivity": "PHI") and declare the sensitivity of each capability in the registry: a direct-move capability preserves the source classification, a date-of-birth-to-age capability clears it, and an enrichment that introduces sensitive data sets it. The composer combines the source tag with the capability’s behavior to derive the target field’s sensitivity, validating that the combination resolves clearly before assembling the pipeline. It then generates the masking artifact for the output (for example, AWS Lake Formation column grants) so consumers get correct masking without a separate masking specification or manual effort.

Recognizing the pattern

This approach works well when you describe workflows using structured specifications, reuse transformation logic across pipelines, require validation before invocation, and require workflows to be deterministic and auditable. You are likely to recognize the pattern in regulated data pipelines for the submission of tabular datasets to oversight agencies.

Consider clinical trial reporting as a concrete example. Data analysts collect raw data from clinical sites and must transform it into a standard submission format such as the Study Data Tabulation Model (SDTM) before submission to agencies such as the US Food and Drug Administration. With specification-driven composition, data analysts define specifications that map collected data about adverse events, demographics, and vital signs to the standard target variables, and the validated output feeds downstream systems such as patient safety and medical monitoring.

That said, this pattern might add unnecessary complexity for simple, one-time data transformations or pipelines with fewer than three to five workflows. Evaluate this pattern’s impact by tracking the reduction in duplicated transformation logic across pipelines, the time required to onboard new datasets, and the number of workflows you can create without modifying application code.

Conclusion

Script-based data pipelines accumulate hidden costs as they grow, including duplicated logic across files and late-breaking validation failures. Specification-driven composition separates workflow intent from processing so you can manage data workflows more consistently at scale. The result is faster dataset onboarding, stronger governance, and pipelines that are transparent enough to trust in regulated environments.

This pattern is especially valuable for regulated reporting pipelines, multi-source data integration, and reusable ETL frameworks where traceability and flexibility matter. By investing a small set of reusable capabilities and a disciplined specification format, you can reduce the engineering effort for new pipelines by treating them more as configuration tasks which may be delegated to system users.

Next steps

To get started, take one existing pipeline and describe it as a specification. Implement a small set of reusable transformation functions and use them to assemble your first composed workflow. The AWS Lambda event-driven architectures guide is a good starting point for wiring S3 uploads to your composer, and the AWS Step Functions and Lambda integration guide will help you orchestrate your capability processors. To monitor the pipeline, see publishing custom CloudWatch metrics.

For a practical first use case, try applying the pattern to a reporting pipeline you maintain today that has three or more variants such as monthly finance reports generated from different source systems. Replace the duplicated scripts with a single composer and a shared capability library, and measure the onboarding time for the next variant. As you expand your capability library, you can apply the same pattern across additional workflows and standardize how transformations are defined and run.


About the author

Cut costs and simplify operations with writable warm storage in Amazon OpenSearch Service

Post Syndicated from Bharav Patel original https://aws.amazon.com/blogs/big-data/cut-costs-and-simplify-operations-with-writable-warm-storage-in-amazon-opensearch-service/

Managing petabytes of search data means making tough choices: keep everything fast and expensive, or make it affordable but read-only. UltraWarm is a proven, cost-effective solution for read-heavy historical data. However, some workloads occasionally need to update historical records, such as late-arriving data or compliance corrections. With UltraWarm, you must migrate those indices back to hot, perform the update, and migrate back. What if you could write directly to your cost-effective warm storage instead?

In this post, I show you how writable warm storage removes the costly migration cycle. You can reduce your infrastructure costs by up to 48 percent and update historical data in seconds instead of hours. I walk through a real-world cost comparison and performance benchmarks, and help you decide when to use writable warm versus UltraWarm.

The challenge with tiered storage

Amazon OpenSearch Service handles data-intensive search and analytics workloads, from real-time log analytics and application monitoring to security event detection. As your data volumes grow from terabytes to petabytes, you face a fundamental question: how do you keep recent data fast while making earlier data affordable?

OpenSearch Service addresses this with a tiered storage architecture:

  • Hot – Highest performance for active indexing and search using instance-attached storage.
  • UltraWarm – Cost-effective, read-only tier backed by Amazon Simple Storage Service (Amazon S3) with local caching for less frequently queried data.
  • Cold – Fully detached from the cluster, with the lowest cost for rarely accessed data. Cold indices must be migrated back to UltraWarm or hot before any reads or writes can be performed.

For immutable log data, this model works well. However, a specific class of workloads hits its limitations when they occasionally need to write to earlier data, and read-only becomes a bottleneck.

Prerequisites

To use writable warm storage, you need the following:

  1. An Amazon OpenSearch Service domain running version 3.3 or later.
  2. OpenSearch Optimized (OI2) instance family support in your AWS Region.
  3. Workloads with a minimum 5-second refresh interval.
  4. Data nodes using the OpenSearch Optimized instance family (OR2 for hot, OI2 for warm).

Note: Writable warm doesn’t currently support the cold storage tier.

The UltraWarm bottleneck

With UltraWarm, updating even a single document requires migrating the index back to hot, performing the write, and migrating it back. This round trip involves a force merge (consolidating index segments), snapshot creation, and shard relocation. These operations consume significant CPU, memory, and disk space on your hot nodes, and they take approximately 130 minutes per 100 GB index. This time was measured on a domain with 3 × r6g.2xlarge hot nodes, 3 × ultrawarm1.large warm nodes, and 3 dedicated leader nodes (US East, N. Virginia), using a single-shard index with one replica. Actual times vary based on domain configuration, shard count, segment count, hot node utilization, and migration queue depth. The result is that you over-provision hot nodes, build complex pipelines, or keep data in hot longer than necessary, which increases cost and complexity.

Introducing writable warm storage

OpenSearch Service now offers writable warm nodes that use OpenSearch Optimized (OI2) instances, the same instance family that powers durable, Amazon S3-backed storage on hot nodes. Because data is already persisted on Amazon S3, tier transitions become a lightweight shard relocation rather than a resource-intensive migration. The Lucene engine, which is OpenSearch’s underlying search library, operates identically on both tiers. As a result, writable warm nodes support active writes, background merges, and periodic refreshes, just like hot nodes.

Late-arriving data, compliance backfills, and corrections that previously required a warm-to-hot-to-warm round trip now resolve with a direct write in seconds. There is no force merge, no snapshot, no shard relocation, and no hot node resource consumption.

Diagram comparing UltraWarm and writable warm data flows. In the UltraWarm legacy flow, data is ingested into the hot tier, migrated to read-only UltraWarm, and any update requires a round trip back to hot. In the writable warm flow, indices transition from hot to writable warm, which accepts reads and writes directly without migrating back to hot.

UltraWarm (legacy) data flow: Data is ingested into the hot tier (SSD, read and write). Index State Management (ISM) policies migrate indices to UltraWarm (Amazon S3-backed, read-only). Any update requires migrating the index back to hot (dashed arrow), writing, then migrating back.

Writable warm (new) data flow: Same ingestion path through hot, with ISM transitioning indices to writable warm. The key difference is that writable warm supports both reads and writes. Late-arriving updates go directly to warm, with no migration back to hot. Because both tiers use Amazon S3 as durable storage through OpenSearch Optimized instances, transitions are lightweight shard relocations, not resource-intensive migrations.

The benefits: cost, operations, and flexibility

Writable warm delivers advantages in three areas: cost, operational simplicity, and flexibility.

Cost

Unlike UltraWarm, which only offers on-demand pricing, OI2 instances support Reserved Instance (RI) pricing, a commitment-based discount model. By committing to a 1-year or 3-year Reserved Instance, you can save 31–52 percent compared to UltraWarm nodes. This makes writable warm significantly more cost-effective for predictable, long-running workloads. The newly introduced Database savings plan for OpenSearch Service provides savings of around 22 percent over UltraWarm instances. Both tiers use Amazon S3 for durable storage, so node failure means only temporary unavailability, not data loss. For cost-sensitive workloads that can tolerate brief downtime during node recovery, you can configure zero replicas on warm indices to reduce costs further.

Real-world cost comparison

Consider a workload ingesting 2 TB/day with 210 days total retention, where updates can arrive at any point. With UltraWarm’s read-only constraint, you must keep data in hot for 30 days before migrating to warm. With writable warm, updates happen directly on warm, so hot retention drops to only 7 days.

At small scale, the hot tier reduction benefit is modest. Writable warm is still cost-effective if you need write capability on warm data, can commit to RI pricing, or value the operational simplicity of eliminating migration pipelines. For purely immutable data with short retention, UltraWarm on-demand might still be cheaper. Use the AWS Pricing Calculator to model your specific scenario.

The following table shows estimated monthly costs using on-demand and All Upfront Reserved Instance (AURI) pricing in the US East (N. Virginia) Region as of March 2026. For the latest pricing, see Amazon OpenSearch Service pricing on the AWS website.

Component Hot + UltraWarm (30d hot / 180d warm) Hot + writable warm (7d hot / 203d warm)
Hot data nodes $12,264 (21 × or2.2xlarge) $12,264 (21 × or2.2xlarge)
Hot EBS cost $10,212.84 (21 * 3986 GB) $2,636
Hot remote storage $2,008.28 $518
Warm data nodes $39,128 (20× ultrawarm1.large) $50,409 (15× oi2.8xlarge)
Amazon S3 storage $9,504 $1,070
Leader nodes $1,307 (3 × m8g.2xlarge) $1,307 (3 × m8g.2xlarge)
On-demand total $74,427 $69,297
1-year AURI $69,674 $43,918 (~36% less)
3-year AURI $67,367 $34,939 (~48% less)
Database savings plan $71,708 $55,406 (~22%)

Operations

Reclaim hot node capacity. Writable warm removes two common causes of hot node over-provisioning: reserving 35 percent of disk space for force merge operations, and maintaining extra capacity to temporarily move data back to hot for writes. You can run your hot tier at higher utilization, which reduces the number of hot nodes you need.

Simpler migrations. UltraWarm migrations are multi-step operations (force merge, snapshot, and shard relocation) that need careful scheduling during low-traffic windows, and they are limited to 10 queued at a time. Writable warm simplifies this to a lightweight shard relocation, with more straightforward ISM policies and no scheduling constraints.

Flexibility

UltraWarm offers only two instance sizes: ultrawarm1.medium (1.5 TiB) and ultrawarm1.large (20 TiB). Writable warm with OI2 instances offers a full range from oi2.large to oi2.16xlarge. Each size addresses up to 5× its local cache size, so you can right-size warm capacity precisely to your workload.

Search performance

We benchmarked search latency using the NYC Taxis workload, comparing writable warm (oi2.large) against UltraWarm nodes. All measurements are P90 latencies.

On the NYC_TAXIS benchmark, writable warm matched or beat UltraWarm on 6 of 7 query types at P90, including lightweight filters, ranges, sorts, and time-histogram aggregations. For most real-world search patterns, writable warm delivers comparable or better performance than UltraWarm, plus the ability to write directly to the tier.

Search performance: writable warm compared to UltraWarm

Task Writable warm node latency in ms UltraWarm latency in ms UltraWarm vs. writable warm diff %
NYC_TAXIS workload type ** ** ** ** ** **
default (P90) 21.287 23.857 12.07223
range (P90) 21.23 21.016 -1.00718
distance_amount_agg (P90) 5,069 3929.23 -22.48406
autohisto_agg (P90) 21.076 22.002 4.39348
date_histogram_agg (P90) 21.363 21.792 2.01031
desc_sort_tip_amount (P90) 23.224 23.797 2.46636
asc_sort_tip_amount (P90) 22.483 22.482 -0.00445

When to choose what

Should you switch from UltraWarm to writable warm? It depends on your workload.

Requirement Writable Warm UltraWarm
Write enabled ✓ Read-only
Reserved Instance pricing ✓ ✗
Instance size flexibility Wide range (large–8xlarge) 2 options only
Cold tier support ✗ ✓
Need for OpenSearch Optimized instance families ✗ ✓
Concurrent tier transitions ✓ ✗ (sequential)
Hot node impact during migration Minimal High (CPU/memory)

Clean up resources

If you created a test domain to evaluate writable warm storage, delete it to avoid ongoing charges. In the OpenSearch Service console, select your domain and choose Delete. This removes all nodes and stops Amazon S3 storage charges for that domain.

Summary

In this post, I showed you how writable warm storage eliminates the costly migration cycle that UltraWarm’s read-only limitation creates. You get up to 36 percent cost savings with 1-year Reserved Instances, faster search performance, and a simpler operational model. Writable warm also removes data transitions between tiers, and Reserved Instance pricing becomes available for warm storage for the first time.

Writable warm requires OpenSearch Service version 3.3 or later with OI2 instances. For domains needing cold tier support, earlier OpenSearch Service versions, or non-optimized instance families, UltraWarm remains the right choice.

Next steps: Start by analyzing your current hot and warm split. How many days of data do you keep in hot only to accommodate occasional updates? Use the AWS Pricing Calculator to model your potential savings, and enable writable warm on a test domain in minutes. At the time of this post, writable warm is supported on OpenSearch Service version 3.3. For step-by-step instructions, see Migrating to writable warm storage in the OpenSearch Service documentation.

Have you tried writable warm storage? I’d love to hear about your experience and any questions you have in the comments.


About the author

Bharav Patel

Bharav Patel

Bharav is a Specialist Solution Architect, Analytics at Amazon Web Services. He primarily works on Amazon OpenSearch Service and helps customers with key concepts and design principles of running OpenSearch workloads on the cloud. Bharav likes to explore new places and try out different cuisines.

AWS Weekly Roundup: Claude Sonnet 5 on AWS, Amazon WorkSpaces for AI agents, AWS service availability updates, and more (July 6, 2026)

Post Syndicated from Daniel Abib original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-sonnet-5-on-aws-amazon-workspaces-for-ai-agents-aws-service-availability-updates-and-more-july-6-2026/

A couple of editions ago I wrote about what I find so energizing about working with startups. Last week I got a fresh dose of it: I spent a few days with the AWS Startups team, listening to stories of founders talking about the problems they’re actually solving. One story that stayed with me came from Marco Negreiros, founder of EyeCare Health, a Brazilian healthtech expanding access to eye care. He shared a striking fact: more than 70% of Brazilian municipalities don’t have a single ophthalmologist. His answer was to put a vision test on the one device almost everyone already carries, the smartphone, so a basic eye screening no longer depends on living near a clinic. Watching a founder turn a gap that big into something that concrete is exactly why I love this space.

AWS Startups team get-together with founders in Brazil

This week, I’ll take a closer look at some key launches, and then cover the quarterly AWS Service Availability updates.

Last week’s launches
Here are some of the launches covered from this past week in the AWS News Blog:

Here are some launches and updates that caught my attention:

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

AWS Service Availability Updates
When the availability of an AWS service or feature changes, we provide customers guidance in AWS Product Lifecycle Changes on available alternatives and support for migration so that disruptions to your operations are minimized. The following lifecycle changes were updated on June 30, 2026.

Services moving to Maintenance (no longer accessible to new customers starting July 30, 2026):

Services entering Sunset:

Services reaching End of Support (as of June 30, 2026):

  • Amazon Chime SDK – Carrier Voice Focus
  • Amazon SageMaker AI – Ground Truth Plus

We understand that changes in availability can impact your operations. For specific guidance, consult the relevant service documentation or contact AWS Support.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Summits – AWS Summits are free events that bring the cloud and AI community together to connect, learn, and explore the latest technologies. Browse the full calendar to find a Summit near you in the second half of 2026.
  • AWS Community Days – Community-led conferences where content is planned, sourced, and delivered by community leaders. If you’re in Latin America, don’t miss AWS Community Day Belo Horizonte on August 22. Registration is open at awscommunityday.com.br.

Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

– Daniel Abib

This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!

Run log analytics for a fraction of the cost with the new engine for Amazon OpenSearch Service

Post Syndicated from Jagadish Kumar original https://aws.amazon.com/blogs/big-data/run-log-analytics-for-a-fraction-of-the-cost-with-the-new-engine-for-amazon-opensearch-service/

Amazon OpenSearch Service is a real-time retrieval engine for AI, search, and analytics at any scale. As log volumes grow 30–40 percent year over year, organizations face rising infrastructure costs and slower analytical queries across their observability data. Teams are forced to choose between retaining the data they need and staying within budget.

We’re introducing a purpose-built log analytics engine for Amazon OpenSearch Service. This new engine delivers up to 4x price performance, 2x faster data ingestion, up to 2x faster analytical queries, and up to 70 percent lower storage costs. You get all of this without sacrificing search capabilities on the same data.

In this post, you learn how to take advantage of these benefits, see how to get started, and review benchmark results at billion-document scale.

How the optimized engine works

The optimized engine is a new engine mode within the same Amazon OpenSearch Service domain. You use the same console, APIs, security model, and networking configuration that you already use with the general-purpose engine.

OpenSearch Service stores all data in Apache Parquet format. For fields configured as searchable, OpenSearch Service also writes the data to the inverted index. Apache Calcite parses and optimizes each query, then routes operations to the engine best suited to execute them: Apache DataFusion for analytical operations on columnar data, or Lucene for search predicates. The two hand off mid-query, so a single query can search log content and aggregate the results without additional roundtrips.

You ingest data through the same REST APIs and client libraries you use today and you don’t need to change your agents or pipelines. The optimized engine supports two query languages: Piped Processing Language (PPL) and SQL. Both execute natively through the vectorized engine. The Domain Specific Language (DSL) query API is not supported on the optimized engine at launch.

Getting started

At launch, the optimized engine is a domain-level setting selected at creation time. You can’t add the optimized engine to an existing domain or enable it on individual indices or fields within a general-purpose domain. To adopt the optimized engine, create a new domain and migrate your ingestion pipelines to it.

Create a new domain in the Amazon OpenSearch Service console and select Observability as your use case. The optimized engine is enabled by default. The console provides a side-by-side comparison of capabilities to help you choose.

Amazon OpenSearch Service console showing the Observability use case selected with a side-by-side comparison of engine capabilities

After your domain is ready, ingest JSON documents through the same Bulk API and client libraries you use today. No changes to your ingestion pipelines or application code are required.

Benefits of the optimized engine for log analytics

The optimized engine for log analytics introduces the following performance and cost improvements:

  • Up to 4x better price-performance compared to the existing general-purpose engine on internal benchmarks, while retaining full-text search for incident investigation.
  • Up to 2x faster analytical queries. The engine uses a vectorized query execution path that processes data in columnar batches for fast results across large datasets.
  • Up to 2x higher ingestion throughput. The append-only columnar write path increases sustained ingestion rates.
  • Up to 70 percent lower storage with columnar storage for aggregation workloads. You can retain up to 3x more data at the same cost.

To demonstrate these improvements, we benchmarked observability workloads at billion-document scale. In the following sections, we explore the benchmark methodology, test environment, and results. We recommend testing the optimized engine with your own workload to validate the gains for your use case.

Benchmark methodology

We used the Telemetry Generator for OpenTelemetry to generate synthetic traces and logs at scale, producing three observability datasets: OTEL traces, OTEL logs, and web server access logs. We stored the generated data as bulk-format NDJSON in Amazon Simple Storage Service (Amazon S3). We then ingested it through a pipeline on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate. The pipeline reads chunks from Amazon S3, transforms timestamps, and writes to the OpenSearch Bulk API, simulating a production observability flow.

We benchmarked on two OpenSearch Service domains running OpenSearch 3.5, each with 9 data nodes in a 3-Availability Zone configuration:

Configuration Optimized Engine Standard Lucene
Instance type 9x or2.4xlarge.search 9x r8g.4xlarge.search
Leader nodes 3x m7g.large.search 3x m7g.large.search
EBS 2,500 GB gp3, 7,500 IOPS, 500 MB/s per node 2,500 GB gp3, 7,500 IOPS, 500 MB/s per node
Engine mode OPTIMIZED General Purpose (best_compression)

We ingested three data sets totaling 24.4 billion documents and 9.5 TB of raw JSON. All indices used 9 primary shards, 1 replica, and Index State Management (ISM)-managed rollover at 50 GB per primary shard. The Lucene baseline used best_compression (zstd) codec with _source enabled, representing the default customer configuration.

The ingestion pipeline ran on 90 Fargate tasks (16 vCPU, 120 GB RAM each, 48 writer threads per task, bulk size of 3,000 documents) in the same virtual private cloud (VPC) as the OpenSearch Service domains.

Results

Ingestion throughput

The optimized engine’s append-only columnar storage writes segments in bulk-optimized batches without per-document stored field overhead.

Metric Optimized Engine Lucene Baseline
Peak throughput 1.78M docs/sec ~647K docs/sec
Cluster CPU at peak 62% 72%
Write rejections 0 0
Total documents ingested 24.4 billion 15.7 billion

The optimized engine sustained 1.78 million documents per second at matched concurrency, approximately 2x the throughput of the Lucene baseline, while consuming less CPU. Both domains ran with zero write rejections. For teams ingesting terabytes per day, the throughput advantage translates to fewer nodes for the same volume, or longer retention on the same infrastructure.

Storage compression

The columnar Parquet format compresses observability data through dictionary encoding of repeated fields, tight packing of numeric columns, and elimination of per-document JSON overhead.

Measured across 24.4 billion documents:

Dataset Documents Source Optimized Engine Lucene (default)

Compression

vs.

source

Savings vs. Lucene
Web logs 8.76B 2,360 GB 254 GB 614 GB 89% 59%
OTEL logs 8.20B 3,720 GB 815 GB 1,549 GB 78% 47%
OTEL traces 7.43B 4,131 GB 841 GB 1,790 GB 80% 53%
Total 24.4B 9,539 GB 1,910 GB 3,953 GB 80% 52%

The optimized engine stores the same data at 5x compression versus raw JSON (80 percent savings). Against the default Lucene configuration (_source enabled, what most domains run), the optimized engine uses roughly half the storage. The optimized engine derives _source from Parquet columns on read, eliminating the need to store the raw JSON blob while still allowing document retrieval.

Analytical query performance

We measured query latency on a typical observability dashboard pattern: analytical aggregations scoped to a 15-minute time window over billions of log events. The optimized engine uses row-group pruning on the @timestamp column to skip data outside the query window, reading only the relevant subset.

Query pattern Dataset Optimized Engine Lucene baseline Speedup
Error count by service OTEL logs 717 ms 2.8 s 3.9x
Log volume by host OTEL logs 252 ms 17.6 s 70x
5xx errors by service and method OTEL logs 171 ms 885 ms 5.2x
Top services by error OTEL traces 635 ms 569 ms ~1x
Point lookup (single traceId) OTEL traces 394 ms 783 ms 2x

All queries scoped to a 15-minute window. Index sizes: 8.2 billion OTEL log events, 7.4 billion OTEL trace spans.

The optimized engine completes time-filtered analytical queries in 171 ms to 717 ms over billions of documents. The advantage is most pronounced on unfiltered aggregations (log volume by host: 70x) where the columnar engine reads only the columns needed. On queries where the Lucene inverted index provides strong predicate selectivity (top services by error on traces), performance is comparable between the two engines.

Search and point lookups

The optimized engine retains the Lucene inverted index alongside columnar storage. When the query planner recognizes a selective lookup (such as retrieving a single trace by ID), the planner routes the query to the inverted index rather than scanning columnar data. In our benchmark, a single traceId lookup across 7.4 billion spans returned in 165 ms.

This means a real investigation can use both engines in sequence: broad aggregations to localize the problem, then a point lookup to pull the offending trace, all from the same domain.

Now available

The optimized engine for Amazon OpenSearch Service is generally available today in all commercial AWS Regions (Regions other than the AWS GovCloud (US) Regions and the China Regions) where OpenSearch Optimized Instances are available.

Pricing follows standard Amazon OpenSearch Service rates for instances and storage, with no additional premium for the optimized engine. For more information, see Amazon OpenSearch Service Pricing.

To learn more about configuring and using the optimized engine, see Optimized for Log Analytics in the Amazon OpenSearch Service documentation. For an overview of the service, visit Amazon OpenSearch Service Log Analytics.

Give it a try and send feedback to AWS re:Post for Amazon OpenSearch Service or through your usual AWS Support contacts.


About the authors

Jagadish Kumar

Jagadish Kumar

Jagadish is a Senior Solutions Architect at Amazon Web Services, focused on OpenSearch and analytics workloads.

Rohin Bhargava

Rohin Bhargava

Rohin is a Senior Product Manager for Amazon OpenSearch Service.

Michael Supangkat

Michael Supangkat

Michael is a Solutions Architect at Amazon Web Services specializing in search and observability.

Implement multi-tenant search with Amazon OpenSearch Serverless next generation

Post Syndicated from Jon Handler original https://aws.amazon.com/blogs/big-data/implement-multi-tenant-search-with-amazon-opensearch-serverless-next-generation/

Learn how to implement cost-effective multi-tenant search using Amazon OpenSearch Serverless next-generation architecture with scale-to-zero compute and simplified routing through per-account, regional endpoints.

Building multi-tenant search architectures requires balancing data isolation with operational cost and complexity. In this post, we provide code examples for an implementation of multi-tenant search using a collection-per-tenant model with Amazon OpenSearch Serverless per-account, regional endpoints. Collection-per-tenant provides data and workload isolation. The regional endpoint simplifies routing requests for indexing and searching data.

Amazon OpenSearch Serverless is a serverless deployment option for Amazon OpenSearch Service that simplifies infrastructure management, index tuning, and data lifecycle management. OpenSearch Serverless automatically provisions and scales resources to provide consistently fast data ingestion rates and millisecond query response times during changing usage patterns and application demand.

The multi-tenant search problem

In search workloads, a tenant is a logical unit of data and the queries against that data. An eCommerce site has product categories. Each category is a tenant. A blog-hosting platform has blogs. Each blog is a tenant. Tenants map to resources in different ways. In the siloed model, each tenant gets its own container: a domain, collection, or index. In the pooled model, tenants share a container. The hybrid model silos large tenants and pools smaller ones together. Regardless of model, you need a mapping between tenant identifiers and the containers that hold their data, so your application routes requests correctly.

OpenSearch Serverless classic offered a collection-per-tenant strategy that simplified, but did not remove, the need for maintaining a tenant-container mapping. In addition, the cost structure of maintaining collection-per-tenant in classic was not ideal. Classic shared hardware across collections with the same AWS Key Management Service (AWS KMS) key. Tenants with different keys could not share hardware. The cost of the solution was the minimum monthly collection cost multiplied by the tenant count. Building for hundreds or thousands of tenants was cost-prohibitive. Collection groups improved this by allowing hardware sharing across AWS KMS keys, but compute costs were still driven by your indexed data, even during idle periods.

With the next-generation architecture, collection groups scale compute to zero. You pay for compute only when a tenant is actively indexing or searching (storage charges still apply). The addition of the regional endpoint further simplifies multi-tenant workloads by routing traffic to any collection through a single hostname. Together, scale-to-zero compute and the regional endpoint make the collection-per-tenant model both economically viable and operationally straightforward.

The OpenSearch Serverless per-account endpoint

OpenSearch Serverless next generation introduces a per-account, regional endpoint that serves all collections through a single hostname:

https://<account-id>.aoss.<region>.on.aws

The x-amz-aoss-collection-name or x-amz-aoss-collection-id header identifies the target collection on each request. This means one connection pool, one TLS session, and one endpoint to manage regardless of how many collections you have.

From a client perspective, you create a single OpenSearch client pointed at the regional endpoint and route requests by setting a header:

def get_opensearch_client(account_id: str, region: str) -> OpenSearch:
    """Create an OpenSearch client using the regional endpoint."""
    host = f"{account_id}.aoss.{region}.on.aws"
    auth = get_aws4auth(region)

    return OpenSearch(
        hosts=[{"host": host, "port": 443}],
        http_auth=auth,
        use_ssl=True,
        verify_certs=True,
        connection_class=RequestsHttpConnection,
        timeout=60,
    )

Every subsequent request includes the routing header to target a specific collection:

headers = {"x-amz-aoss-collection-name": collection_name}

This is a significant improvement over the classic architecture, where each collection had its own endpoint and you needed to manage separate connections for each.

Collection per tenant with query routing

The architecture is straightforward: one collection group holds all tenant collections, and the regional endpoint handles routing.

Create a collection group with scale-to-zero

client.create_collection_group(
    name="amazon-pqa-cg",
    generation="NEXTGEN",
    standbyReplicas="ENABLED",
    capacityLimits={
        "minIndexingCapacityInOCU": 0,
        "maxIndexingCapacityInOCU": 8,
        "minSearchCapacityInOCU": 0,
        "maxSearchCapacityInOCU": 8,
    },
)

When you set minIndexingCapacityInOCU and minSearchCapacityInOCU to 0, OpenSearch Serverless scales down your compute to 0 OpenSearch Compute Units (OCUs) when they are idle for 10 minutes. You pay only for the storage for your indices. If you want to maintain compute and avoid cold starts, set minIndexingCapacityInOCU or minSearchCapacityInOCU to a value greater than 0.

Create one collection per tenant

Each product category maps to its own collection within the group:

client.create_collection(
    name=name,
    type="SEARCH",
    collectionGroupName=COLLECTION_GROUP_NAME,
)

When choosing a collection name for your tenants, consider privacy, name length, and future ease of upgrading your application. You can use a hash function to map tenant identifiers to collection names.

import hashlib

def collection_name_for_tenant(tenant_id: str) -> str:
    """Generate an opaque collection name from a tenant identifier."""
    return hashlib.sha256(tenant_id.encode()).hexdigest()[:16]

Collection names are visible in API calls and logs. If your tenant ID contains personally identifiable information (PII), that information is also visible in logs. Hashing the tenant ID obfuscates the sensitive information.

OpenSearch Serverless has a 64-character limit on collection names. Your tenant ID can be longer than that. Hashing helps stay within this limit.

You might also want to add a prefix to collection names so that you can use wildcard patterns in access policies. For example, naming collections pqa-a1b2c3d4 lets you write a single data access policy matching collection/pqa-*. Including a version component in the name (such as pqa-v2-a1b2c3d4) makes it straightforward to create new collections during schema migrations without disrupting existing tenants.

Index data using the regional endpoint

A single OpenSearch client handles all collections. The x-amz-aoss-collection-name header routes each request to the correct collection:

headers = {"x-amz-aoss-collection-name": collection_name}

# Build bulk request
action = {"index": {"_index": index_name, "_id": doc["question_id"]}}
batch.append(json.dumps(action))
batch.append(json.dumps(doc))

# Send bulk request routed to the target collection
body = "\n".join(batch) + "\n"
resp = os_client.bulk(body=body, headers=headers)

Query a specific tenant’s data

Searching works the same way. Set the header to target the tenant’s collection:

os_client = get_opensearch_client(account_id, region)
headers = {"x-amz-aoss-collection-name": collection_name}

query = {
    "size": 3,
    "query": {
        "match": {
            "question_text": "4k resolution hdmi"
        }
    },
}

resp = os_client.search(index="questions", body=query, headers=headers)

The application layer maps a tenant ID (in this case, a product category) to a collection name, and the regional endpoint handles the rest. No connection pool management, no endpoint lookups, no per-tenant client instances.

Limitations

There are practical constraints to consider when adopting this pattern.

Cold start latency. When a collection group has scaled to zero compute, the first request takes approximately 10 seconds while capacity provisions. For latency-sensitive tenants, you can send a lightweight warmup query (such as a match_all with size=1) before production traffic arrives.

Collection group limits. There are account-level limits on the number of collections and collection groups. Check the Amazon OpenSearch Serverless quotas for current numbers if you are planning thousands of tenants.

Security policy size. Encryption, network, and data access policies list collection resource patterns. Because tenant count grows, these policy documents grow linearly. Use wildcard patterns to stay within OpenSearch Serverless policy size limits.

No cross-collection queries. Each search request targets exactly one collection. If you need to query across tenants for analytics or global search, you need an aggregation layer or a separate shared collection.

Conclusion

In this post, we showed how the next-generation OpenSearch Serverless architecture makes the collection-per-tenant model practical for multi-tenant search. Scale-to-zero reduces the minimum cost for inactive tenants, fitting the compute resources to the demands of tenants. The regional endpoint eliminates the operational complexity of managing per-tenant connections. You get full data isolation between tenants, independent scaling for each tenant’s workload, and a single endpoint to manage in your application code.

For more information, see the Amazon OpenSearch Serverless documentation.


About the author

Jon Handler

Jon Handler

Jon is a Senior Principal Solutions Architect for Search Services at Amazon Web Services. Jon works closely with OpenSearch and Amazon OpenSearch Service, providing help and guidance to a broad range of customers who have search and log analytics workloads. Prior to joining AWS, Jon’s career as a software developer included four years of coding a large-scale, eCommerce search engine.

Automating IT support with AI: How Nexthink uses OpenSearch Service to power self-service issue resolution

Post Syndicated from Rafael Ribeiro, Moe Haidar original https://aws.amazon.com/blogs/big-data/automating-it-support-with-ai-how-nexthink-uses-opensearch-service-to-power-self-service-issue-resolution/

This is a guest post by Rafael Ribeiro and Moe Haidar, at Nexthink, in partnership with AWS.

Nexthink is the leader in digital employee experience, helping enterprises improve how employees interact with technology in the workplace. The company gives IT teams real-time visibility into endpoint performance, application usage, and employee sentiment across millions of devices worldwide.

At the heart of Nexthink’s innovation is Spark, an autonomous artificial intelligence (AI) agent that automates IT support. Spark resolves IT issues for employees, from troubleshooting application crashes to resetting configurations and running remediation scripts. Rather than routing tickets or providing scripted responses, the agent takes direct action, achieving a 77% resolution rate at first contact without human escalation.

Spark operates at enterprise scale, deployed across 12 AWS Regions to serve global customers with low-latency responses.

In this post, we explore how Nexthink combined Amazon OpenSearch Service vector search, Amazon Bedrock, and infrastructure as code to power the Spark agent’s retrieval layer.

The challenge: Why vector search for AI agents?

For an AI agent to autonomously resolve IT issues, it must quickly retrieve the most relevant context from a vast knowledge base. Traditional keyword search falls short because:

  • Semantic understanding matters: An employee asking “my laptop is running slow” should match articles about “system performance optimization” even without exact keyword overlap.
  • Accurate retrieval drives correct outcomes: The quality of an AI agent’s response is only as good as the context it retrieves. When the agent pulls the right documentation, scripts, and historical resolutions, it produces accurate, safe actions. When retrieval is imprecise, the consequences can be severe. An agent acting on the wrong context could run destructive commands like rm -rf *, wipe critical data, or apply an incorrect fix that escalates the problem. Accurate vector search is the guardrail that keeps autonomous agents grounded in verified, relevant knowledge.
  • Speed is critical: Enterprise users expect near-instant responses, so retrieval must run in sub-second time across millions of documents.

This led Nexthink to implement Amazon OpenSearch Service with vector search capabilities, using Amazon Titan Text Embeddings V2 through Amazon Bedrock for embedding generation. With this architecture, Spark performs semantic search across all knowledge sources, retrieving contextually relevant information that drives accurate, autonomous issue resolution.

High-level architecture

The following diagram illustrates the high-level architecture of Nexthink’s Spark agent implementation with Amazon OpenSearch Service.

High-level architecture of Nexthink’s Spark AI agent, showing Amazon Elastic Kubernetes Service hosting the agent, Amazon OpenSearch Service as the vector store, and Amazon Bedrock providing the embedding model

Architecture components

Amazon Elastic Kubernetes Service (Amazon EKS) hosts the Spark agent, which interprets user queries, retrieves relevant context, and runs autonomous resolutions. With container orchestration, the agent scales horizontally across Nexthink’s 12 AWS Regions while maintaining consistent response times. The agent communicates with Amazon OpenSearch Service to perform semantic searches, retrieving the most contextually relevant documentation and automation scripts for each user’s issue.

Amazon OpenSearch Service functions as the central vector store, providing the k-Nearest Neighbors (k-NN) capabilities required for semantic search. OpenSearch Service stores document embeddings (dense vector representations of text content) alongside traditional metadata fields. When the AI agent submits a query, OpenSearch Service performs approximate nearest neighbor (ANN) searches to find documents with semantically similar embeddings, even when exact keywords don’t match. This vector search capability, combined with the proven scalability and managed infrastructure of OpenSearch Service, makes it well suited for AI agent architectures that require fast, accurate context retrieval.

Amazon Bedrock provides the foundation models used to generate text embeddings. Nexthink uses Amazon Titan Text Embeddings V2, hosted on Amazon Bedrock, to convert both documents and queries into dense vector representations. OpenSearch Service integrates natively with Amazon Bedrock through the OpenSearch ML Connector, which handles embedding generation at both index and query time.

Data ingestion pipeline

A critical component of any AI agent architecture is the data ingestion pipeline. This mechanism transforms raw documents into searchable, semantically indexed content in OpenSearch Service. For Spark, the pipeline must handle diverse data sources while automatically generating vector embeddings for semantic search.

Step 1: Staging and preprocessing layer

Staging layer in Amazon S3

Knowledge bases (KBs) are staged in Amazon Simple Storage Service (Amazon S3) before being processed through the ingestion pipeline. Amazon S3 provides durable storage, versioning capabilities, and integration with OpenSearch Service ingestion mechanisms. When documentation updates occur, new versions are uploaded to Amazon S3, which triggers the ingestion pipeline to reprocess and re-embed the content.

Event-driven streaming with Apache Kafka

IT tickets, agent interactions, and remote actions are processed through Apache Kafka for reliable message delivery during traffic spikes. Its consumer group model lets the ingestion pipeline scale horizontally based on event volume.

Step 2: Embedding generation during indexing time

Nexthink uses ingest pipelines inside OpenSearch Service to process the data at ingestion time, including generating text embeddings. When documents are sent to OpenSearch Service, the text_embedding processor inside the ingest pipeline automatically invokes the machine learning (ML) Connector to generate embeddings.

The ML Connector is the OpenSearch Service built-in framework for integrating external ML services. It handles request signing between OpenSearch Service and Amazon Bedrock, parses the Amazon Bedrock response to extract embeddings, maps them to index fields, and manages retries on failure. This eliminated the need for custom integration code and accelerated Nexthink’s time to market.

The following ingestion pipeline configuration demonstrates how to configure the text_embedding processor.

{
  "description": "Embedding ingestion pipeline for Spark AI Agent",
  "processors": [
    {
      "text_embedding": {
        "model_id": "<bedrock-connector-model-id>",
        "field_map": {
          "content": "content_embedding"
        }
      }
    }
  ]
}

In this configuration:

  • model_id: References the registered ML model connected to Amazon Bedrock.
  • field_map: Maps the source text field (content) to the target embedding field (content_embedding).

Step 3: Embeddings and data structure in OpenSearch Service

Nexthink stores embeddings alongside textual and metadata information in their k-NN index. For the vector field, they use Hierarchical Navigable Small World (HNSW) with the Lucene engine, as shown in the following example.

...
"content_embedding": {
  "type": "knn_vector",
  "dimension": 1024,
  "method": {
    "name": "hnsw",
    "space_type": "innerproduct",
    "engine": "lucene"
  }
},
"document_type": {
  "type": "keyword"
},
"tenant_id": {
  "type": "keyword"
}
...

In this configuration:

Multi-tenant search and retrieval

Enterprise AI agent deployments must address a critical challenge: making sure that users only access data they’re authorized to see. For Nexthink, serving multiple enterprise customers from a shared infrastructure requires robust multi-tenant security. Each customer’s knowledge base, automation scripts, and support tickets must remain isolated while the shared vector index continues to perform well.

The following diagram illustrates the search flow from user query to ranked results.

Search flow showing how a user query travels through the Spark agent, OpenSearch Service neural search, the ML Connector to Amazon Bedrock for embedding, and tenant-filtered k-NN retrieval to produce ranked results

Tenant management

Nexthink stores information about each tenant inside the tenant_id field. This design lets permission filters run efficiently alongside vector similarity searches. Additionally, Nexthink stores the tenant_id as a keyword type in the index mapping shared previously, so that filtering runs without the overhead of text analysis. Instead of pre-filtering with k-NN queries through a score script filter, the OpenSearch engine uses an intelligent decision-based approach for k-NN filtering called efficient filtering.

Neural query example with efficient filtering

OpenSearch’s neural search simplifies vector search by handling embedding generation as part of the query itself. Instead of requiring the application to call an embedding model separately and then submit a raw k-NN query with a vector, a neural query accepts plain text and uses the registered ML Connector to generate the embedding on the fly. As a result, the Spark agent can send natural-language queries directly to OpenSearch Service without any client-side embedding logic.

The following query demonstrates how Nexthink combines neural search with tenant isolation through efficient filtering in OpenSearch Service.

{
  "query": {
    "bool": {
      "must": [
        {
          "neural": {
            "content_embedding": {
              "query_text": "laptop running slow",
              "model_id": "<bedrock-connector-model-id>",
              "k": 50
            }
          }
        }
      ],
      "filter": [
        {
          "term": {
            "tenant_id": "customer-123"
          }
        }
      ]
    }
  }
}

In this query structure:

  • bool.must: Contains the neural search clause that performs semantic matching against document embeddings.
  • bool.filter: Applies the tenant isolation constraint, so that only documents belonging to customer-123 are returned.

Nexthink’s contribution to the technical community

A key principle in Nexthink’s architecture is treating infrastructure as code. With deployments spanning 12 AWS Regions, manual provisioning would be error-prone and time-consuming. Therefore, Nexthink uses several infrastructure as code (IaC) technologies, including Terraform, to provision resources.

Although the Terraform provider supports core OpenSearch Service resources like indices and index templates, it lacked support for some of the ML Commons resources required to integrate Amazon Bedrock:

  • ML Connectors: Required to establish connections to external ML services like Amazon Bedrock.
  • ML Model Groups: Needed to organize and manage related models.
  • ML Models: Required to register models that use the connectors.

Without these resources, Nexthink initially relied on workarounds using local-exec provisioners and null_resource blocks to call the OpenSearch Service API directly. This approach was fragile, difficult to maintain, and didn’t integrate well with Terraform’s state management.

Contributing back

Rather than maintaining a private fork indefinitely, Nexthink chose to contribute their custom Terraform resources back to the OpenSearch Project community. This decision aligned with their engineering values to help other organizations implement similar architectures and contribute to the broader community.

Open source contribution links

The Terraform provider contributions are being added to the official OpenSearch project repository:

  • Pull Request: Add support for ML Connector, ML Model Group, and ML Model resources #280.
  • Feature Request: Contribution – Support for ML resources #281.

These contributions let any organization provision OpenSearch Service ML resources with Terraform, which streamlines the deployment of AI agent architectures that integrate with Amazon Bedrock or other ML services.

Conclusion

Nexthink’s implementation of Amazon OpenSearch Service for the Spark agent demonstrates how vector search capabilities can power autonomous IT support at enterprise scale. By combining semantic search with multi-tenant security and infrastructure as code practices, Nexthink achieved a 77% resolution rate at first contact, so that employees can resolve IT issues without human escalation.

Get started

Ready to build your own AI agent with vector search capabilities? Here are your next steps:

  1. Explore Amazon OpenSearch Service vector search features in the OpenSearch Service documentation.
  2. Configure ML Connectors for Amazon Bedrock using the ML Commons plugin guide.
  3. Automate with Terraform using the contributed resources in the terraform-provider-opensearch repository.

The combination of Amazon OpenSearch Service, Amazon Bedrock, and infrastructure as code practices provides a foundation for building intelligent, context-aware AI agents that deliver business value.


About the authors

Rafael Ribeiro

Rafael Ribeiro

Rafael is a Software Engineer at Nexthink, focusing on infrastructure and DevOps for AI teams.

Moe Haidar

Moe Haidar is Head of Agentic AI and Engineering at Nexthink, where he leads AI architecture and strategy alongside the development of Spark, the company’s autonomous personal IT agent that resolves employee issues at scale.

Hajer Bouafif

Hajer Bouafif

Hajer is an Analytics Specialist Solutions Architect at Amazon Web Services. She focuses on Amazon OpenSearch Service and helps customers design and build well-architected analytics workloads in diverse industries. Hajer enjoys spending time outdoors and discovering new cultures.

Luca Perrozzi

Luca Perrozzi

Luca is a Solutions Architect at AWS, based in Switzerland. He focuses on innovation topics at AWS, especially in the area of Artificial Intelligence. Luca holds a PhD in particle physics and has 15 years of hands-on experience as a research scientist and software engineer.

Building AI shopping agent using Amazon Bedrock AgentCore Runtime and Amazon OpenSearch Service

Post Syndicated from Omama Khurshid original https://aws.amazon.com/blogs/big-data/building-ai-shopping-agent-using-amazon-bedrock-agentcore-runtime-and-amazon-opensearch-service/

In this post, we explore how to build an online shopping AI agent. We focus on its architecture and implementation with Amazon OpenSearch Service, Amazon Bedrock AgentCore, and Strands Agents. Amazon Bedrock AgentCore is an agentic platform for deploying and operating those agents and tools securely at scale without managing infrastructure. AgentCore Runtime is the secure, serverless runtime that hosts your Strands Agents and tools as containerized applications. Strands Agents is an open source SDK for building AI agents. In this SDK, an agent is defined by a model, tools, and a prompt. Tools are callable functions that allow agents to perform actions beyond text generation, such as API calls, database queries, and file operations. The framework lets the model autonomously plan steps and invoke tools to complete tasks.

Today’s AI shopping assistants understand natural language, context, and shopping intent, creating a more human-like interaction. These assistants handle complex shopping requirements, such as “Find me a formal dress under $200 that’s appropriate for a summer wedding.” They maintain conversation history, process follow-up questions naturally, and provide personalized recommendations based on user preferences and past interactions. Customers can use visual search to upload images of items that they want, and the AI finds similar products across multiple retailers, matching styles and patterns. The goal is to provide instant, relevant, and personalized assistance at scale, creating a more efficient shopping journey for consumers worldwide.

AI agents combined with Retrieval Augmented Generation (RAG) on Amazon OpenSearch Service represent an evolution in conversational search. This integration builds AI agents on enriched catalogs, supporting context-aware and autonomous search experiences while maintaining accuracy and relevance through grounded responses.

Solution overview

The following diagram illustrates the solution architecture of an AI-powered online shopping agent built using Strands Agents, Amazon Bedrock AgentCore Runtime, and Amazon OpenSearch Service. For simplicity, the diagram doesn’t show authentication and authorization. In a production setup, secure access to the backend by using mechanisms such as Amazon API Gateway, AWS Identity and Access Management (IAM) roles, or OAuth-based authentication.

Architecture diagram showing an AI shopping agent: a user prompt flows from the front end to AgentCore Runtime, which routes the request to a Strands Retail Agent that calls a search tool against Amazon OpenSearch Service and an Amazon Bedrock LLM, then returns a natural-language response to the user.

The following is a walkthrough of the reference architecture:

  1. The user submits a question through the front-end application. AgentCore Runtime receives the request and routes it to the Strands Retail Agent.
  2. The Strands Agent processes the task and invokes the search_product_catalog tool.
  3. OpenSearch Service performs semantic search and returns relevant product results.
  4. The Strands Agent invokes Amazon Bedrock large language models (LLMs) to generate a natural language response.
  5. The agent response is returned to the user through the front end.

Walkthrough

The following section walks you through how to build an online shopping AI agent.

Prerequisites

To implement this solution, you need an AWS account. You also need an OpenSearch Service domain with OpenSearch version 2.13 or later. You can use an existing domain or create a new domain.

To use the vector search capabilities of OpenSearch Service with Strands Agents on AgentCore, you use ingest pipelines. These ingestion pipelines apply built-in processors to pre-process your documents before you index them in OpenSearch Service.

You use the text_embedding processor, which relies on the ML Commons plugin and a registered embedding model—Amazon Nova Multimodal Embeddings on Amazon Bedrock. OpenSearch Service uses the ML Commons plugin to generate vector embedding for your data and uses the same model to convert incoming queries into vectors. This supports semantic search across your indexed content.

You extend your semantic search backend by adding an agent built with Strands Agents and deployed on Amazon Bedrock AgentCore.

Code samples provided in this post are tested in Python 3.11. You only need to install Python 3.11 in your environment to execute the python scripts. You also need Node.js 18 or later installed to use the AgentCore CLI. The provided code scripts will deploy into your AWS account so make sure your terminal has access to necessary AWS credentials.

Install AgentCore CLI

Install the AgentCore CLI globally using npm:

npm install -g @aws/agentcore

Python Dependencies

You also need to create a requirements.txt file with following dependencies in your workspace to deploy the agents.

boto3
uv
opensearch-py
requests-aws4auth
strands-agents
strands-agents-tools
bedrock-agentcore

Run pip install -r requirements.txt in your terminal to install the required dependencies. To avoid conflicts with other dependencies in your system, you can use a virtual environment.

Now, walk through each step.

Step 1: Configure IAM permissions

Complete the following steps to register the Nova Multimodal Embeddings model with OpenSearch Service and verify that your OpenSearch Service domain has permission to invoke the Amazon Bedrock API.

  1. Go to the IAM console and create a new role with a custom trust policy. Add the following trust policy.
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Statement1",
          "Effect": "Allow",
          "Principal": {
            "Service": "opensearchservice.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }

  2. Skip adding a permission policy.
  3. Give your role a name and create it. For this post, we use OpenSearchBedrockEmbeddingRole as the role name. OpenSearch Service uses this role to invoke the Nova Multimodal Embeddings model on Amazon Bedrock.
  4. On the Permissions tab, attach an inline policy with the following permissions. For this post, we name this policy OpenSearchBedrockEmbeddingPolicy.
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "Statement1",
          "Effect": "Allow",
          "Action": [
            "bedrock:InvokeAgent",
            "bedrock:InvokeModel"
          ],
          "Resource": [
            "arn:aws:bedrock:us-east-1::foundation-model/*"
          ]
        }
      ]
    }

  5. Create a passRole policy with the following JSON document and assign it to the IAM role that creates the ML connector. This lets the principal running the Python code pass the OpenSearchBedrockEmbeddingRole to OpenSearch. Replace <your-aws-account-id> with your own AWS account ID.
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "iam:PassRole",
          "Resource": "arn:aws:iam::\$<your-aws-account-id>:role/OpenSearchBedrockEmbeddingRole"
        }
      ]
    }

  6. By using fine-grained access control (FGAC), map the IAM role as a backend role for the ml_full_access role in the OpenSearch Dashboards Security plugin. This mapping lets the user create ML connectors:
    1. Log in to OpenSearch Dashboards and open the Security page from the navigation menu.
    2. Choose Roles and select ml_full_access.
    3. Choose Mapped Users and Manage Mapping.
    4. Under Backend roles, add the ARN of the IAM role that you created in the previous steps.

Animated demo of OpenSearch Dashboards Security plugin showing the ml_full_access role with the Mapped Users tab open and an IAM role being added as a backend role.

Step 2: Connect to the model by using OpenSearch ML Connectors

In this section, you create an ML connector to link OpenSearch Service with the Bedrock Nova Multimodal Embeddings model. You then register and deploy the model so you can use it for neural search queries.

  1. Create a file named create-connector.py with the following code. Replace <your hostname>, <your region>, and <your account id> placeholders within the code.
import boto3
import requests
from requests_aws4auth import AWS4Auth
host = '<your hostname>'##CHANGE THIS
region = '<your region>' ##CHANGE THIS
account_id = '<your account id>' ##CHANGE THIS
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
path = '/_plugins/_ml/connectors/_create'
url = host + path
payload = {
"name": "Amazon Bedrock Nova multimodal model - text embedding",
"description": "Test connector for Amazon Bedrock Nova multimodal model - text embedding",
"version": 1,
"protocol": "aws_sigv4",
"credential": {
"roleArn": f"arn:aws:iam::{account_id}:role/OpenSearchBedrockEmbeddingRole"
},
"parameters": {
"region": region,
"service_name": "bedrock",
"model": "amazon.nova-2-multimodal-embeddings-v1:0",
"input_docs_processed_step_size": 1,
"dimensions": 1024,
"embeddingTypes": [
"float"
],
"truncationMode": "NONE"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"headers": {
"content-type": "application/json",
"x-amz-content-sha256": "required"
},
"url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
"request_body": "{\\n \\"taskType\\": \\"SINGLE_EMBEDDING\\",\\n \\"singleEmbeddingParams\\": {\\n \\"embeddingPurpose\\": \\"GENERIC_INDEX\\",\\n \\"embeddingDimension\\": ${parameters.dimensions},\\n \\"text\\": {\\n \\"truncationMode\\": \\"${parameters.truncationMode}\\",\\n \\"value\\": \\"${parameters.inputText}\\"\\n }\\n }\\n}",
"pre_process_function": "connector.pre_process.bedrock.nova.text_embedding",
"post_process_function": "connector.post_process.bedrock.nova.embedding"
}
]
}
headers = {"Content-Type": "application/json"}
r = requests.post(url, auth=awsauth, json=payload, headers=headers, timeout=15)
print(r.status_code)
print(r.text)
  1. Run python create-connector.py in your terminal by using the IAM role with ml_full_access and passRole permissions created in the previous step. This script creates a connector between OpenSearch Service and the Bedrock Nova Multimodal Embeddings model.
  2. The program responds with connector_id. Take a note of it. Then, navigate to OpenSearch Dashboards and open Dev Tools. Create a model group against which to register this model in the OpenSearch Service domain.
POST /_plugins/_ml/model_groups/_register
{
  "name": "agent-conversational-search-model-group",
  "description": "A model group for bedrock Nova embedding models used for conversational search"
}
  1. Register a model by using connector_id and model_group_id.
POST /_plugins/_ml/models/_register
{
  "name": "nova-2-multimodal-embedding-v1",
  "function_name": "remote",
  "model_group_id": "<model group id>",
  "description": "Nova 2 Multimodal Embeddings Model",
  "connector_id": "<connector id>",
  "interface": {}
}
  1. Run the following API call to deploy the model. Use the registered model ID from the previous step.
POST /_plugins/_ml/models/<registered-model-id>/_deploy

Step 3: Create an ingest pipeline for data indexing

Use the following code to create an ingest pipeline for data indexing. The pipeline establishes a connection to the embedding model, retrieves the embedding for the title field, and stores it in the OpenSearch index.

PUT /_ingest/pipeline/nova_multimodal_embedding
{
  "description": "Text embedding pipeline using nova_multimodal_embedding",
  "processors": [
    {
      "text_embedding": {
        "model_id": "<deployed model id>",
        "field_map": {
          "title": "title_vector"
        }
      }
    }
  ]
}

Step 4: Create an index for storing data

Create an index named product for storing data by using Dev Tools. This index stores raw text and 1024-dimensional embeddings of the title field, and uses the ingest pipeline you created in the previous step.

PUT /product
{
  "settings": {
    "index": {
      "default_pipeline": "nova_multimodal_embedding",
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "standard"
      },
      "title_vector": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "space_type": "cosinesimil",
          "engine": "lucene"
        }
      },
      "price": {
        "type": "float"
      },
      "description": {
        "type": "text"
      },
      "category": {
        "type": "keyword"
      },
      "image_url": {
        "type": "text"
      },
      "rating": {
        "properties": {
          "rate": {
            "type": "float"
          },
          "count": {
            "type": "integer"
          }
        }
      }
    }
  }
}

Step 5: Ingest sample data

Use the following code to ingest the sample product data in Dev Tools.

POST /_bulk
{"index": {"_index": "product", "_id": "2"}}
{"id":2,"title":"Mens Casual Premium Slim Fit T-Shirts","price":22.3,"description":"Slim-fitting style, contrast raglan long sleeve, three-button henley placket, light weight & soft fabric for breathable and comfortable wearing.","category":"men's clothing","image":"https://fakestoreapi.com/img/71-3HjGNDUL._AC_SY879._SX._UX._SY._UY_.jpg","rating":{"rate":4.1,"count":259}}
{"index": {"_index": "product", "_id": "3"}}
{"id":3,"title":"Mens Cotton Jacket","price":55.99,"description":"great outerwear jackets for Spring/Autumn/Winter, suitable for many occasions, such as working, hiking, camping, mountain/rock climbing, cycling, traveling or other outdoors.","category":"men's clothing","image":"https://fakestoreapi.com/img/71li-ujtlUL._AC_UX679_.jpg","rating":{"rate":4.7,"count":500}}
{"index": {"_index": "product", "_id": "4"}}
{"id":4,"title":"Mens Casual Slim Fit","price":15.99,"description":"The color could be slightly different between on the screen and in practice.","category":"men's clothing","image":"https://fakestoreapi.com/img/71YXzeOuslL._AC_UY879_.jpg","rating":{"rate":2.1,"count":430}}
{"index": {"_index": "product", "_id": "5"}}
{"id":5,"title":"John Hardy Women's Legends Naga Gold & Silver Dragon Station Chain Bracelet","price":695,"description":"From our Legends Collection, the Naga was inspired by the mythical water dragon that protects the ocean's pearl.","category":"jewelery","image":"https://fakestoreapi.com/img/71pWzhdJNwL._AC_UL640_QL65_ML3_.jpg","rating":{"rate":4.6,"count":400}}

Step 6: Query the index

Run the following API call to test semantic search by using the Nova Multimodal Embeddings model.

GET /product/_search
{
  "_source": false,
  "fields": [
    "title",
    "price",
    "category",
    "image"
  ],
  "size": 3,
  "query": {
    "neural": {
      "title_vector": {
        "query_text": "jacket",
        "model_id": "<deployed model id>",
        "k": 5
      }
    }
  }
}

The output of the preceding query should look like the following.

{
  ...
  "hits": {
    "total": {
      "value": 4,
      "relation": "eq"
    },
    "max_score": 0.8333229,
    "hits": [
      {
        "_index": "product",
        "_id": "3",
        "_score": 0.8333229,
        "fields": {
          "image": [
            "https://fakestoreapi.com/img/71li-ujtlUL._AC_UX679_.jpg"
          ],
          "title": [
            "Mens Cotton Jacket"
          ],
          "category": [
            "men's clothing"
          ],
          "price": [
            55.99
          ]
        }
      }
      ...
    ]
  }
}

Step 7: Create an agent with Strands and Bedrock AgentCore Runtime

Now, create the Strands Agent that uses Anthropic Claude Sonnet 4.6 on Amazon Bedrock to search products from the OpenSearch Service index. To do so:

  • Import the Runtime app with from bedrock_agentcore.runtime import BedrockAgentCoreApp.
  • Initialize the app in your code with app = BedrockAgentCoreApp().
  • Create the OpenSearch Service connection and search query with the @tool decorator.
  • Decorate the invocation function with the @app.entrypoint decorator.
  • Let AgentCore Runtime control the running of the agent with app.run().

Now, complete the following steps:

  1. Make sure that you have installed the necessary dependencies from the Prerequisites section of this post.
  2. Create and save a file named search_agent.py with the following code. Replace <your hostname>, <your region>, and <your account id> placeholders within the code.
    from strands import Agent, tool
    import argparse
    import json
    from bedrock_agentcore.runtime import BedrockAgentCoreApp
    from strands.models import BedrockModel
    import boto3
    from opensearchpy import OpenSearch, RequestsHttpConnection
    from requests_aws4auth import AWS4Auth
    app = BedrockAgentCoreApp()
    @tool
    def search_products(query: str, size: int = 5):
    try:
    # OpenSearch configuration
    host = '' ## CHANGE THIS, DOMAIN ENDPOINT WITHOUT HTTPS!
    region = '' ##CHANGE THIS
    model_id= '' ##CHANGE THIS with your deployed model id in OpenSearch
    service = 'es'
    credentials = boto3.Session().get_credentials()
    awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
    # Create OpenSearch client
    client = OpenSearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
    )
    """Search products in OpenSearch using neural search"""
    search_body = {
    "_source": False,
    "fields": ["title", "price", "category", "image"],
    "size": size,
    "query": {
    "neural": {
    "title_vector": {
    "query_text": query,
    "model_id": model_id,
    "k": 3
    }
    }
    }
    }
    response = client.search(
    body=search_body,
    index="product"
    )
    products = []
    for hit in response['hits']['hits']:
    fields = hit.get('fields', {})
    product = {
    'title': fields.get('title', [''])[0] if fields.get('title') else '',
    'price': fields.get('price', [''])[0] if fields.get('price') else '',
    'category': fields.get('category', [''])[0] if fields.get('category') else '',
    'image': fields.get('image', [''])[0] if fields.get('image') else ''
    }
    products.append(product)
    return f"Found {len(products)} products: {json.dumps(products, indent=2)}"
    except Exception as e:
    return f"Search error: {str(e)}"
    model_id = "global.anthropic.claude-haiku-4-5-20251001-v1:0"
    model = BedrockModel(
    model_id=model_id,
    )
    agent = Agent(
    model=model,
    tools=[search_products],
    system_prompt="You're a helpful assistant. You can do product search, and tell the product details."
    )
    @app.entrypoint
    def strands_agent_bedrock(payload):
    """
    Invoke the agent with a payload
    """
    user_input = payload.get("prompt")
    print("User input:", user_input)
    response = agent(user_input)
    return response.message['content'][0]['text']
    if __name__ == "__main__":
    #strands_agent_bedrock({"prompt": "Search jacket"}) ##UNCOMMENT THIS FOR TESTING
    #app.run() ##UNCOMMENT THIS FOR DEPLOYMENT, MAKE SURE THE ABOVE LINE IS COMMENTED WHEN YOU ARE DEPLOYING TO AGENTCORE

    This deploys your agent locally for testing purposes.

  3. Navigate to the IAM console and add the AmazonBedrockLimitedAccess permission policy to the principal running the code.
  4. Navigate to OpenSearch Dashboards and, from the left menu, choose Security plugin, then choose Roles.
  5. Choose Create Role.
  6. Name the role agentcore-permissions.
  7. Under cluster permissions, add cluster:admin/opensearch/ml/models/get and cluster:admin/opensearch/ml/predict.
  8. Under index permissions, enter product* as the index pattern. Add search and get permissions.
  9. Create the role.
  10. Choose the role you created, switch to the Mapped Users tab, choose Manage mapping, and add the role that you use for running the Python code as a backend role.
  11. Uncomment the line strands_agent_bedrock({"prompt": "Search jacket"}) and make sure the app.run() line is commented in the code.
  12. Run python search_agent.py in your terminal to start the shopping agent. The output should look similar to the following.
    "Here are the jacket search results:\n\n1. **Mens Cotton Jacket** - $55.99\n2. **Mens Casual Slim Fit** - $15.99\n3. **Mens Casual Premium Slim Fit T-Shirts** - $22.30\n4. **John Hardy Women's Legends Naga Gold & Silver Dragon Station Chain Bracelet** - $695.00\n\nThe most relevant jacket option is the **Mens Cotton Jacket** at $55.99. Would you like to know more about any of these products?

  13. Comment strands_agent_bedrock({"prompt": "Search jacket"}) and uncomment the app.run() line in the code before going into the next step.

Step 8: Configure and launch your agent to Bedrock AgentCore Runtime

The AgentCore CLI is a command-line tool provided by AWS that simplifies deployment of agents to Amazon Bedrock AgentCore Runtime. When you run the CLI deployment command, it automates the entire deployment workflow: it creates the necessary IAM execution role with proper permissions, packages your Python application code along with its dependencies, uses AWS CodeBuild to build an optimized Docker container image, pushes that container image to Amazon Elastic Container Registry (ECR), and finally provisions the AgentCore Runtime environment that hosts your containerized agent. This eliminates the need for manual Dockerfile creation, container builds, or infrastructure management.

  1. Before you start this step, make sure you have gone through section 7 and installed the AgentCore CLI and Python dependencies listed in the Prerequisites section.
  2. Create a policy named AgentCoreAccessPolicy with the following permissions and attach it to the role running the code. Replace <ACCOUNT_ID> and <REGION> placeholders.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BedrockAgentCoreServiceAccess",
      "Effect": "Allow",
      "Action": [
        "bedrock-agentcore:*"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ECRAuthorizationToken",
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ECRRepositoryAccess",
      "Effect": "Allow",
      "Action": [
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage",
        "ecr:PutImage",
        "ecr:InitiateLayerUpload",
        "ecr:UploadLayerPart",
        "ecr:CompleteLayerUpload",
        "ecr:DescribeRepositories",
        "ecr:CreateRepository",
        "ecr:ListImages",
        "ecr:DescribeImages"
      ],
      "Resource": [
        "arn:aws:ecr:<REGION>:<ACCOUNT_ID>:repository/bedrock-agentcore-*"
      ]
    },
    {
      "Sid": "CodeBuildProjectAccess",
      "Effect": "Allow",
      "Action": [
        "codebuild:CreateProject",
        "codebuild:UpdateProject",
        "codebuild:StartBuild",
        "codebuild:BatchGetBuilds",
        "codebuild:DeleteProject"
      ],
      "Resource": "*"
    },
    {
      "Sid": "IAMRoleManagement",
      "Effect": "Allow",
      "Action": [
        "iam:CreateRole",
        "iam:AttachRolePolicy",
        "iam:PutRolePolicy",
        "iam:GetRole",
        "iam:GetRolePolicy",
        "iam:PassRole",
        "iam:DeleteRole",
        "iam:DeleteRolePolicy",
        "iam:DetachRolePolicy",
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:aws:iam::<ACCOUNT_ID>:role/BedrockAgentCoreExecutionRole-*",
        "arn:aws:iam::<ACCOUNT_ID>:role/AmazonBedrockAgentCoreSDKCodeBuild-*",
        "arn:aws:iam::<ACCOUNT_ID>:role/aws-service-role/runtime-identity.bedrock-agentcore.amazonaws.com/AWSServiceRoleForBedrockAgentCoreRuntimeIdentity"
      ]
    },
    {
      "Sid": "CloudWatchLogsAccess",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:PutResourcePolicy"
      ],
      "Resource": "arn:aws:logs:<REGION>:<ACCOUNT_ID>:log-group:/aws/bedrock-agentcore/*"
    },
    {
      "Sid": "CloudWatchLogsResourcePolicy",
      "Effect": "Allow",
      "Action": [
        "logs:PutResourcePolicy"
      ],
      "Resource": "*"
    },
    {
      "Sid": "S3BucketManagement",
      "Effect": "Allow",
      "Action": [
        "s3:CreateBucket",
        "s3:PutBucketPolicy",
        "s3:PutBucketVersioning",
        "s3:PutBucketPublicAccessBlock",
        "s3:PutLifecycleConfiguration",
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::bedrock-agentcore-codebuild-sources-<ACCOUNT_ID>-*",
        "arn:aws:s3:::bedrock-agentcore-codebuild-sources-<ACCOUNT_ID>-*/*"
      ]
    }
  ]
}
  1. Create a file named agentcore.yaml in your project directory with the following configuration. Replace <REGION>, <ACCOUNT_ID>, and <OPENSEARCH_DOMAIN_NAME> placeholders:
# AgentCore Runtime Configuration
  runtime:
    name: shopping-search-agent-runtime
    entrypoint: search_agent.py:strands_agent_bedrock
    region: <REGION>  # CHANGE THIS - Your AWS region (e.g., us-east-1)

    execution_role:
      create: true
      name: BedrockAgentCoreExecutionRole-shopping-agent
      policies:
        - policy_name: BedrockAndOpenSearchAccess
          policy_document:
            Version: "2012-10-17"
            Statement:
              - Sid: BedrockModelAccess
                Effect: Allow
                Action:
                  - bedrock:InvokeModel
                  - bedrock:InvokeModelWithResponseStream
                Resource:
                  - arn:aws:bedrock:*::foundation-model/*
                  - arn:aws:bedrock:<REGION>:<ACCOUNT_ID>:inference-profile/*  # CHANGE THIS - Replace <REGION> and <ACCOUNT_ID>
              - Sid: OpenSearchAccess
                Effect: Allow
                Action:
                  - es:ESHttpGet
                  - es:ESHttpPost
                  - es:ESHttpPut
                Resource: arn:aws:es:<REGION>:<ACCOUNT_ID>:domain/<OPENSEARCH_DOMAIN_NAME>/*  # CHANGE THIS - Replace all three placeholders
              - Sid: CloudWatchLogsAccess
                Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: arn:aws:logs:<REGION>:<ACCOUNT_ID>:log-group:/aws/bedrock-agentcore/runtimes/*  # CHANGE THIS - Replace <REGION> and <ACCOUNT_ID>
              - Sid: ECRImageAccess
                Effect: Allow
                Action:
                  - ecr:GetAuthorizationToken
                  - ecr:BatchGetImage
                  - ecr:GetDownloadUrlForLayer
                Resource: "*"

    container:
      architecture: arm64  # Options: arm64 or x86_64
      requirements_file: requirements.txt

    ecr:
      auto_create: true
  1. Run the following command in your terminal to deploy the agent to AgentCore Runtime:
    • Create the AgentCore project and replace with your search agent.
      agentcore create --name ShoppingAgent --defaults
      cd ShoppingAgent
      cp ../search_agent.py app/ShoppingAgent/main.py

    • Add OpenSearch and other dependencies:
      cd app/ShoppingAgent
      uv add opensearch-py requests-aws4auth boto3
      cd ../..

    • Deploy the agent to Agentcore Runtime. This process takes approximately 5-10 minutes.
      agentcore deploy

    • Once deployment completes, verify the runtime status:
      agentcore status

      You should see:

      ShoppingAgent: Deployed - Runtime: READY (arn:aws:bedrock-agentcore:<REGION>:<ACCOUNT_ID>:runtime/ShoppingAgent_...)
      
      URL: https://bedrock-agentcore.<REGION>.amazonaws.com/runtimes/.../invocations

Step 9: Configure OpenSearch Service access

Map your AgentCore execution role to an OpenSearch backend role so the agent can access your data.

  1. Navigate to OpenSearch Dashboards. From the left menu, choose the Security plugin, then choose Roles.
  2. Search for agentcore-permissions and choose the role. Then, navigate to the Mapped Users tab, choose Manage mapping, and add arn:aws:iam::<ACCOUNT_ID>:role/AmazonBedrockAgentCoreSDKRuntime-us-east-1-custom as a backend role. Replace the <ACCOUNT_ID> placeholder with your account ID.

Animated demo of OpenSearch Dashboards Security plugin showing the agent-permissions role with the Mapped Users tab open and the AgentCore SDK runtime IAM role added as a backend role.

Step 10: Invoke the Bedrock AgentCore Runtime

You can test the agent in Agent Sandbox. Enter the prompt Search jacket less than 50$, and the agent returns the relevant result from the OpenSearch Service index with a summary.

Agent Sandbox console showing a shopping agent response that returns the Mens Cotton Jacket as the relevant result for the prompt “Search jacket less than 50$”.

In real-world scenarios, you can design a search application with a Strands Agent deployed in AgentCore Runtime. You can add AgentCore Memory, which gives your AI agents the ability to remember past interactions and provide more context-aware, personalized conversations.

Cleanup

To avoid incurring future charges, delete the resources created while building this solution:

  1. Delete the OpenSearch Service domain.
  2. Delete the Amazon Bedrock AgentCore Runtime resources.

Conclusion

In this post, you saw how to create a conversational search with Amazon OpenSearch Service and Strands Agents. You also learned how to deploy the agent on Amazon Bedrock AgentCore Runtime. You can further enhance this shopping agent by using other AgentCore capabilities. For example, AgentCore Memory retains user preferences and past interactions across sessions, AgentCore Identity manages shopper authentication and access control, and AgentCore Observability helps you monitor and debug agent behavior in production. Together, these services help you build shopping experiences that deliver instant, relevant assistance at scale.

Now it’s your turn. Build your own conversational search experience by integrating OpenSearch Service and Strands Agents with your product catalog. To learn more, see the Amazon OpenSearch Service and Amazon Bedrock AgentCore detail pages.


About the authors

Omama Khurshid

Omama Khurshid

Omama is an GTM Specialist Solutions Architect Analytics at Amazon Web Services. She focuses on helping customers across various industries build reliable, scalable, and efficient solutions. Outside of work, she enjoys spending time with her family, listening to music, and learning new technologies.

Jumana Nagaria

Jumana Nagaria

Jumana is a Prototyping Architect at AWS. She builds innovative prototypes with customers to solve their business challenges. She is passionate about cloud computing and data analytics. Outside of work, Jumana enjoys travelling, reading, painting, and spending quality time with friends and family.

Canberk Keles

Canberk Keles

Canberk is a Solutions Architect at Amazon Web Services, helping software companies achieve their business goals by leveraging AWS technologies. He is part of OpenSearch specialist community within AWS and has been guiding customers harness the power of OpenSearch. Outside of work, he enjoys sports, reading, traveling and playing video games.

Amazon OpenSearch Service: Mechanisms to secure your domain

Post Syndicated from Imtiaz Sayed original https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-mechanisms-to-secure-your-domain/

Imagine you’re building a product search feature for your website or storing customer records in Amazon OpenSearch Service to power full-text search. The moment that real user data enters your domain, security becomes essential.

Whether your workload is a public-facing website search, an internal application querying sensitive data, or a pipeline handling personally identifiable information (PII), the questions you face are the same:

  • Who should be allowed to connect to my domain?
  • How do I authenticate users and services?
  • How do I make sure that even authenticated users only see data they are entitled to see?
  • How do I satisfy regulatory requirements such as HIPAA, PCI DSS, or SOC 2?

This post offers an overview of the security mechanisms available for Amazon OpenSearch Service, spanning authentication and authorization, encryption, and network access controls. You learn how to implement fine-grained access control, manage AWS Identity and Access Management (IAM) roles, and secure data both in transit and at rest for both public and virtual private cloud (VPC) access domains.

Scope: This post covers security for Amazon OpenSearch Service managed clusters only. It doesn’t cover Amazon OpenSearch Serverless, which uses a different security model. For serverless security, see Amazon OpenSearch Serverless security in the AWS documentation.

To begin, let’s look at the security layers in Amazon OpenSearch Service.

Amazon OpenSearch Service security layers

Amazon OpenSearch Service has multi-layer security. The following diagram illustrates the multi-layer security in Amazon OpenSearch Service.

Diagram showing the three security layers of Amazon OpenSearch Service: Network, Domain access policy, and Fine-grained access control

Figure 1: Multi-layer security.

The three main layers of security are network, domain access policy, and fine-grained access control.

Network – The first security layer is the network, which determines whether requests reach an OpenSearch Service domain. If you choose Public access when you create a domain, requests from any internet-connected client can reach the domain endpoint. If you choose VPC access, clients must connect to the Amazon Virtual Private Cloud (Amazon VPC) (and the associated security groups must permit it) for a request to reach the endpoint.

Domain access policy – The second security layer is the domain access policy. After a request reaches a domain endpoint, the resource-based access policy allows or denies the request access to a given URI. The access policy accepts or rejects requests at the edge of the domain, before they reach data or indexes in OpenSearch itself.

Fine-grained access control – The third and final security layer is fine-grained access control. After a resource-based access policy allows a request to reach a domain endpoint, fine-grained access control evaluates the user credentials and either authenticates the user or denies the request. If fine-grained access control authenticates the user, it fetches all OpenSearch internal roles mapped to that user and uses the full set of permissions to determine how to handle the request.

With fine-grained access control, you can control access to your data in Amazon OpenSearch Service. For example, depending on who makes the request, you might want to hide certain fields in your documents or exclude certain documents altogether. With fine-grained access control, you can:

  • Define role-based access control to determine who can perform which actions on which indexes, documents, and fields.
  • Define security at the index, document, and field level to allow access to only required data.

Fine-grained access control requires OpenSearch or Elasticsearch 6.7 or later. It also requires HTTPS for all traffic to the domain, encryption of data at rest, and node-to-node encryption. Depending on how you configure the advanced features of fine-grained access control, more processing of your requests might require compute and memory resources on individual data nodes. After you turn on fine-grained access control, you can’t turn it off. For more details, see Fine-grained access control in Amazon OpenSearch Service in the AWS documentation.

To learn more about security features in an OpenSearch Service domain, let’s start by configuring a new public access domain. We discuss a VPC access domain later in the post.

Public access domain

With a public access domain, you can configure an OpenSearch Service domain so that the domain endpoint is accessible from the internet.

The AWS console for Amazon OpenSearch Service provides a guided wizard that you can use to configure and reconfigure your provisioned Amazon OpenSearch Service domains. Follow the Tutorial: Configure a domain with the internal user database and HTTP basic authentication in the AWS documentation to configure a domain with basic authentication and validate fine-grained access control.

Let’s review some important configuration attributes for a public access domain.

Network:

Public access. To simplify the network access configurations, you can use Public access, but for production workloads, we recommend VPC access.

With the domain in public access, you have several options to secure access. While you can use a resource-based access policy to restrict access to specific IAM principals or IP addresses, the recommended approach is to turn on fine-grained access control (FGAC) and use it as the primary mechanism for securing your domain. With FGAC turned on, you can set an open access policy (allowing all traffic to reach the domain) and let FGAC handle authentication and authorization at the index, document, and field level.

When using IAM-based authentication with FGAC, you should map IAM roles to backend roles in OpenSearch. You can use backend roles to assign permissions to groups of users based on their IAM role, rather than managing individual user mappings. This is especially important because if your IAM federation or authentication mechanism changes, the backend role mappings make sure of consistent access control within OpenSearch.

Amazon OpenSearch Service network configuration screen with Public access selected

Figure 2: Use public access domain.

Fine-grained access control: Fine-grained access control provides numerous features to help you keep your data secure, such as document-level security, field-level security, read-only users, and OpenSearch Dashboards/Kibana tenants. Fine-grained access control requires a primary user, which is the administrator identity we discuss through the rest of this post.

The primary user is the administrator identity for your OpenSearch domain. This user can set up additional users in Amazon OpenSearch Service, assign roles to them, and assign permissions for those roles. You can choose username and password authentication for the primary user or use an IAM identity. You use these credentials to log in to OpenSearch Dashboards. Following the best practices on choosing your primary user, you should move to an IAM primary user for production workloads.

Fine-grained access control can be applied regardless of how you log in. You can follow your organization’s suggested authentication mechanism and apply fine-grained access control on top of it.

FGAC provides security at multiple levels to meet your security needs:

  • Index-level security – Controls who can create, search, read, write, update, or delete within specific indexes.
  • Document-level security – Restricts which documents within an index a user can see, using OpenSearch query filters (for example, only show documents where department: “sales”).
  • Field-level security – Controls which fields within documents are visible (include or exclude specific fields).
  • Field masking – Anonymizes sensitive field data (for example, hash a release_date or SSN field) rather than hiding it entirely.

Fine-grained access control supports several authentication mechanisms, including HTTP basic authentication using an internal user database, Amazon Cognito for web-based Dashboards access, SAML for enterprise identity provider integration, JSON Web Tokens (JWT) for token-based authentication, and AWS Identity and Access Management with SigV4 signing for IAM users and roles.

Encryption:

Amazon OpenSearch Service encrypts data both in transit and at rest. When you turn on fine-grained access control, encryption is required—the corresponding settings are automatically turned on and can’t be changed. These include Transport Layer Security (TLS 1.2 or later) for requests to the domain and for traffic between nodes in the domain, and encryption of data at rest through AWS Key Management Service (AWS KMS).

For encryption at rest, OpenSearch Service supports three key types: AWS owned keys, AWS managed keys, and customer managed keys. While AWS owned keys provide a quick-start option with no additional configuration, customer managed keys are the recommended best practice. Customer managed keys give you full control over the encryption key lifecycle, including key rotation policies, granular access control through key policies, and the ability to audit key usage through AWS CloudTrail. To use a customer managed key, create a symmetric encryption key in AWS KMS and select it when configuring your domain’s encryption settings.

For a basic public access domain with FGAC, all traffic reaches the domain freely (no VPC restriction), and an open access policy is used so no SigV4 signing is needed. FGAC then takes over, authenticating users through the internal user database (username/password) and enforcing role-based permissions at the index, document, and field level.

The public access configuration we discussed is useful for development and testing, but for production workloads, a best practices deployment combines VPC access, IAM-based authentication, and fine-grained access control. This approach layers all three security mechanisms—network isolation, identity verification, and granular permissions—to protect your domain end to end.

VPC access domain

Placing your OpenSearch Service domain inside a VPC restricts network-level access to resources within the VPC or connected networks. Traffic between your applications and the OpenSearch endpoint doesn’t traverse the public internet, and you can use security groups to further limit which entities can communicate with the domain. OpenSearch Service places a VPC endpoint (VPCe) using AWS PrivateLink into one, two, or three subnets of your VPC depending on your Availability Zone configuration. For high availability (HA), turn on multiple Availability Zones with each subnet in a different zone within the same AWS Region. For more details, see Launching your Amazon OpenSearch Service domains within a VPC.

For this best practices deployment, we use an IAM primary user with Amazon Cognito authentication for OpenSearch Dashboards and for fine-grained access control. We configure a primary IAM role and a limited IAM role, associate them with users in Amazon Cognito through a user pool and identity pool, and then use fine-grained access control to manage permissions. The primary user can then sign in to OpenSearch Dashboards, create backend roles, map the limited user to a restricted role, and enforce granular access at the index, document, and field level. For more details, see Tutorial: Configure a domain with an IAM master user and Amazon Cognito authentication in the AWS documentation.

The following high-level steps detail what’s needed to configure a VPC access domain with Amazon Cognito users. These steps use the Amazon Cognito user pool for authentication. The same basic process works for any Cognito authentication provider that lets you assign different IAM roles to different users.

  • Create an Amazon Cognito user pool.
  • Add users in the user pool for the primary user and a limited-access user.
  • Create an Amazon Cognito identity pool.
  • Update the IAM role for the primary user to allow access to OpenSearch Dashboards.
  • Create an IAM role for the limited user.
  • Create the domain.

You can follow Creating and managing Amazon OpenSearch Service domains in the AWS documentation to provision a domain. The following sections describe some important attributes for the domain.

Network:

VPC access. Public access isn’t recommended for production workloads. We recommend that you use VPC access for all production workloads. Pick the VPC, subnets, and security group that you have created for the OpenSearch domain.

Amazon OpenSearch Service network configuration screen with VPC access selected and VPC, subnet, and security group fields populated

Figure 4: Use VPC access.

Fine-grained access control:

Turn on fine-grained access control with OS[MasterUserRole] as the primary user. You can follow steps in Tutorial: Configure a domain with an IAM master user and Amazon Cognito authentication to create OS[MasterUserRole].

Amazon OpenSearch Service fine-grained access control configuration with an IAM ARN selected as the primary user

Figure 5: Turn on fine-grained access control with an IAM role.

Fine-grained access control provides numerous features to help you keep your data secure, such as document-level security, field-level security, read-only users, and OpenSearch Dashboards/Kibana tenants. Fine-grained access control requires a primary user.

The primary user is the administrator identity for your OpenSearch domain. This user can set up additional users in Amazon OpenSearch Service, assign roles to them, and assign permissions for those roles. You can choose username and password authentication for the primary user or use an IAM identity. You use these credentials to log in to OpenSearch Dashboards. Following the best practices on choosing your primary user, you should choose an IAM primary user for production workloads.

Fine-grained access control can be applied regardless of how you log in. You can follow your organization’s suggested authentication mechanism and apply fine-grained access control on top of it.

Amazon Cognito authentication:

To turn on Amazon Cognito authentication, select Enable Amazon Cognito authentication and choose the Amazon Cognito user pool and Amazon Cognito identity pool for your OpenSearch Dashboards.

Amazon OpenSearch Service authentication configuration with Amazon Cognito enabled and a user pool and identity pool selected

Figure 6: Turn on Amazon Cognito authentication.

Access policy:

The access policy controls whether a request is accepted or rejected when it reaches the Amazon OpenSearch Service domain. You can configure a domain-level access policy to allow access to your Amazon OpenSearch Service domain.

Amazon OpenSearch Service domain access policy editor showing a JSON policy granting access to the configured IAM principals

Figure 7: Configure domain-level access to the domain.

Encryption:

Amazon OpenSearch Service encrypts data both in transit and at rest. When you turn on fine-grained access control, encryption is required—the corresponding settings are automatically turned on and can’t be changed. These include Transport Layer Security (TLS 1.2 or later) for requests to the domain and for traffic between nodes in the domain, and encryption of data at rest through AWS KMS.

For encryption at rest, OpenSearch Service supports three key types: AWS owned keys, AWS managed keys, and customer managed keys. While AWS owned keys provide a quick-start option with no additional configuration, customer managed keys are the recommended best practice. Customer managed keys give you full control over the encryption key lifecycle, including key rotation policies, granular access control through key policies, and the ability to audit key usage through AWS CloudTrail. To use a customer managed key, create a symmetric encryption key in AWS KMS and select it when configuring your domain’s encryption settings.

With these configurations, you can configure your Amazon OpenSearch domain and OpenSearch Service Dashboards so that they’re accessible only within the chosen VPC. For your production scenario, you can follow your organization’s approved mechanism to access the resources in a VPC. You can access OpenSearch Service Dashboards with a primary user to create a limited-access role and map it to the IAM role with limited access to validate fine-grained access control.

Conclusion

In this post, we looked at the important security configurations for a public and a VPC-based Amazon OpenSearch domain. You can examine more settings for fine-grained access control in the OpenSearch Dashboards Security section.

If you have feedback about this post, submit comments in the Comments section. If you have questions about this post, start a new thread on the Amazon OpenSearch Service forum or contact AWS Support.


About the author

Imtiaz (Taz) Sayed

Imtiaz (Taz) Sayed

Imtiaz (Taz) Sayed is the WW Tech Leader for Analytics at AWS. He enjoys engaging with the community on all things data and analytics. He can be reached through LinkedIn.

Narendra Gupta

Narendra Gupta

Narendra is a Specialist Solutions Architect at AWS, helping customers on their cloud journey with a focus on AWS analytics services. Outside of work, Narendra enjoys learning new technologies, watching movies, and visiting new places.

Akhilesh Dube

Akhilesh Dube

Akhilesh is a Senior Analytics Solutions Architect at AWS. He possesses more than two decades of expertise in working with databases and analytics products. His primary role involves collaborating with enterprise clients to design robust data analytics solutions while offering comprehensive technical guidance on a wide range of AWS Analytics and AI/ML services.

Nishchai JM

Nishchai JM

Nishchai is an Analytics and GenAI Specialist Solutions Architect at Amazon Web services. He specializes in building larger scale distributed applications and help customer to modernize their workload on Cloud. He thinks Data is new oil and spends most of his time in deriving insights out of the Data.

Building a scalable user search layer on top of Amazon Cognito

Post Syndicated from Philip Chen original https://aws.amazon.com/blogs/architecture/building-a-scalable-user-search-layer-on-top-of-amazon-cognito/

Imagine a teammate who needs to find a user across thousands of accounts with only a partial email address, a last name, and a known access level. How quickly can your team respond? If your use case involves straightforward searches on standard Amazon Cognito attributes, the built-in ListUsers API is likely all you need. But for advanced scenarios involving custom attributes, fuzzy matching, complex filtering, and sub-second response times, a dedicated search layer is the right investment.

Amazon Cognito provides robust user authentication and management capabilities for modern applications. As applications scale, development teams typically implement advanced search functionality to find users by partial email match, segment group membership, or audit across multiple custom attributes.

In this post, we show how to build a comprehensive scalable user search layer on top of Amazon Cognito using AWS Lambda, Amazon DynamoDB, and Amazon OpenSearch Service.

Solution overview

This solution extends Amazon Cognito with advanced search capabilities using AWS Lambda, Amazon DynamoDB, and Amazon OpenSearch Serverless.

Key capabilities:

  • Multiple search types: Exact match, prefix match, and fuzzy search
  • Complex filtering: Query across email, phone, groups, and registration date simultaneously
  • High performance: Sub-second response times at any scale
  • Automatic synchronization: Real-time updates as users authenticate or update profiles
  • API-driven: RESTful API with pagination support

The architecture uses Cognito Lambda triggers to capture user data during authentication, stores it in DynamoDB, and indexes it in OpenSearch Serverless through DynamoDB Streams. The following architecture diagram illustrates how these components work together.

Figure 1: Solution architecture for Searchable Cognito Users

Walkthrough

The solution architecture demonstrates two flows: Ingestion flow and Search flow.

Ingestion flow

The ingestion flow captures and indexes user data through two paths: Cognito Lambda triggers and AWS CloudTrail. Together, these paths maintain synchronization between the search index and Cognito without requiring manual intervention or scheduled batch jobs.

1. Cognito Lambda triggers

This path captures user data during authentication events using a Cognito trigger Lambda function that handles two trigger types: Post-confirmation and Pre-token generation. The post-confirmation trigger creates the initial user record on sign-up, while the pre-token generation trigger tracks login activity and app client information on each subsequent authentication. The pre-token generation trigger also provides access to the user’s group membership in the event payload, which is indexed as a searchable field. The flow operates through the following steps:

  1. Client initiates sign-up or login — User submits authentication request to Amazon Cognito.
  2. Post-confirmation trigger — On sign-up, Cognito invokes the Cognito trigger Lambda which creates the initial user record in the DynamoDB user table with profile attributes (email, name, groups).
  3. Pre-token generation trigger — On each login, Cognito invokes the Cognito trigger Lambda which updates the user’s login timestamp and app client information in the DynamoDB user table.
  4. Stream processing — DynamoDB Streams detects the new or updated record and triggers the OSS ingest Lambda.
  5. Index updated — OSS ingest Lambda processes the stream event and indexes the user data in OpenSearch Serverless.

Note: The Cognito Lambda triggers are deployed in a VPC. Cognito enforces a 5-second timeout on trigger functions. If you’re extending these triggers with additional functionality or already using post-confirmation or pre-token generation triggers, ensure the combined execution time stays well within this limit. Consider provisioned concurrency if cold starts are a concern.

Figure 2: User Data Ingestion via Cognito Lambda Triggers

2. CloudTrail

This path captures admin-initiated user changes that occur outside the authentication flow, such as creating users using the Cognito console or CLI. These actions don’t trigger Cognito Lambda triggers, so CloudTrail and EventBridge bridge the gap. The flow operates through the following steps:

  1. Admin action performed — User performs an admin action in Amazon Cognito (for example, create user, update attributes, add to group, disable user).
  2. API call logged — AWS CloudTrail captures the Cognito admin API call.
  3. EventBridge rule matched — An Amazon EventBridge rule matches the Cognito admin event.
  4. CloudTrail event Lambda invoked — EventBridge invokes the CloudTrail event consumption Lambda, which reads the current user state from Cognito and upserts the profile in the DynamoDB user table.
  5. Stream change event — DynamoDB Streams emits the change event.
  6. Invoke OSS Lambda — The stream event triggers the OSS ingest Lambda.
  7. Index user data — OSS ingest Lambda indexes the updated user data in OpenSearch Serverless.

Figure 3: User Data Ingestion via CloudTrail

Figure 4: Data model for indexed user attributes in Amazon DynamoDB

Search flow

With the search flow, authorized users can query the indexed user directory:

  1. Query submission — Authenticated user submits search query through the UI.
  2. Request validation — API Gateway receives the request with the Cognito JWT token and validates it using the Cognito authorizer.
  3. Search execution — Upon successful validation, the search Lambda function is invoked with the search parameters.
  4. OpenSearch query — Lambda assumes a read-only role for OpenSearch Service access and executes the query against the OpenSearch Serverless index.
  5. Results returned — Lambda formats and returns the query results to the frontend, where the UI displays them in a paginated format.

Figure 5: Search Flow Sequence Diagram

Figure 6: Demo UI user search integration on multiple properties

Figure 7: Demo UI user search integration on auto-suggest

Try it yourself

Ready to see this solution in action? The repository includes everything you need to deploy a complete working implementation in your own AWS environment.

The source code for this solution is available on GitHub at: https://github.com/aws-samples/sample-user-search-layer-for-cognito.

The repository includes everything you need: AWS CDK infrastructure code, Lambda function implementations, a React frontend, and documentation. You can have a fully functional searchable user directory running in your account in under 20 minutes. When you’re finished testing, clean up all resources to avoid ongoing charges.

Conclusion

In this post, you learned how to extend Amazon Cognito with advanced search capabilities. By combining OpenSearch Serverless, DynamoDB Streams, and Lambda functions, you can build a scalable, event-driven architecture that automatically maintains a searchable user directory with sub-second query performance.

This pattern unlocks powerful use cases: support teams can quickly locate users across thousands of accounts, administrators can segment users by group membership for targeted communications, and compliance teams can audit user attributes with complex filtering.

To dive deeper into the AWS services powering this solution:

About the authors

AWS Weekly Roundup: Claude Opus 4.8 on AWS, Aurora MySQL with Kiro Powers, and more (June 1, 2026)

Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-opus-4-8-on-aws-aurora-mysql-with-kiro-powers-and-more-june-1-2026/

In my last Week in Review post, I shared what I’d been hearing from customers in the AI-Driven Development Lifecycle (AI-DLC) workshops I’ve been delivering. Last week I was back at it, this time in Denver for a two-day AI-DLC workshop, where I helped facilitate 17 teams to deliver nearly 20 separate use cases in just two days. The pace of acceleration that AI-DLC unlocks—especially when paired with tools like Claude Code on Amazon Bedrock—is fundamentally changing how businesses operate. Traditional roles within software development teams are collapsing into smaller, AI-augmented squads, and the paradigm shift is beginning to take place right in front of us. To learn more about how to utilize various AI tools, visit the GitHub repository of AI-DLC workflow.

This shift is also reshaping how AWS account teams (solutions architects, customer solutions managers, and technical account managers) collaborate with customers. It’s becoming less about handing off advisory design documents and more about building alongside them in real time. It’s a genuinely exciting moment to be in the middle of the change, and this week’s headline launch — Anthropic’s most capable model yet, now on AWS — is going to push that pace even further.

Now, let’s get into this week’s AWS news…

Headlines
Claude Opus 4.8 on AWS — Anthropic’s most capable generally available model is now accessible through both Amazon Bedrock and the Claude Platform on AWS. Opus 4.8 is built for agentic coding, knowledge work, and extended autonomous task execution — it sustains longer autonomous sessions with deeper reasoning, recovers from errors, and synthesizes information across lengthy documents. For coding workloads, it reads codebases like an engineer, plans before it edits, and holds context across long sessions. On Amazon Bedrock, you get AWS-managed features like Guardrails, Knowledge Bases, and data residency; on the Claude Platform on AWS, you get Anthropic’s native APIs unified with AWS billing. To learn more, visit the deep-dive blog post.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Introducing the next generation of AWS Resilience Hub — A reimagined Resilience Hub gives SREs and developers a unified framework to define resilience standards, evaluate applications against them, and demonstrate compliance across an entire portfolio. It introduces modular resilience policies (covering service-level objectives (SLOs), multi-AZ/Region DR, and data recovery), business-oriented application modeling, generative AI-powered assessments aligned with the Well-Architected and Resilience Analysis Frameworks, and automatic dependency discovery via DNS query log analysis. Integration with AWS Organizations enables organization-wide resilience management from a single delegated administrator account.
  • Introducing the next generation of Amazon OpenSearch Serverless for building agentic AI applications — Amazon OpenSearch Serverless is now a fully managed search and vector engine purpose-built for agentic AI applications. It scales from zero to thousands of requests per second—roughly 20x faster than the prior generation—delivers up to 60% cost savings versus peak-provisioned clusters, and adds GPU acceleration plus new SEARCH and VECTORSEARCH collection types. Native integrations with Vercel, Kiro, Claude Code, and Cursor through OpenSearch Agent Skills make it straightforward to plug into your agent stack.
  • New assessment capabilities in AWS Transform — AWS Transform expands with new tools to help you build migration business cases and evaluate TCO before moving workloads to AWS. You can ingest data from RVTools exports, CMDB data, the AWS Transform discovery tool, and third-party discovery tools, then run what-if scenarios across region, utilization, and service mapping for EC2, FSx, S3, SQL Server on EC2, and virtual desktops. The release also adds Agentic Readiness Analysis (ARA) and Modernization Analysis (MODA), which scan code repositories in 5 to 30 minutes per repo to surface severity-tagged findings with file-level evidence and AWS-mapped remediation guidance.
  • Amazon Aurora MySQL with Kiro Powers — Aurora MySQL now integrates with Kiro Powers, drawing from a curated repository of pre-packaged MCP servers, steering files, and hooks validated by Kiro partners. Developers can execute both data plane tasks (queries, schema management) and control plane tasks (cluster management) in natural language, with dynamic guidance for Aurora MySQL Serverless scaling, RDS-to-Aurora migration, and replication setup. The companion Database Blog post explains how the agent produces the API calls, SQL, and configuration for you to review and run — available via one-click install from the Kiro IDE or webpage.
  • Amazon WorkSpaces Applications now supports Windows Desktop OS — You can now bring your own Windows Desktop licenses to Amazon WorkSpaces Applications and stream full Windows desktops and applications from AWS-hosted dedicated hardware. BYOL eliminates OS fees (you pay only for compute and streaming infrastructure), supports eligible Microsoft 365 Apps for enterprise, and gives users a matching experience between local and remote environments — same workflows, shortcuts, and navigation in both.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.

Learn more about AWS, browse and join upcoming AWS-led in-person and virtual events, startup events, and developer-focused events as well as AWS Summits and AWS Community Days. Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development.

That’s all for this week. Check back next Monday for another Weekly Roundup!

-Micah

Introducing the next generation of Amazon OpenSearch Serverless for building your agentic AI applications

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-the-next-generation-of-amazon-opensearch-serverless-for-building-your-agentic-ai-applications/

Today, we’re announcing the next generation of Amazon OpenSearch Serverless, a fully managed search and vector engine designed for customers building AI agents. The next generation of OpenSearch Serverless scales from zero to thousands of requests per second and back to zero when idle, offering up to 60% cost savings compared to the cost of OpenSearch Service clusters provisioned for peak capacity.

The next generation of OpenSearch Serverless creates resources in seconds and scales capacity up to 20 times faster than the previous generation. With instant resource creation and native integrations with AI development platforms like Vercel and Kiro, you can deploy production-ready search and vector backends for your AI agents in minutes without managing infrastructure.

The next generation of OpenSearch Serverless in action
To get started with the next generation of OpenSearch Serverless, choose Create collection in the Serverless menu in the Amazon OpenSearch Service console.

Create NextGen collection with instant auto scaling and scale-to-zero for cost optimization. At launch, we support full-text search and vector search only for the collection type. If you want to use the existing OpenSearch Serverless infrastructure, choose Switch to Classic.

Choose Express create, the fastest way to create collection. No configuration is required—the default settings and matching security policies are applied automatically. Some configuration options can be changed later.

When you choose Create collection, OpenSearch Serverless will provision resources in seconds.

You can also create a collection of OpenSearch Serverless with AWS Command Line Interface (AWS CLI) or AWS SDKs. Here is a sample CLI command to create a collection group.

aws opensearchserverless create-collection-group \
    --name channy-nextgen-group \
    --standby-replicas ENABLED \
    --generation NEXTGEN \
    --description "My NextGen collection group" \
    --capacity-limits '{
        "maxIndexingCapacityInOCU": 10,
        "maxSearchCapacityInOCU": 10,
        "minIndexingCapacityInOCU": 0,
        "minSearchCapacityInOCU": 0
    }' \
    --region "us-east-1"

Now, you can create a collection that inherits the generation from its parent collection group. Supported collection types: SEARCH and VECTORSEARCH.

aws opensearchserverless create-collection \
    --name channy-nextgen-collection \
    --type SEARCH \
    --collection-group-name channy-nextgen-group \
    --standby-replicas ENABLED \
    --description "My collection in NextGen group" \
    --region "us-east-1"

To learn more about managing the next generation of OpenSearch Serverless, visit the Amazon OpenSearch Serverless documentation.

Building your agents faster with OpenSearch Serverless
To support building production-ready agent applications in Vercel, you can now create a new OpenSearch collection or connect your existing OpenSearch Serverless collection within the Vercel console. Create a search backend in seconds and add features on-demand as your application grows. To learn more, visit AWS for Vercel.

You can go from idea to working prototype in minutes using Claude Code, Cursor, and Kiro. OpenSearch Agent Skills provide a repository of skills that bring OpenSearch intelligence directly into your agent. Each skill encapsulates domain knowledge, best practices, and multi-step execution logic for a specific workflow–so your agent not only gets results, but understands how they were achieved. You can also use the OpenSearch Launchpad in Kiro Powers to accelerate search applications with guided, end-to-end architecture planning.

Now available
The next generation of Amazon OpenSearch Serverless is generally available today and is available in all AWS commercial Regions where Amazon OpenSearch Serverless is currently available.

The next generation of OpenSearch Serverless charges for the compute you use in OpenSearch Compute Units (OCUs) for indexing, search, and GPU acceleration. You are charged separately for storage in GB-month. For more information, see Amazon OpenSearch Service Pricing.

Give it a try and send feedback to the AWS re:Post for Amazon OpenSearch Service or through your usual AWS Support contacts.

— Channy