All posts by Nikhil Agarwal

Best practices for right-sizing Amazon OpenSearch Service domains

2026-02-18 Nikhil Agarwal

Post Syndicated from Nikhil Agarwal original https://aws.amazon.com/blogs/big-data/best-practices-for-right-sizing-amazon-opensearch-service-domains/

Amazon OpenSearch Service is a fully managed service for search, analytics, and observability workloads, helping you index, search, and analyze large datasets with ease. Making sure your OpenSearch Service domain is right-sized—balancing performance, scalability, and cost—is critical to maximizing its value. An over-provisioned domain wastes resources, whereas an under-provisioned one risks performance bottlenecks like high latency or write rejections.

In this post, we guide you through the steps to determine if your OpenSearch Service domain is right-sized, using AWS tools and best practices to optimize your configuration for workloads like log analytics, search, vector search, or synthetic data testing.

Why right-sizing your OpenSearch Service domain matters

Right-sizing your OpenSearch Service domain provides optimal performance, reliability, and cost-efficiency. An undersized domain leads to high CPU utilization, memory pressure, and query latency, whereas an oversized domain drives unnecessary spend and resource waste. By continuously matching domain resources to workload characteristics such as ingestion rate, query complexity, and data growth, you can maintain predictable performance without overpaying for unused capacity.

Beyond cost and performance, right-sizing facilitates architectural agility. It helps make sure your cluster scales smoothly during traffic spikes, meets SLA targets, and sustains stability under changing workloads. Regularly tuning resources to match actual demand optimizes infrastructure efficiency and supports long-term operational resilience.

Key Amazon CloudWatch metrics

OpenSearch Service provides Amazon CloudWatch metrics that offer insights into various aspects of your domain’s performance. These metrics fall into 16 different categories, including cluster metrics, EBS volume metrics, and instance metrics. To determine if your OpenSearch Service domain is misconfigured, monitor these common symptoms that indicate resizing or optimization may be necessary. These are caused by imbalances in resource allocation, workload demands, or configuration settings. The following table summarizes these parameters:

CloudWatch Metrics	Parameter
CPU Utilization Metrics	`CPUUtilization:` Average CPU usage across all data nodes. Optimal range: 60-80% for sustained workloads Primary control plane CPU utilization (for dedicated primary nodes): Average CPU usage on primary nodes. Optimal range: Under normal conditions <50%
Memory Utilization Metrics	`JVMMemoryPressure:` Percentage of heap memory used across data nodes. Optimal range: 65–85% Note: With Garbage First Garbage Collector (G1GC), JVM may delay collections to optimize performance. Evaluate `JVMMemoryPressure` together with GC metrics (Old Gen usage and GC pause time) to confirm true pressure trends. `MasterJVMMemoryPressure`: Heap usage on dedicated primary nodes. Optimal range: <80% Note: Occasional spikes are normal during state updates; sustained high memory pressure warrants scaling or tuning.
Storage Metrics	`StorageUtilization`: Percentage of storage space used. Optimal range: 70–85% `FreeStorageSpace`: Available storage in MB. Critical threshold: When approaching the read-only threshold.
Node Level Search and Indexing Performance (These latencies are not per-request latencies or rate, but at node level based on shards assigned to a node.)	`SearchLatency`: Average time for search requests. Baseline establishment: Monitor during normal operations. `IndexingLatency`: Average time for indexing operations. Impact: Can indicate CPU or I/O bottlenecks. `SearchRate` and `IndexingRate`: Requests per minute for search and indexing. Usage: Correlate with latency metrics to understand performance impact.
Cluster Health Indicators	`ClusterStatus.yellow` and `ClusterStatus.red`: Yellow status: Some replica shards are unassigned. Red status: Some primary shards are unassigned (data loss risk). Nodes What it measures: Number of nodes in the cluster. Usage: Track node failures and recovery patterns.

Signs of under-provisioning

Under-provisioned domains struggle to handle workload demands, leading to performance degradation and cluster instability. Look for sustained resource pressure and operational errors that signal the cluster is running beyond its limits. For monitoring, you can set CloudWatch alarms to catch early signals of stress and prevent outages or degraded performance. The following are critical warning signs:

High CPU utilization for data nodes (>80%) sustained over time (such as more than 10 minutes)
High CPU utilization for primary nodes (>60%) sustained over time (such as more than 10 minutes)
JVM memory pressure consistently high (>85%) for data and primary nodes
Storage utilization reaching high (>85%)
Increasing search latency with stable query patterns (increasing by 50% from baseline)
Frequent cluster status yellow/red events
Node failures under normal load conditions

When resources are constrained, the end-user experience suffers with slower searches, failed indexing, and system errors. The following are key performance impact indicators:

Search timeouts increasing
Indexing delays growing
Circuit breaker exceptions
Rejected execution exceptions in logs

Remediation recommendations

The following table summarizes CloudWatch metric symptoms, possible causes, and potential solutions.

CloudWatch metric symptom	Causes and solution
`FreeStorageSpace` drops <20%	Storage pressure occurs when data volume outgrows local storage due to high ingestion, long retention without cleanup, or unbalanced shards. Lack of tiering (such as UltraWarm) further worsens capacity issues. Solution: Free up space by deleting unused indexes or automating cleanup with ISM and use force merge on read-only indexes to reclaim storage. If pressure persists, scale vertically or horizontally, use UltraWarm or cold storage for older data, and adjust shard counts at rollover for better balance.
`CPUUtilization` and `JVMMemoryPressure` consistently >70%	High CPU or JVM pressure arises when instance sizes are too small or shard counts per node are excessive, leading to frequent GC pauses. Inefficient shard strategy, uneven distribution, and poorly optimized queries or mappings further spike memory usage under heavy workloads. Solution: Address high CPU/JVM pressure by scaling vertically to larger instances (such as from r6g.large to r6g.xlarge) or adding nodes horizontally. Optimize shard counts relative to heap size, smooth out peak traffic, and use slow logs to pinpoint and tune resource-heavy queries.
`SearchLatency` or `IndexingLatency` spikes >500 milliseconds	Thread pool rejections often stem from resource contention like high CPU/JVM pressure or GC pauses. Inefficient shard sizing, over-sharding, and overly complex queries (deep aggregations, frequent cache evictions) further increase overhead and push tasks into rejection. Solution: Reduce query latency by optimizing queries with profiling, tuning shard sizes (10–50 GB each), and avoiding over-sharding. Improve parallelism by scaling the cluster, adding replicas for read capacity, increasing cache through larger nodes, and setting appropriate query timeouts.
`ThreadpoolRejected` metrics indicate queued requests	Thread pool rejections occur when high concurrent requests overflow queues beyond capacity, especially with undersized nodes limited by vCPU-based threads. Sudden unscaled traffic spikes further overwhelm pools, causing tasks to be dropped or delayed. Solution: Mitigate thread pool rejections by enforcing shard balance across nodes, scaling horizontally to boost thread capacity, and managing client load with retries and reduced concurrency. Monitor search queues, right-size instances for vCPUs, and cautiously tune thread pool settings to handle bursty workloads.
`ThroughputThrottle` or `IopsThrottle` reach 1	I/O throttling arises when Amazon EBS or Amazon EC2 limits are exceeded, such as gp3’s 125 MBps baseline, or when burst credits are depleted due to sustained spikes. Mismatched volume types and heavy operations like bulk indexing without optimized storage further amplify throughput bottlenecks. Solution: Address I/O throttling by upgrading to gp3 volumes with higher baseline or provisioning extra IOPS and consider I/O-optimized instances like i3/i4 families while monitoring burst balance. For sustained workloads, scale nodes or schedule heavy operations during off-peak hours to avoid hitting throughput caps.

Signs of over-provisioning

Over-provisioned clusters show consistently low utilization across CPU, memory, and storage, suggesting resources far exceed workload demands. Identifying these inefficiencies helps reduce unnecessary spend without impacting performance. You can use CloudWatch alarms to track cluster health and cost-efficiency metrics over 2–4 weeks to confirm sustained underutilization:

Low CPU utilization for data and primary nodes (<40%) sustained over time
Low JVM memory pressure for data and primary nodes (<50%)
Excessive free storage (>70% unused)
Underutilized instance types for workload patterns

Monitor cluster indexing and search latencies constantly as the cluster is being downsized—these latencies should not increase if the cluster is eliminating unused capacity. Also, it’s recommended to reduce nodes one at a time and continue to observe latencies to continue further downturn. By right-sizing instances, reducing node counts, and adopting cost-efficient storage options, you can align resources to actual usage. Optimizing shard allocation further supports balanced performance at a lower cost.

Best practices for right-sizing

In this section, we discuss best practices for right-sizing.

Iterate and optimize

Right-sizing is an ongoing process, not a one-time exercise. As workloads evolve, continuously monitor CPU, JVM memory pressure, and storage utilization using CloudWatch to make sure they remain within healthy thresholds. Rising latency, queue buildup, or unassigned shards often signal capacity or configuration issues that require attention.

Regularly review slow logs, query latency, and ingestion trends to identify performance bottlenecks early. If search or indexing performance degrades, consider scaling, rebalancing shards, or adjusting retention policies. Periodic reviews of instance sizes and node count help align cost with demand, maintaining 200-millisecond latency targets while avoiding over-provisioning. Consistent iteration helps your OpenSearch Service domain remain performant and cost-efficient over time.

Establish baselines

Monitor for 2–4 weeks after initial deployment and document peak usage patterns and seasonal variations. Record performance during different workload types. Set appropriate CloudWatch alarm thresholds based on your baselines.

Regular review process

Conduct weekly metric reviews during initial optimization and monthly assessments for stable workloads. Conduct quarterly right-sizing exercises for cost optimization.

Scaling strategies

Consider the following scaling strategies:

Vertical scaling (instance types) – Use larger instance types when performance constraints stem from CPU, memory, or JVM pressure, and overall data volume is within a single node’s capacity. Choose memory-optimized instances (such as r8g, r7g, or r7i) for heavy aggregation or indexing workloads. Use compute-optimized instances (c8g, c7g, or c7i) for CPU-bound workloads such as query-heavy or log-processing environments. Vertical scaling is ideal for smaller clusters or testing environments where simplicity and cost-efficiency are priorities.

Horizontal scaling (node count) – Add more data nodes when storage, shard count, or query concurrency increases beyond what a single node can handle. Maintain an odd number of primary-eligible nodes (typically three or five) and use dedicated primary nodes for clusters with more than 10 data nodes. Deploy across three Availability Zones for high availability in production. Horizontal scaling is preferred for large, production-grade workloads requiring fault tolerance and sustained growth. Use _cat/allocation?v to verify shard distribution and node balance:

GET /_cat/allocation/node_name_1,node_name_2,node_name_3

Optimize storage configuration

Use the latest generation of Amazon EBS General Purpose (gp) volumes for improved performance and cost-efficiency compared to earlier versions. Monitor storage growth trends using ClusterUsedSpace and FreeStorageSpace metrics. Maintain data utilization below 50% of total storage capacity to allow for growth and snapshots.

Choose storage tiers based on performance and access patterns—for example, enable UltraWarm or cold storage for large, infrequently accessed datasets. Move older or compliance-related data to cost-efficient tiers (for analytics or WORM workloads) only after ensuring the data is immutable.

Use the _cat/indices?v API to monitor index sizes and refine retention or rollover policies accordingly:

GET /_cat/indices/index1,index2,index3

Analyze shard configuration

Shards directly affect performance and resource usage, so an appropriate shard strategy should be used. The indexes that have heavy ingestion and searches should have a number of shards in the order of number of nodes for better efficiency across all data nodes in the cluster. We recommend keeping shard sizes between 10–30 GB for search workloads and up to 50 GB for log analytics workloads and limit to <20 shards per GB of JVM heap.

Run _cat/shards?v to confirm even shard distribution and no unassigned shards. Evaluate over-sharding by checking JVMMemoryPressure (>80%) or SearchLatency spikes (>200 milliseconds) from excessive shard coordination. Assess under-sharding if IndexingLatency (>200 milliseconds) or low SearchRate indicates limit parallelism. Use _cat/allocation?v to identify unbalanced shard sizes or hot spots on nodes:

GET /_cat/allocation/node_name_1,node_name_2,node_name_3

Handling unexpected traffic spikes

Even well right-sized OpenSearch Service domains can face performance challenges during sudden workload surges, such as log bursts, search traffic peaks, or seasonal load patterns. To handle such unexpected spikes effectively, consider implementing the following best practices:

Enable Auto-Tune – Automatically adjust cluster settings based on current usage and traffic patterns
Distribute shards effectively – Avoid shard hotspots by using balanced shard allocation and index rollover policies
Pre-warm clusters for known events – For expected peak periods (end-of-month reports, marketing campaigns), temporarily scale up before the spike and scale down afterward
Monitor with CloudWatch alarms – Set proactive alarms for CPU, JVM memory, and thread pool rejections to catch early stress indicators

Deploy CloudWatch alarms

CloudWatch alarms perform an action when a CloudWatch metric exceeds a specified value for some amount of time to take remediation action proactively.

Conclusion

Right-sizing is a continuous process of observing, analyzing, and optimizing. By using CloudWatch metrics, OpenSearch Dashboards, and best practices around shard sizing and workload profiling, you can make sure your domain is efficient, performant, and cost-effective. Right-sizing your OpenSearch Service domain helps provide optimal performance, cost-efficiency, and scalability. By monitoring key metrics, optimizing shards, and using AWS tools like CloudWatch, ISM, and Auto Scaling, you can maintain a high-performing cluster without over-provisioning.

For more information about right-sizing OpenSearch Service domains, refer to Sizing Amazon OpenSearch Service domains.

How to track Amazon OpenSearch Service domain-level cost

2024-09-19 Nikhil Agarwal

Post Syndicated from Nikhil Agarwal original https://aws.amazon.com/blogs/big-data/how-to-track-amazon-opensearch-service-domain-level-cost/

Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch domains in AWS to perform interactive log analytics, real-time application monitoring, website search, and more. Understanding OpenSearch service spend per domain is crucial for effective cost management, optimization, and informed decision-making. Amazon OpenSearch Service Pricing is based on three dimensions: instances, storage, and data transfer. Storage pricing depends on the chosen storage type and also the storage tier. Visibility into domain-level charges enables accurate budgeting, efficient resource allocation, fair cost attribution across projects, and overall cost transparency.

In this post, we show you how to view the OpenSearch Service domain-level cost using AWS Cost Explorer. For example, the account in the following screenshot has five OpenSearch Service domains deployed.

Using AWS Cost Explorer, you can see the cost at the service level by default but not at an individual domain level. However, users can still breakdown the cost using a dimension like Usage type. The simplest approach to gain domain level visibility is by enabling resource-level data in AWS Cost Explorer. There are no additional charges for enabling resource-level data at daily granularity in AWS Cost Explorer.

If you need domain-level cost data beyond 14 days then either you can setup a Data Export/CUR or you can use user-defined cost allocation tags. User-defined cost allocation tags offer benefits such as cost categorization and cost allocation to categorize and group your AWS costs across cost centers and based on criteria that are meaningful to your organization, such as projects, departments, environments, or applications. This provides better visibility and granularity into your cost breakdown compared to just looking at resource-level costs.

Overview

This post demonstrates how to use user-defined cost allocation tags attached to a cluster using these high-level steps:

Add a user-defined cost allocation tag to an OpenSearch Service domain
Activate the user-defined cost allocation tag
Analyze OpenSearch Service domain costs using AWS Cost Explorer and tags

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account
Amazon OpenSearch Service domains
Access to the AWS Billing and Cost Management dashboard and AWS Cost Explorer

1. Add a user-defined cost allocation tag to an OpenSearch Service domain

The user-defined cost allocation tags are key-value pairs and user will need to define both a key and a value to an OpenSearch Service domain using one of the following methods:

AWS Management Console
AWS Command Line Interface (AWS CLI)
Amazon OpenSearch Service configuration API
AWS SDK
AWS CloudFormation or Terraform

AWS Management Console

To add a user-defined cost allocation tag using the AWS Management Console, follow these steps:

In the AWS Management Console, under Analytics, choose Amazon OpenSearch Service.
Select the domain you want to add tags to and go to the Tags
Choose Add tags and then Add new tag.
Enter a tag and an optional value.
Choose Save.

The following screenshot shows the Add tags window.

AWS CLI

To add a user-defined cost allocation tag using the AWS CLI, you can use the aws opensearch add-tags command to add tags to an OpenSearch Service domain. The command requires the domain Amazon Resource Name (ARN) and a list of tags to be added. Use the following syntax.

add-tags --arn=<domain_arn> --tag-list Key=<key>,Value=<value>

Example:

aws opensearch add-tags –arn arn:aws:es:us-east-1:123456789123:domain/opensearchtestdomain –tag-list Key=opensearchdomain,Value=opensearchtestdomain

Amazon OpenSearch Service configuration API

You can use the Amazon OpenSearch Service configuration API to create, configure, and manage OpenSearch Service domains. Use the following AddTags command to tag an OpenSearch Service domain.

POST /2021-01-01/tags HTTP/1.1 
Content-type: application/json 
{ 
    "ARN": "arn:aws:es:us-east-1:123456789123:domain/opensearchtestdomain", 
    "TagList": [ 
        { 
            "Key": "opensearchdomain", 
            "Value": "opensearchtestdomain" 
        } 
    ] 
}

AWS SDK

You can programmatically add tags to an OpenSearch Service domain using the AWS OpenSearch SDK. The SDK provides methods to interact with Amazon OpenSearch Service API and manage tags. For example, Python client can use the client.add_tags command to tag a domain. You must provide values for domain_arn, tag_key, and tag_value.

import boto3 
client = boto3.client('opensearch') 
response = client.add_tags ( 
    ARN = ‘arn:aws:es:us-east-1:123456789123:domain/opensearchtestdomain’, 
    TagList=[ 
    { 
        ‘Key’: ‘opensearchdomain’, 
        ‘Value’: ‘opensearchtestdomain’ 
    } 
  ] 
)

AWS CloudFormation or Terraform

When provisioning an OpenSearch Service domain using CloudFormation or Terraform, you can define the tags as part of the resource configuration by using AWS::OpenSearchService::Domain Tag.

Resources 
    OpenSearchDomain: 
        Type: AWS::OpenSearchService::Domain 

Properties
    DomainName: arn:aws:es:us-east-1:123456789123:domain/opensearchtestdomain

Tags 
    - Key: opensearchdomain 
    - Value: opensearchtestdomain

After applying a user-defined tag to the OpenSearch Service domain, use the following AWS CLI command to verify that the tag has been applied.

aws opensearch list-tags –arn <ARN>

Example:

aws opensearch list-tags –arn arn:aws:es:us-east-1:123456789123:domain/opensearchtestdomain

Troubleshooting

The add-tags command can fail in the following scenarios, so make sure all the values are entered correctly:

Invalid resource ARN – The command will fail if the provided ARN for the OpenSearch Service domain is invalid or does not exist.
Insufficient permissions – Verify that the IAM user or role you’re using to run the OpenSearch Service commands has the necessary permissions to access the OpenSearch Service domain and perform the desired actions, such as adding tags.
Exceeded tag limit – The OpenSearch Service domain has limit of up to 10 tags, so if the number of tags you are trying to add exceeds this limit, the command will fail.

For ease of use and best results, use the Tag Editor to create and apply user-defined tags. The Tag Editor provides a central, unified way to create and manage your user-defined tags. For more information, refer to Working with Tag Editor in the AWS Resource Groups User Guide.

2. Activate the user-defined cost allocation tag

User-defined cost allocation tags are tags that you define, create, and apply to resources, and it may take up to 24 hours for the tag keys to appear on your cost allocation tags page for activation in the Billing and Cost Management console.
After you select your tags for activation, it can take an additional 24 hours for tags to activate and be available for use in Cost Explorer. Use the following steps to activate the user-defined cost allocation tags you created in previous steps.

As shown in the following screenshot, on the Billing and Cost Management dashboard, in the navigation pane, select Cost Allocation Tags.
To activate the tag, under User-defined cost allocation tags, enter opensearchdomain to search for your tag name, select it, and choose Activate. This confirms that Cost Explorer and your AWS Cost and Usage Reports (CUR) will include these tags.

In general, cost allocation tags cannot be deleted and can only be deactivated. However, you can exclude the tag that you do not want in the CUR report or in AWS Cost Explorer and only include tags that are needed.

3. Analyze OpenSearch Service domain cost using AWS Cost Explorer and tags

AWS Cost Explorer only displays tags starting from the date when you have enabled user-defined cost allocation tags and not from when the resource was tagged. Therefore, even if your resources had tags for a long time, AWS Cost Explorer will show “No tag key” for all of the previous days until the date when tag was enabled, but users can request to backfill tags. To analyze OpenSearch Service domain costs using AWS Cost Explorer and tags, follow these steps:

On the Billing and Cost Management console, in the navigation pane, under Cost analysis, choose Cost Explorer.
In the Report parameters help panel on the right, under Group by, for Dimension, select Tag. Under Tag, choose the opensearchtestdomain tag key that you created.
Under Applied filters, choose OpenSearch Service.

The following screenshot shows the CUR dashboard.

Costs

There is no additional fee or charge for using the user-defined cost allocation tags in AWS Cost Explorer. However, an excessive number of tags can increase the size of your CUR file. Your CUR file contains your usage and cost data, including tags you apply, so more tags mean more data in the file. CUR data is stored in Amazon Simple Storage Service (Amazon S3), so larger CUR file could increase storage cost.

The best practice is to be selective about which tags you enable and how many you use. Start with tags that provide the most value for attributes such as cost allocation and analytics. Monitor your CUR file size over time and add and remove tags thoughtfully.

Conclusion

This post outlines a solution for AWS customers to gain visibility into their OpenSearch Service workload costs on a per-domain basis using AWS Cost Explorer and user-defined cost allocation tags. This approach enables greater cost transparency and control, making it easier to allocate costs accurately and make informed decisions about Amazon OpenSearch service workload usage. The process involves adding a cost allocation tag to each OpenSearch Service domain, activating the user-defined tag, and then analyzing the costs in AWS Cost Explorer based on the tag. By implementing this solution, customers can obtain granular insights into OpenSearch Service workload costs at the domain level, facilitating precise cost attribution and better alignment of costs with business requirements.

For more resources, refer to the following:

About the Authors

Nikhil Agarwal is a Sr. Technical Manager with Amazon Web Services. He is passionate about helping customers achieve operational excellence in their cloud journey and actively working on technical solutions. He is an artificial intelligence (AI/ML) and analytics enthusiastic, he deep dives into customer’s ML and OpenSearch service specific use cases. Outside of work, he enjoys traveling with family and exploring different gadgets.

Rick Balwani is an Enterprise Support Manager responsible for leading a team of Technical Account Mangers (TAMs) supporting AWS independent software vendor (ISV) customers. He works to ensure customers are successful on AWS and can build cutting-edge solutions. Rick has a background in DevOps and system engineering.

Ashwin Barve is a Sr. Technical Manager with Amazon Web Services. In his role, Ashwin leverages his experience to help customers align their workloads with AWS best practices and optimize resources for maximum cost savings. Ashwin is dedicated to assisting customers through every phase of their cloud adoption, from accelerating migrations to modernizing workloads.

Amazon CloudWatch metrics for Amazon OpenSearch Service storage and shard skew health

2023-08-21 Nikhil Agarwal

Post Syndicated from Nikhil Agarwal original https://aws.amazon.com/blogs/big-data/amazon-cloudwatch-metrics-for-amazon-opensearch-service-storage-and-shard-skew-health/

Amazon OpenSearch Service is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in AWS to perform interactive log analytics, real-time application monitoring, website search, and more. OpenSearch is an open source, distributed search and analytics suite.

When working with OpenSearch Service, shard strategy is key. Shards distribute your workload across the data nodes of your cluster. When creating an index, you tell OpenSearch Service how many primary shards to create and how many replicas to create of each shard. The primary shards are independent partitions of the full dataset. OpenSearch Service automatically distributes your data across the primary shards in an index. Our recommendation is to use two replicas for your index. For example, if you set your index’s shard count to three primary shards and two replicas, you will have a total of nine shards. Properly configured indexes can help boost overall domain performance, whereas a misconfigured index will lead to storage and performance skew.

OpenSearch Service distributes the shards in your indexes to the data nodes in your domain, ensuring that no primary shard and its replicas are placed on the same node. The data for the shards are stored in the node’s storage. If your indexes (and therefore their shards) are very different sizes, the storage used on the data nodes in the domain will be unequal, or skewed. Storage skew leads to uneven memory and CPU utilization, intermittent and uneven latency, and uneven queueing and rejecting of requests. Therefore, it’s important to configure and maintain indexes such that shards can be distributed evenly across the data nodes of your cluster.

In this post, we explore how to deploy Amazon CloudWatch metrics using an AWS CloudFormation template to monitor an OpenSearch Service domain’s storage and shard skew. This solution uses an AWS Lambda function to extract storage and shard distribution metadata from your OpenSearch Service domain, calculates the level of skew, and then pushes this information to CloudWatch metrics so that you can easily monitor, alert, and respond.

Solution overview

The solution and associated resources are available for you to deploy into your own AWS account as a CloudFormation template. The template deploys the following resources:

An AWS Identity and Access Management (IAM) role for the Lambda function called OpensearchSkewMetricsLambdaRole. This allows write access to CloudWatch metrics and access to the CloudWatch log group and OpenSearch APIs.
An AWS Lambda function called Opensearch-SkewMetricsPublisher-py.
An Amazon CloudWatch log group for the Lambda function called /aws/lambda/Opensearch-skewmetrics-publisher-py.
An Amazon EventBridge rule for the Lambda function called EventRuleForOSSkew.
The following CloudWatch metrics for the Lambda function:
- aws_/<region-name>/<MetricIdentifier>/_storagemetric
- aws_/<region-name>/<MetricIdentifier>/_shardmetric

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account.
An OpenSearch Service domain.
This post requires you to add a Lambda role to the OpenSearch Service domain’s security configuration access policy. If your domain is using fine-grained access control, then you need to follow the steps as described in the section Mapping roles to users to enable access for the newly deployed Lambda execution role to the domain after deploying the CloudFormation template.

Deploy the CloudFormation template

To deploy the CloudFormation template, complete the following steps:

Log in to your AWS account.
Select the Region where you’re running your OpenSearch Service domain.
To launch your CloudFormation stack, choose Launch Stack
For Stack name, enter a name for the stack (maximum length 30 characters).
For MetricIdentifier, enter a unique identifier that will help you identify the custom CloudWatch metrics for your domain.
For OpensearchDomainURL, enter the domain endpoint that you are monitoring.
Choose Next.
Select I acknowledge that AWS CloudFormation might create IAM resources, then choose Create stack.
Wait for the stack creation to complete.
On the Lambda console, choose Functions in the navigation pane.
Choose the Lambda function called Opensearch-SkewMetricsPublisher-py-<stackname>.
In the Code section, choose Test.
Keep the default values for the test event and run a quick test.

Make sure to grant the Lambda execution role permission to the OpenSearch Service domain’s resource-based policy, if you are using one. If fine-grained access control is enabled on the domain, then follow the steps in Mapping roles to users (as mentioned in the prerequisites) to allow the Lambda function to read from the domain in read-only access.

The Lambda function that sends OpenSearch domain metrics to CloudWatch is set to a default frequency of 1 day. You can change this configuration to monitor the domain at the required granularity by updating the event schedule for the rule deployed by the CloudFormation stack on the EventBridge console. Note that if the frequency is set to 1 minute, this will trigger the Lambda function every minute and will increase the Lambda cost.

This solution uses the cat/allocation API, which provides the number of data nodes in the domain along with each data node’s number of shards and storage usage attributes. For further details on domain storage and shard skew, refer to Node shard and storage skew. The Lambda function processes and sorts each data node’s storage and shard skew from the average value. Any data node’s skew above 10% from the average is generally considered to be significantly skewed. This will start to impact CPU, network, and disk bandwidth usage because the nodes with the highest storage utilization tend to be the resource-strained nodes, whereas nodes with less than 10% usage represent underutilized capacity.

Refer to Demystifying Elasticsearch shard allocation for details related to shard size and shard count strategy. In general, we recommend keeping shard sizes between 10–30 GB for workloads where search latency is a key performance objective and 30–50 GB for write-heavy workloads. For shard count, we recommend maintaining index shard counts that are divisible by the data node count. For additional details, refer to Sizing Amazon OpenSearch Service domains and Shard strategy.

View skew metrics in CloudWatch

After you run this solution in your account, it will create two CloudWatch metrics for monitoring. To access these CloudWatch metrics, use the following steps:

On the CloudWatch console, under Metrics in the navigation pane, choose All metrics.
Choose Browse and select Custom namespaces. You should see two custom metrics ending with _storageworkspace and _shardworkspace, respectively.
Choose either of the custom metrics and then select NodeID.
On the list of node IDs, select all the nodes displayed in the list, and the graph will be plotted automatically.

You can hover the mouse over the plotted lines to see the node skew information.

The following screenshots show examples of how the CloudWatch metrics will appear on the console.

The storage skew metrics will be similar to the following screenshot. Storage skew metrics shows the domain storage skew. If you hover over the graph, it shows the node list with available nodes in the domain. This list is sorted by the storage size (largest to smallest). The Lambda function will periodically post the latest storage skew results.

The shard skew metrics will be similar to the following screenshot. Shard skew metrics show the domain shard skew. If you hover over the graph, it shows the node list with available nodes in the domain. This list is sorted by the shard size (largest to smallest). The Lambda function will periodically post the latest storage skew results.

Storage skew occurs when one or more nodes within the domain has significantly more storage than other nodes. The CloudWatch metric will show higher deviation of storage usage for these nodes vs. other nodes. Similarly, shard skew occurs when one or more nodes has significantly more shards than others nodes. The CloudWatch metric will show higher deviation for these nodes vs. other nodes in the domain. When the domain storage or shard skew is detected, you can raise a support case to work with the AWS team for remediation actions. See How do I rebalance the uneven shard distribution in my Amazon OpenSearch Service cluster for information on how to take remediation actions to configure your domain shard strategy for optimal performance.

Costs

The cost associated with using this solution would be minimal, around few cents per month since it generates CloudWatch metrics. The solution also runs Lambda code, and in this case the Lambda functions make API calls. For pricing details, refer to Amazon CloudWatch Pricing and AWS Lambda Pricing.

Clean up

If you decide that you no longer want to keep the Lambda function and associated resources, you can navigate to the AWS CloudFormation console, choose the stack, and choose Delete.

If you want to add the CloudWatch skew monitor metrics mechanism back in at any point, you can create the stack again from the CloudFormation template.

Conclusion

You can use this solution to get a better understanding of your OpenSearch Service domain’s storage and shard skew to improve its performance and possibly lower the cost of operating your domain. See Use Elasticsearch’s _rollover API For efficient storage distribution for more details related to shard allocation and efficient storage distribution strategy.

About the authors

Nikhil Agarwal is Sr. Technical Manager with Amazon Web Services. He is passionate about helping customers achieve operational excellence in their cloud journey and working activity on technical solutions. He is also AI/ML enthusiastic and deep dives into customer’s ML-specific use cases. Outside of work, he enjoys traveling with family and exploring different gadgets.

Karthik Chemudupati is a Principal Technical Account Manager (TAM) with AWS, focused on helping customers achieve cost optimization and operational excellence. He has more than 19 years of IT experience in software engineering, cloud operations and automations. Karthik joined AWS in 2016 as a TAM and worked with more than dozen Enterprise Customers across US-West. Outside of work, he enjoys spending time with his family.

Gene Alpert is a Senior Analytics Specialist with AWS Enterprise Support. He has been focused on our Amazon OpenSearch Service customers and ecosystem for the past three years. Gene joined AWS in 2017. Outside of work he enjoys mountain biking, traveling, and playing Population:One in VR.

Why right-sizing your OpenSearch Service domain matters

Key Amazon CloudWatch metrics

Signs of under-provisioning

Remediation recommendations

Signs of over-provisioning

Best practices for right-sizing

Iterate and optimize

Establish baselines

Regular review process

Scaling strategies

Optimize storage configuration

Analyze shard configuration

Handling unexpected traffic spikes

Deploy CloudWatch alarms

Conclusion

Overview

Prerequisites

1. Add a user-defined cost allocation tag to an OpenSearch Service domain

AWS Management Console

AWS CLI

Amazon OpenSearch Service configuration API

AWS SDK

AWS CloudFormation or Terraform

Troubleshooting

2. Activate the user-defined cost allocation tag

3. Analyze OpenSearch Service domain cost using AWS Cost Explorer and tags

Costs

Conclusion

About the Authors

Solution overview

Prerequisites

Deploy the CloudFormation template

View skew metrics in CloudWatch

Costs

Clean up

Conclusion

About the authors

The collective thoughts of the interwebz