Post Syndicated from Gene Alpert original https://aws.amazon.com/blogs/big-data/automating-index-state-management-for-amazon-opensearch-service-successor-to-amazon-elasticsearch-service/
When it comes to time-series data, it’s more common to access new data than existing data, such as the last four hours or one day. Often, application teams must maintain multiple indexes for diverse data workloads, which bring new requirements to set up a custom solution to manage the index lifecycles. This becomes tedious as the indexes grow, and it results in housekeeping overhead.
Amazon OpenSearch Service (successor to Amazon Elasticsearch Service) enables you to automate recurring index management activities. This avoids having to use any additional tools to manage the index lifecycles. Index State Management (ISM) lets you create a policy that automates these operations based on index age, size, and other conditions, all from within your Amazon OpenSearch Service domain.
In this post, we discuss how you can implement a policy to automate routine index management tasks and apply them to indexes and index patterns.
Prerequisites
Before you get started, make sure to complete the following prerequisites:
- Have Elasticsearch 6.8 or later (required to use ISM and UltraWarm) or Set up a new Amazon OpenSearch Service domain with UltraWarm enabled.
- Register a manual snapshot repository for your new domain.
- Make sure that your user role has sufficient permissions to access the Kibana console of the Amazon OpenSearch Service domain. If required, validate and configure the access to your domains.
Use case
Amazon OpenSearch Service supports three storage tiers: the default “hot” for active writing and low-latency analytics, UltraWarm for read-only data up to three petabytes at one-tenth of the hot tier cost, and cold for unlimited long-term archival. Although hot storage is used for indexing and providing the fastest access, UltraWarm complements the hot storage tier by providing less expensive storage for older and less-frequently accessed data. This is done while maintaining the same interactive analytics experience. Rather than attached storage, UltraWarm nodes use Amazon Simple Storage Service (Amazon S3) and a sophisticated caching solution to improve performance.
To demonstrate the functionality, we present a sample use case of handling time-series data in daily indices. In this use case, we automatically snapshot each index after 24 hours, then we migrate the index from the default hot state to UltraWarm storage after two days, cold storage after 30 days, and finally delete the index after 60 days.
After you create the Amazon OpenSearch Service domain and register a snapshot repository, complete the following steps:
- Wait for the domain status to turn active and choose the Kibana endpoint.
- Log in using the Kibana UI endpoint.
- On Kibana’s splash page, add all of the sample data listed by choosing Try our sample data and choosing Add data.
- After adding the data, choose Index Management (the IM icon on the left navigation pane), which lands into the Index Policies page.
- Choose Create policy.
- For Name policy, enter
ism-policy-example.
- Replace the default policy with the following policy:
{
"policy": {
"description": "Demonstrate a hot-snapshot-warm-cold-delete workflow.",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [],
"transitions": [
{
"state_name": "snapshot",
"conditions": {
"min_index_age": "24h"
}
}
]
},
{
"name": "snapshot",
"actions": [
{
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "30m"
},
"snapshot": {
"repository": "snapshot-repo",
"snapshot": "ism-snapshot"
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": {
"min_index_age": "2d"
}
}
]
},
{
"name": "warm",
"actions": [
{
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "1h"
},
"warm_migration": {}
}
],
"transitions": [
{
"state_name": "cold",
"conditions": {
"min_index_age": "30d"
}
}
]
},
{
"name": "cold",
"actions": [
{
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "1h"
},
"cold_migration": {
"start_time": null,
"end_time": null,
"timestamp_field": "@timestamp",
"ignore": "none"
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "60d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"cold_delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"index-*"
],
"priority": 100
}
]
}
}
Note that the last section of the policy, “ism_template”, is used to automatically attach the ISM policy to any newly created index that matches the index_patterns. You can define multiple patterns.
Furthermore, you can use the ISM API to programmatically work with policies and managed indexes.
- Choose Create. You can now see your index policy on the Index Policies page.
- On the Indices page, search for
kibana_sample, which should list all of the sample data indexes that you added earlier.
- Select all of the indexes and choose Apply policy.
- From the Policy ID drop-down menu, choose the policy created in the previous step.
- Choose Apply.
The policy is now assigned and starts managing the indexes. On the Managed Indices page, you can observe the status as Initializing.
When initialization is complete, the status changes to Running.
You can also set a refresh frequency to refresh the managed indexes’ status information.
Demystifying the policy
In this section, we explain the index policy and its structure.
Policies are JSON documents that define the following:
- The states an index can be in
- Any actions that you want the plugin to take when an index enters the state
- Conditions that must be met for an index to move or transition into a new state
The policy document begins with basic metadata such as description, the default_state that the index should enter, and finally a series of state definitions.
A state is the status that the managed index is currently in. A managed index can only be in one state at a time. Each state has associated actions that are run sequentially upon entering a state, and transitions that are checked after all of the actions are complete.
The first state is hot. In this use case, no actions are defined in this hot state. Moreover, the managed indexes land in this state initially, and then transition to snapshot, warm, cold, and finally delete. Transitions define the conditions that must be met for a state to change (in this case, change to snapshot after the index crosses 24 hours).
We can quickly verify the status in the Index Managemet section of Kibana. Each index will have state details listed under Policy managed indices.
You can also verify this on the Amazon OpenSearch Service console under the Ultrawarm Storage usage and Cold Storage usage columns.
The last state of the policy document marks the indexes to delete based on the actions. This policy state assumes that your index is non-critical and no longer receiving write requests. Having zero replicas carries some risk of data loss.
Among other actions, you can also configure the notification to send policy action information to Chime, Slack, or a webhook URL. Complete details can be found here.
The following screenshot shows the index status on the console.
Additional information on ISM policies
If you have an existing Amazon OpenSearch Service domain with no UltraWarm support (because of any missing prerequisites), you can use policy operations read_only and reduces_replicas to replace the warm state. The following code is the policy template for these two states:
{
"name": "reduce_replicas",
"actions": [{
"replica_count": {
"number_of_replicas": 0
}
}],
"transitions": [{
"state_name": "read_only",
"conditions": {
"min_index_age": "2d"
}
}]
},
{
"name": "read_only",
"actions": [
{
"read_only": {}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "3d"
}
}
]
},
Summary
In this post, you learned how to use the ISM feature with UltraWarm and cold storage tiers for Amazon OpenSearch Service. The walkthrough illustrated how to manage indexes using this plugin with a sample lifecycle policy.
For more information about the ISM plugin, see Index State Management. If you need enhancements or have other feature requests, then please file an issue. To get involved with the project, see Contributing Guidelines.
A big takeaway for me as I evaluated the ISM plugin in Amazon OpenSearch Service was that the ISM plugin is fully compatible and works on any deployment of the open source OpenSearch software. For more information, see the Index State Management in OpenSearch documentation.
About the Author
Gene Alpert is a Big Data and Analytics Consultant with Amazon Web Services. He helps AWS customers optimize their cloud data operations.
