Post Syndicated from Joshua Luo original https://aws.amazon.com/blogs/big-data/deploy-amazon-opensearch-serverless-with-terraform/
Amazon OpenSearch Serverless provides the search and analytical functionality of OpenSearch without the manual overhead of configuring, managing, and scaling OpenSearch clusters. It automatically scales the resources based on your workload, and you only pay for the resources consumed. Managing OpenSearch Serverless is simple, but with infrastructure as code (IaC) software like Terraform, you can simplify your resource management even more.
This post demonstrates how to use Terraform to create, deploy, and clean up OpenSearch Serverless infrastructure.
Solution overview
To create and deploy an OpenSearch Serverless collection with security and access policies using Terraform, you need to follow these steps:
- Initialize the Terraform configuration.
- Create an encryption policy.
- Create an OpenSearch Serverless collection.
- Create a network policy.
- Create a virtual private cloud (VPC) endpoint.
- Create a data access policy.
- Deploy using Terraform.
Prerequisites
This post assumes that you’re familiar with GitHub and Git commands.
For this walkthrough, you need the following:
Initialize the Terraform configuration
The sample code is available in the Terraform GitHub repo in the terraform-provider-aws/examples/opensearchserverless directory. This configuration will get you started with OpenSearch Serverless. First, clone the repository to your workstation and navigate to the directory:
$ git clone https://github.com/hashicorp/terraform-provider-aws.git && \
cd ./terraform-provider-aws/examples/opensearchserverless
Initialize the configuration to install the aws provider by running the following command:
$ terraform init
The Terraform configuration first defines the version of Terraform required and configures the AWS provider to launch resources in the Region defined by the aws_region variable:
# Require Terraform 0.12 or greater
terraform {
required_version = ">= 0.12"
}
# Set AWS provider region
provider "aws" {
region = var.aws_region
}
The variables used in this Terraform configuration are defined in the variables.tf file. This post assumes the default values are used:
variable "aws_region" {
description = "The AWS region to create things in."
default = "us-east-1"
}
variable "collection_name" {
description = "Name of the OpenSearch Serverless collection."
default = "example-collection"
}
Create an encryption policy
Now that the provider is installed and configured, the Terraform configuration moves on to defining OpenSearch Serverless policies for security. OpenSearch Serverless uses AWS Key Management Service (AWS KMS) to encrypt your data. The encryption is managed by an encryption policy. To create an encryption policy, use the aws_opensearchserverless_security_policy resource, which has a name parameter, a type of encryption, a JSON string that defines the policy, and an optional description:
# Creates an encryption security policy
resource "aws_opensearchserverless_security_policy" "encryption_policy" {
name = "example-encryption-policy"
type = "encryption"
description = "encryption policy for ${var.collection_name}"
policy = jsonencode({
Rules = [
{
Resource = [
"collection/${var.collection_name}"
],
ResourceType = "collection"
}
],
AWSOwnedKey = true
})
}
This encryption policy is named example-encryption-policy, applies to a collection named example-collection, and uses an AWS owned key to encrypt the data.
Create an OpenSearch Serverless collection
You can organize your OpenSearch indexes into a logical grouping called a collection. Create a collection using the aws_opensearchserverless_collection resource, which has a name parameter, and optionally, description, tags, and type:
# Creates a collection
resource "aws_opensearchserverless_collection" "collection" {
name = var.collection_name
depends_on = [aws_opensearchserverless_security_policy.encryption_policy]
}
This collection is named example-collection. If type is not specified, a time series collection is created. Supported collection types can be found in the Terraform documentation for the aws_opensearchserverless_collection resource. OpenSearch Serverless requires encryption at rest, so an applicable encryption policy is required before a collection can be created. The Terraform configuration explicitly defines this dependency using the depends_on meta-argument. Errors can arise if this dependency is not defined.
Now that a collection has been created with an AWS owned KMS key, the Terraform configuration goes on to define the network and data access policy to configure access to the collection.
Create a network policy
A network policy allows access to your collection either over the public internet or through OpenSearch Serverless-managed VPC endpoints. Similar to the encryption policy, to create a network policy, use the aws_opensearchserverless_security_policy resource, which has a name parameter, a type of network, a JSON string that defines the policy, and an optional description:
# Creates a network security policy
resource "aws_opensearchserverless_security_policy" "network_policy" {
name = "example-network-policy"
type = "network"
description = "public access for dashboard, VPC access for collection endpoint"
policy = jsonencode([
{
Description = "VPC access for collection endpoint",
Rules = [
{
ResourceType = "collection",
Resource = [
"collection/${var.collection_name}"
]
}
],
AllowFromPublic = false,
SourceVPCEs = [
aws_opensearchserverless_vpc_endpoint.vpc_endpoint.id
]
},
{
Description = "Public access for dashboards",
Rules = [
{
ResourceType = "dashboard"
Resource = [
"collection/${var.collection_name}"
]
}
],
AllowFromPublic = true
}
])
}
This network policy is named example-network-policy and applies to the collection named example-collection. This policy only allows access to the collection’s OpenSearch endpoint through a VPC endpoint, but allows public access to the OpenSearch Dashboards endpoint.
You’ll notice the VPC endpoint has not been defined yet, but it is referenced in the network policy. Terraform determines this dependency automatically and will not create the network policy until the VPC endpoint has been created.
Create a VPC endpoint
A VPC endpoint enables you to privately access your OpenSearch Serverless collection using AWS PrivateLink (for more information, refer to Access AWS services through AWS PrivateLink). Create a VPC endpoint using the aws_opensearchserverless_vpc_endpoint resource, where you define name, vpc_id, subnet_ids , and optionally, security_group_ids:
# Creates a VPC endpoint
resource "aws_opensearchserverless_vpc_endpoint" "vpc_endpoint" {
name = "example-vpc-endpoint"
vpc_id = aws_vpc.vpc.id
subnet_ids = [aws_subnet.subnet.id]
security_group_ids = [aws_security_group.security_group.id]
}
Creating a VPC and all the required networking resources is out of scope for this post, but the minimum required VPC resources are created here in a separate file to demonstrate the VPC endpoint functionality. Refer to Getting Started with Amazon VPC to learn more.
Create a data access policy
The configuration defines a data source that looks up information about the context Terraform is currently running in. This data source is used when defining the data access policy. More information can be found in the Terraform documentation for aws_caller_identity.
# Gets access to the effective Account ID in which Terraform is authorized
data "aws_caller_identity" "current" {}
A data access policy allows you to define who has access to collections and indexes. The data access policy is defined using the aws_opensearchserverless_access_policy resource, which has a name parameter, a type parameter set to data, a JSON string that defines the policy, and an optional description:
# Creates a data access policy
resource "aws_opensearchserverless_access_policy" "data_access_policy" {
name = "example-data-access-policy"
type = "data"
description = "allow index and collection access"
policy = jsonencode([
{
Rules = [
{
ResourceType = "index",
Resource = [
"index/${var.collection_name}/*"
],
Permission = [
"aoss:*"
]
},
{
ResourceType = "collection",
Resource = [
"collection/${var.collection_name}"
],
Permission = [
"aoss:*"
]
}
],
Principal = [
data.aws_caller_identity.current.arn
]
}
])
}
This data access policy allows the current AWS role or user to perform collection-related actions on the collection named example-collection and index-related actions on the indexes in the collection.
Deploy using Terraform
Now that you have configured the necessary resources, apply the configuration using terraform apply. Before creating the resources, Terraform will describe all the resources that will be created so you can verify your configuration:
$ terraform apply
...
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
...
Plan: 13 to add, 0 to change, 0 to destroy.
Changes to Outputs:
+ collection_enpdoint = (known after apply)
+ dashboard_endpoint = (known after apply)
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value:
If this is the first OpenSearch Serverless collection in your account, applying the configuration may take over 10 minutes because Terraform waits for the collection to become active.
Apply complete! Resources: 13 added, 0 changed, 0 destroyed.
Outputs:
collection_enpdoint = "..."
dashboard_endpoint = "..."
You have now deployed an OpenSearch Serverless time series collection with policies to configure encryption and access to the collection!
Clean up
The resources will incur costs as long as they are running, so clean up the resources when you are done using them. Use the terraform destroy command to do this:
$ terraform destroy
...
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
...
Plan: 0 to add, 0 to change, 13 to destroy.
Changes to Outputs:
...
Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value:
Answer yes to run this plan and destroy the infrastructure.
Destroy complete! Resources: 13 destroyed.
All resources created during this walkthrough have now been deleted.
Conclusion
In this post, you created an OpenSearch Serverless collection. Using IaC software like Terraform can make it simple to manage your resources like OpenSearch Serverless collections, encryption, network, and data access policies, and VPC endpoints.
Thank you to all the open-source contributors who help maintain OpenSearch and Terraform.
Try using OpenSearch Serverless with Terraform to simplify your resource management. Check out the Getting started with Amazon OpenSearch Serverless workshop and the Amazon OpenSearch Serverless Developer Guide to learn more about OpenSearch Serverless.
About the authors
Joshua Luo is a Software Development Engineer for Amazon OpenSearch Serverless. He works on the systems that enable customers to manage and monitor their OpenSearch Serverless resources. He enjoys bouldering, photography, and videography in his free time.
Satish Nandi is a Senior Technical Product Manager for Amazon OpenSearch Service.
Omar Gonzalez is a Senior Solutions Architect at Amazon Web Services in Southern California with more than 20 years of experience in IT. He is passionate about helping customers drive business value through the use of technology. Outside of work, he enjoys hiking and spending quality time with his family.
Matt Miller is a Business Development Principal at AWS. In his role, Matt drives customer and partner adoption for the AWS Clean Rooms service specializing in advertising and marketing industry use cases. Matt believes in the primacy of privacy enhanced data collaboration and interoperability underpinning data-driven marketing imperatives from customer experience to addressable advertising. Prior to AWS, Matt led strategy and go-to market efforts for ad technologies, large agencies, and consumer data products purpose-built to inform smarter marketing and deliver better customer experiences.
Joshua Luo is a Software Development Engineer for Amazon OpenSearch Serverless. He works on the systems that enable customers to manage and monitor their OpenSearch Serverless resources. He enjoys bouldering, photography, and videography in his free time.
Satish Nandi is a Senior Technical Product Manager for Amazon OpenSearch Service.
Raj Ramasubbu is a Senior Analytics Specialist Solutions Architect focused on big data and analytics and AI/ML with Amazon Web Services. He helps customers architect and build highly scalable, performant, and secure cloud-based solutions on AWS. Raj provided technical expertise and leadership in building data engineering, big data analytics, business intelligence, and data science solutions for over 18 years prior to joining AWS. He helped customers in various industry verticals like healthcare, medical devices, life science, retail, asset management, car insurance, residential REIT, agriculture, title insurance, supply chain, document management, and real estate.