Post Syndicated from Jon Handler original https://aws.amazon.com/blogs/big-data/supercharge-your-rag-applications-with-amazon-opensearch-service-and-aryn-docparse/

The old adage “garbage in, garbage out” applies to all search systems. Whether you are building for ecommerce, document retrieval, or Retrieval Augmented Generation (RAG), the quality of your search results depends on the quality of your search documents. Downstream, RAG systems improve the quality of generated answers by adding relevant data from other systems to the generative prompt. Most RAG solutions use a search engine to search for this relevant data. To get great responses, you need great search results, and to get great search results, you need great data. If you don’t properly partition, extract, enrich, and clean your data before loading it, your search results will reflect the poor quality of your search documents.

Aryn DocParse segments and labels PDF documents, runs OCR, extracts tables and images, and more. It turns your messy documents into beautiful, structured JSON, which is the first step of document extract, transform, and load (ETL). DocParse runs the open source Aryn Partitioner and its state-of-the-art, open source deep learning DETR AI model trained on over 80,000 enterprise documents. This leads to up to 6 times more accurate data chunking and 2 times improved recall on vector search or RAG when compared to off-the-shelf systems. The following screenshot is an example of how DocParse would segment a page in an ETL pipeline. You can visualize labeled bounding boxes for each document segment using the Aryn Playground.

In this post, we demonstrate how to use Amazon OpenSearch Service with purpose-built document ETL tools, Aryn DocParse and Sycamore, to quickly build a RAG application that relies on complex documents. We use over 75 PDF reports from the National Transportation Safety Board (NTSB) about aircraft incidents. You can refer to the following example document from the collection. As you can see, these documents are complex, containing tables, images, section headings, and complicated layouts.

Let’s get started!

Prerequisites

Complete the following prerequisite steps:

1. Create an OpenSearch Service domain. For more details, see Creating and managing Amazon OpenSearch Service domains. You can create a domain using the AWS Management Console, AWS Command Line Interface (AWS CLI), or SDK. Be sure to choose public access for your domain, and set up a user name and password for your domain’s primary user so that you can run the notebook from your laptop, Amazon SageMaker Studio, or an Amazon Elastic Compute Cloud (EC2) instance. To keep costs low, you can create an OpenSearch Service domain with a single t3.small search node in a dev/test configuration for this example. Take note of the domain’s endpoint to use in later steps.
2. Get an Aryn API key.
3. You will be using Anthropic’s Claude large language model (LLM) on Amazon Bedrock in the ETL pipeline, so make sure your notebook has access to AWS credentials with the required permissions.
4. Have access to a Jupyter environment to open and run the notebook.

Use DocParse and Sycamore to chunk data and load OpenSearch Service

Although you can generate an ETL pipeline to load your OpenSearch Service domain using the Aryn DocPrep UI, we will instead focus on the underlying Sycamore document ETL library and write a pipeline from scratch.

Sycamore was designed to make it straightforward for developers and data engineers to define complex data transformations over large collections of documents. Borrowing some ideas from popular dataflow frameworks like Apache Spark, Sycamore has a core abstraction called the DocSet. Each DocSet represents a collection of unstructured documents, and is scalable from a single document to many thousands. Each document in a DocSet has an arbitrary set of key-value properties as metadata, as well as an ordered list of elements. An Element corresponds to a chunk of the document that can be processed and embedded separately, such as a table, headline, text passage, or image. Like documents, Elements can also contain arbitrary key-value properties to encode domain- or application-specific metadata.

Notebook walkthrough

We’ve created a Jupyter notebook that uses Sycamore to orchestrate data preparation and loading. This notebook uses Sycamore to create a data processing pipeline that sends documents to DocParse for initial document segmentation and data extraction, then runs entity extraction and data transforms, and finally loads data into OpenSearch Service using a connector.

Copy the notebook into your Amazon SageMaker JupyterLab space, launch it using a Python kernel, then walk through the cells along with the following procedures.

To install Sycamore with the OpenSearch Service connector and local inference features necessary to create vector embeddings, run the first cell of the notebook:

!pip install 'sycamore-ai[opensearch,local-inference]'

In the second cell of the notebook, fill in your ARYN_API_KEY. You should be able to complete the example in the notebook for less than $1.

Cell 3 does the initial work of reading the source data and preparing a DocSet for that data. After initializing the Sycamore context and setting paths, this code calls out to DocParse to create a partitioned_docset:

partitioned_docset = (
  docset.partition(
    partitioner=ArynPartitioner(
      extract_table_structure=True,
      extract_images=True
    )
  ).materialize(
      path="./opensearch-tutorial/partitioned-docset",
      source_mode=sycamore.MATERIALIZE_USE_STORED
    )
)
partitioned_docset.execute()

The previous code uses materialize to create and save a checkpoint. In future runs, the code will use the materialized view to save a few minutes of time. partitioned_docset.execute() forces the pipeline to execute. Sycamore uses lazy execution to create efficient query plans, and would otherwise execute the pipeline at a much later step.

After this step, each document in the DocSet now includes the partitioned output from DocParse, including bounding boxes, text content, and images from that document, stored as elements.

Entity extraction

Part of the key to building good retrieval for RAG is adding structured information that enables accurate filtering for the search query. Sycamore provides LLM-powered transforms that can extract this information and store it as structured properties, enriching the document. Sycamore can do unsupervised or supervised schema extraction, where it pulls out fields based on a JSON schema you provide. When executing these types of transforms, Sycamore will take a specified number of elements from each document, use an LLM to extract the specified fields, and include them as properties in the document.

Cell 4 uses supervised schema extraction, setting the schema as the fields you want to extract. You can add additional information that is passed to the LLM performing the entity extraction. The location property is an example of this:

schema = {
            'type': 'object',
            'properties': {'accidentNumber': {'type': 'string'},
                           'dateAndTime': {'type': 'date'},
                           'location': {
                             'type': 'string', 
                             'description': 'US State where the incident occured'
                           },
                           'aircraft': {'type': 'string'},
                           'aircraftDamage': {'type': 'string'},
                           'injuries': {'type': 'string'},
                           'definingEvent': {'type': 'string'}},
            'required': ['accidentNumber',
                         'dateAndTime',
                         'location',
                         'aircraft']
    }

schema_name = 'FlightAccidentReport'
property_extractor=LLMPropertyExtractor(llm=llm, num_of_elements=20, schema_name=schema_name, schema=schema)

The LLMPropertyExtractor uses the schema you provided to add additional properties to the document. Next, summarize the images to add additional information to improve retrieval.

Image summarization

There’s more information in your documents than just text—as the saying goes, a picture is worth 1,000 words! When your documents contain images, you can capture the information in those images using Sycamore’s SummarizeImages transform. SummarizeImages uses an LLM to compute a text summary for the image, then adds the summary to that element. Sycamore will also send related information about the image, like a caption, to the LLM to aid with summarization. The following code (in cell 4) takes advantage of DocParse type labeling to automatically apply SummarizeImages to image elements:

enriched_docset = enriched_docset.transform(SummarizeImages, summarizer=LLMImageSummarizer(llm=llm))

This cell can take up to 20 minutes to complete.

Now that your image elements contain additional retrieval information, it’s time to clean and normalize the text in the elements and extracted entities.

Data cleaning and formatting

Unless you are in direct control of the creation of the documents you are processing, you will likely need to normalize that data and make it ready for search. Sycamore makes it straightforward for you to clean messy data and bring it to a regular form, fixing data quality issues.

For example, in the NTSB data, dates in the incident report are not all formatted the same way, and some US state names are shown as abbreviations. Sycamore makes it straightforward to write custom transformations in Python, and also provides several useful cleaning and formatting transforms. Cell 4 uses two functions in Sycamore to format the state names and dates:

formatted_docset = (
  enriched_docset
  
  # Converts state abbreviations to their full names.
  .map(lambda doc: USStateStandardizer.standardize(
    doc, key_path = ["properties","entity","location"])
  )

  # Converts datetime into a common format
  .map(lambda doc: DateTimeStandardizer.standardize(
    doc, key_path = ["properties","entity","dateTime"])
  )
)

The elements are now in normal form, with extracted entities and image descriptions. The next step is to merge together semantically related elements to create chunks.

Create final chunks and vector embeddings

When you prepare for RAG, you create chunks—parts of the full document that are related information. You design your chunks so that as a search result they can be added to the prompt to provide a unit of meaning and information. There are many ways to approach chunking. If you have small documents, sometimes the whole document is a chunk. If you have larger documents, sentences, paragraphs, or even sections can be a chunk. As you iterate on your end application, it’s common to adjust the chunking strategy to fine-tune the accuracy of retrieval. Sycamore automates the process of building chunks by merging together the elements of the DocSet.

At this stage of the processing in cell 4, each document in our DocSet has a set of elements. The following code merges elements together using a chunking strategy to create larger elements that will improve query results. For instance, the DocSet might have an element that is a table and an element that is a caption for that table. Merging those elements together creates a chunk that’s a better search result.

We will use Sycamore’s Merge transform with the GreedySectionMerger merging strategy to add elements in the same document section together into larger chunks:

merger = GreedySectionMerger(
  tokenizer=HuggingFaceTokenizer(
    "sentence-transformers/all-MiniLM-L6-v2"),
  max_tokens=512
)
chunked_docset = formatted_docset.merge(merger=merger)

With chunks created, it’s time to add vector embeddings for the chunks.

Create vector embeddings

Use vector embeddings to enable semantic search in OpenSearch Service. With semantic search, retrieve documents that are close to a query in a multidimensional space, rather than by matching words exactly. In RAG systems, it’s common to use semantic search along with lexical search for a hybrid search. Using hybrid search, you get best-of-all-worlds retrieval.

The code in cell 4 creates vector embeddings for each chunk. You can use a variety of different AI models with Sycamore’s embed transform to create vector embeddings. You can run these locally or use a service like Amazon Bedrock or OpenAI. The embedding model you choose has a huge impact on your search quality, and it’s common to experiment with this variable as well. In this example, you create embeddings locally using a model called GTE:

model_name = "thenlper/gte-small"
embedded_docset = chunked_docset.spread_properties(["entity", "path"]).explode().embed(
      embedder=SentenceTransformerEmbedder(batch_size=10_000, model_name=model_name)
)
embedded_docset = embedded_docset.materialize(
  path="./opensearch-tutorial/embedded-docset",
  source_mode=sycamore.MATERIALIZE_USE_STORED
)
embedded_docset.execute()

You use materialize again here, so you can checkpoint the processed DocSet before loading. If there is an error when loading the indexes, you can retry without running the last few steps of the pipeline again.

Load OpenSearch Service

The final ETL step is loading the prepared data into OpenSearch Service vector and keyword indexes to power hybrid search for the RAG application. Sycamore makes loading indexes straightforward with its set of connectors. Cell 5 adds configuration, specifying the OpenSearch Service domain endpoint and what indexes to create. If you’re following along, be sure to replace YOUR-DOMAIN-ENDPOINT, YOUR-OPENSEARCH-USERNAME, and YOUR-OPENSEARCH-PASSWORD in cell 5 with the actual values.

If you copied your domain endpoint from the console, it will start with the https:// URL scheme. When you replace YOUR-DOMAIN-ENDPOINT, be sure to remove https://.

In cell 6, Sycamore’s OpenSearch connector loads the data into an OpenSearch index:

embedded_docset.write.opensearch(
    os_client_args=openSearch_client_args,
    index_name="aryn-rag-demo",
    index_settings=index_settings,
)

Congratulations! You’ve completed some of the core processing steps to take raw PDFs and prepare them as a source for retrieval in a RAG application. In the next cells, you will run a couple of RAG queries.

Run a RAG query on OpenSearch using Sycamore

In cell 7, Sycamore’s query and summarize functions create a RAG pipeline on the data. The query step uses OpenSearch’s vector search to retrieve the relevant passages for RAG. Then, cell 8 runs a second RAG query that filters on metadata that Sycamore extracted in the ETL pipeline, yielding even better results. You could also use an OpenSearch hybrid search pipeline to perform hybrid vector and lexical retrieval.

Cell 7 asks “What was common with incidents in Texas, and how does that differ from incidents in California?” Sycamore’s summarize_data transform runs the RAG query, and uses the LLM specified for generation (in this case, it’s Anthropic’s Claude):

Based on the provided data, it appears that the common factor among the incidents 
in Texas was that many of them involved substantial aircraft damage, with some resulting 
in injuries or fatalities. The incidents covered a range of aircraft types, including small
planes like Cessnas and Pipers, as well as a helicopter. The defining events varied, 
including loss of control on the ground, engine failures, fuel issues, and collisions 
with terrain or objects.

In contrast, the incidents in California seemed to primarily involve substantial aircraft
damage as well, but with fewer injuries reported. The defining events included loss of 
control on the ground, collisions during takeoff or landing, and a miscellaneous/other event.
One key difference is that the Texas incidents included a fatal accident (CEN23FA084) 
involving a Piper PA46 that resulted in 4 fatalities and 1 serious injury after impacting 
terrain. The California incidents did not appear to have any fatal accidents based on the 
provided data.

Additionally, while both states had incidents involving loss of control on the ground, the 
Texas incidents seemed to have a higher proportion of engine failures, fuel issues, and 
collisions with terrain or objects as defining events compared to California.

Overall, while both states experienced aviation incidents resulting in substantial aircraft
damage, the Texas incidents tended to be more severe in terms of injuries and fatalities, 
with a higher prevalence of engine failures, fuel issues, and terrain/object collisions as 
contributing factors.

Using metadata filters in a RAG query

Cell 8 makes a small adjustment to the code to add a filter to the vector search, filtering for documents from incidents with the location of California. Filters increase the accuracy of chatbot responses by removing irrelevant data from the result the RAG pipeline passes to the LLM in the prompt.

To add a filter, cell 8 adds a filter clause to the k-nearest neighbors (k-NN) query:

os_query["query"]["knn"]["embedding"]["filter"] = {"match_phrase": {"properties.entity.location": "California"}}

The output from the RAG query is as follows:

Based on the database entries provided, several incidents occurred in California during January 2023:

1. On January 12th, a Cessna 180K aircraft sustained substantial damage in a collision during takeoff 
or landing at Agua Caliente Springs, California. There was 1 person on board with no injuries reported.

2. On January 20th, a Cessna 195A aircraft sustained substantial damage due to a los of control on the 
ground at Calexico, California. There were 3 people on board with no injuries.  

3. On January 15th, a Piper PA-28-180 aircraft sustained substantial damage in a miscellaneous incident 
at San Diego, California during an instructional flight. There were 4 people on board with no injuries.

4. On January 1st, a Cessna 172 aircraft sustained substantial damage in a collision during takeoff or 
landing at Watsonville, California during an instructional flight. There was 1 serious injury reported.

5. On January 27th, a Cessna T210N aircraft sustained substantial damage when it descended into a ravine 
and impacted the ground about 2,000 feet short of the runway threshold at Murrieta, California. There were
1 serious injury and 1 minor injury reported. The engine did not respond during the landing approach.

The details provided in the database entries, such as aircraft type, location, date/time, damage level, 
injuries, and a brief description of the defining event, serve as evidence for these incidents occurring 
in California during the specified time period.

Clean up

Be sure to clean up the resources you deployed for this walkthrough:

1. Delete your OpenSearch Service domain.
2. Remove any Jupyter environments you created.

Conclusion

In this post, you used Aryn DocParse and Sycamore to parse, extract, enrich, clean, embed, and load data into vector and keyword indexes in OpenSearch Service. You then used Sycamore to run RAG queries on this data. Your second RAG query used an OpenSearch filter on metadata to get a more accurate result.

The way in which your documents are parsed, enriched, and processed has a significant impact on the quality of your RAG queries. You can use the examples in this post to build your own RAG systems with Aryn and OpenSearch Service, and iterate on the processing and retrieval strategies as you build your generative AI application.

About the Authors

Jon Handler is Director of Solutions Architecture for Search Services at Amazon Web Services, based in Palo Alto, CA. Jon works closely with OpenSearch and Amazon OpenSearch Service, providing help and guidance to a broad range of customers who have search and log analytics workloads for OpenSearch. Prior to joining AWS, Jon’s career as a software developer included four years of coding a large-scale ecommerce search engine. Jon holds a Bachelor of the Arts from the University of Pennsylvania, and a Master’s of Science and a PhD in Computer Science and Artificial Intelligence from Northwestern University.

Jon is the founding Chief Product Officer at Aryn. Prior to that, he was the SVP of Product Management at Dremio, a data lake company. Earlier, Jon was a Director at AWS, and led product management for in-memory database services (Amazon ElastiCache and Amazon MemoryDB for Redis), Amazon EMR (Apache Spark and Hadoop), and founded and was GM of the blockchain division. Jon has an MBA from Stanford Graduate School of Business and a BA in Chemistry from Washington University in St. Louis.

Post Syndicated from Dylan Tong original https://aws.amazon.com/blogs/big-data/opensearch-vector-engine-is-now-disk-optimized-for-low-cost-accurate-vector-search/

OpenSearch Vector Engine can now run vector search at a third of the cost on OpenSearch 2.17+ domains. You can now configure k-NN (vector) indexes to run on disk mode, optimizing it for memory-constrained environments, and enable low-cost, accurate vector search that responds in low hundreds of milliseconds. Disk mode provides an economical alternative to memory mode when you don’t need near single-digit latency.

In this post, you’ll learn about the benefits of this new feature, the underlying mechanics, customer success stories, and getting started.

Overview of vector search and the OpenSearch Vector Engine

Vector search is a technique that improves search quality by enabling similarity matching on content that has been encoded by machine learning (ML) models into vectors (numerical encodings). It enables use cases like semantic search, allowing you to consider context and intent along with keywords to deliver more relevant searches.

OpenSearch Vector Engine enables real-time vector searches beyond billions of vectors by creating indexes on vectorized content. You can then run searches for the top K documents in an index that are most similar to a given query vector, which could be a question, keyword, or content (such as an image, audio clip, or text) that has been encoded by the same ML model.

Tuning the OpenSearch Vector Engine

Search applications have varying requirements in terms of speed, quality, and cost. For instance, ecommerce catalogs require the lowest possible response times and high-quality search to deliver a positive shopping experience. However, optimizing for search quality and performance gains generally incurs cost in the form of additional memory and compute.

The right balance of speed, quality, and cost depends on your use cases and customer expectations. OpenSearch Vector Engine provides comprehensive tuning options so you can make smart trade-offs to achieve optimal results tailored to your unique requirements.

You can use the following tuning controls:

• Algorithms and parameters – This includes the following:
  • Hierarchical Navigable Small World (HNSW) algorithm and parameters like ef_search, ef_construct, and m
  • Inverted File Index (IVF) algorithm and parameters like nlist and nprobes
  • Exact k-nearest neighbors (k-NN), also known as brute-force k-NN (BFKNN) algorithm
• Engines – Facebook AI Similarity Search (FAISS), Lucene, and Non-metric Space Library (NMSLIB).
• Compression techniques – Scalar (such as byte and half precision), binary, and product quantization
• Similarity (distance) metrics – Inner product, cosine, L1, L2, and hamming
• Vector embedding types – Dense and sparse with variable dimensionality
• Ranking and scoring methods – Vector, hybrid (combination of vector and Best Match 25 (BM25) scores), and multi-stage ranking (such as cross-encoders and personalizers)

You can adjust a combination of tuning controls to achieve a varying balance of speed, quality, and cost that is optimized to your needs. The following diagram provides a rough performance profiling for sample configurations.

Tuning for disk-optimization

With OpenSearch 2.17+, you can configure your k-NN indexes to run on disk mode for high-quality, low-cost vector search by trading in-memory performance for higher latency. If your use case is satisfied with 90th percentile (P90) latency in the range of 100–200 milliseconds, disk mode is an excellent option for you to achieve cost savings while maintaining high search quality. The following diagram illustrates disk mode’s performance profile among alternative engine configurations.

Disk mode was designed to run out of the box, reducing your memory requirements by 97% compared to memory mode while providing high search quality. However, you can tune compression and sampling rates to adjust for speed, quality, and cost.

The following table presents performance benchmarks for disk mode’s default settings. OpenSearch Benchmark (OSB) was used to run the first three tests, and VectorDBBench (VDBB) was used for the last two. Performance tuning best practices were applied to achieve optimal results. The low scale tests (Tasb-1M and Marco-1M) were run on a single r7gd.large data node with one replica. The other tests were run on two r7gd.2xlarge data nodes with one replica. The percent cost reduction metric is calculated by comparing an equivalent, right-sized in-memory deployment with the default settings.

These tests are designed to demonstrate that disk mode can deliver high search quality with 32 times compression across a variety of datasets and models while maintaining our target latency (under P90 200 milliseconds). These benchmarks aren’t designed for evaluating ML models. A model’s impact on search quality varies with multiple factors, including the dataset.

Disk mode’s optimizations under the hood

When you configure a k-NN index to run on disk mode, OpenSearch automatically applies a quantization technique, compressing vectors as they’re loaded to build a compressed index. By default, disk mode converts each full-precision vector—a sequence of hundreds to thousands of dimensions, each stored as 32-bit numbers—into binary vectors, which represent each dimension as a single-bit. This conversion results in a 32 times compression rate, enabling the engine to build an index that is 97% smaller than one composed of full-precision vectors. A right-sized cluster will keep this compressed index in memory.

Compression lowers cost by reducing the memory required by the vector engine, but it sacrifices accuracy in return. Disk mode recovers accuracy, and therefore search quality, using a two-step search process. The first phase of the query execution begins by efficiently traversing the compressed index in memory for candidate matches. The second phase uses these candidates to oversample corresponding full-precision vectors. These full-precision vectors are stored on disk in a format designed to reduce I/O and optimize disk retrieval speed and efficiency. The sample of full-precision vectors is then used to augment and re-score matches from phase one (using exact k-NN), thereby recovering the search quality loss attributed to compression. Disk mode’s higher latency relative to memory mode is attributed to this re-scoring process, which requires disk access and additional computation.

Early customer successes

Customers are already running the vector engine in disk mode. In this section, we share testimonials from early adopters.

Asana is improving search quality for customers on their work management platform by phasing in semantic search capabilities through OpenSearch’s vector engine. They initially optimized the deployment by using product quantization to compress indexes by 16 times. By switching over to the disk-optimized configurations, they were able to potentially reduce cost by another 33% while maintaining their search quality and latency targets. These economics make it viable for Asana to scale to billions of vectors and democratize semantic search throughout their platform.

DevRev bridges the fundamental gap in software companies by directly connecting customer-facing teams with developers. As an AI-centered platform, it creates direct pathways from customer feedback to product development, helping over 1,000 companies accelerate growth with accurate search, fast analytics, and customizable workflows. Built on large language models (LLMs) and Retrieval Augmented Generation (RAG) flows running on OpenSearch’s vector engine, DevRev enables intelligent conversational experiences.

“With OpenSearch’s disk-optimized vector engine, we achieved our search quality and latency targets with 16x compression. OpenSearch offers scalable economics for our multi-billion vector search journey.”

– Anshu Avinash, Head of AI and Search at DevRev.

Get started with disk mode on the OpenSearch Vector Engine

First, you need to determine the resources required to host your index. Start by estimating the memory required to support your disk-optimized k-NN index (with the default 32 times compression rate) using the following formula:

Required memory (bytes) = 1.1 x ((vector dimension count)/8 + 8 x m) x (vector count)

For instance, if you use the defaults for Amazon Titan Text V2, your vector dimension count is 1024. Disk mode uses the HNSW algorithm to build indexes, so “m” is one of the algorithm parameters, and it defaults to 16. If you build an index for a 1-billion vector corpus encoded by Amazon Titan Text, your memory requirements are 282 GB.

If you have a throughput-heavy workload, you need to make sure your domain has sufficient IOPs and CPUs as well. If you follow deployment best practices, you can use instance store and storage performance optimized instance types, which will generally provide you with sufficient IOPs. You should always perform load testing for high-throughput workloads, and adjust the original estimates to accommodate for higher IOPs and CPU requirements.

Now you can deploy an OpenSearch 2.17+ domain that has been right-sized to your needs. Create your k-NN index with the mode parameter set to on_disk, and then ingest your data. If you already have a k-NN index running on the default in_memory mode, you can convert it by switching the mode to on_disk followed by a reindex task. After the index is rebuilt, you can downsize your domain accordingly.

Conclusion

In this post, we discussed how you can benefit from running the OpenSearch Vector Engine on disk mode, shared customer success stories, and provided you tips on getting started. You’re now set to run the OpenSearch Vector Engine at as low as a third of the cost.

To learn more, refer to the documentation.

About the Authors

Dylan Tong is a Senior Product Manager at Amazon Web Services. He leads the product initiatives for AI and machine learning (ML) on OpenSearch including OpenSearch’s vector database capabilities. Dylan has decades of experience working directly with customers and creating products and solutions in the database, analytics and AI/ML domain. Dylan holds a BSc and MEng degree in Computer Science from Cornell University.

Vamshi Vijay Nakkirtha is a software engineering manager working on the OpenSearch Project and Amazon OpenSearch Service. His primary interests include distributed systems.

Post Syndicated from Mulugeta Mammo original https://aws.amazon.com/blogs/big-data/intel-accelerators-on-amazon-opensearch-service-improve-price-performance-on-vector-search-by-up-to-51/

This post is co-written with Mulugeta Mammo and Akash Shankaran from Intel.

Today, we’re excited to announce the availability of Intel Advanced Vector Extensions 512 (AVX-512) technology acceleration on vector search workloads when you run OpenSearch 2.17+ domains with the 4^th generation Intel Xeon Intel instances on the Amazon OpenSearch Service. When you run OpenSearch 2.17 domains on C/M/R 7i instances, you can gain up to 51% in vector search performance at no additional cost compared to previous R5 Intel instances.

Increasingly, application builders are using vector search to improve the search quality of their applications. This modern technique involves encoding content into numerical representations (vectors) that can be used to find similarities between content. For instance, it’s used in generative AI applications to match user queries to semantically similar knowledge articles providing context and grounding for generative models to perform tasks. However, vector search is computationally intensive, and higher compute and memory requirements can lead to higher costs than traditional search. Therefore, cost optimization levers are important to achieve a favorable balance of cost vs. benefit.

OpenSearch Service is a managed service for the OpenSearch search and analytics suite, which includes support for vector search. By running your OpenSearch 2.17+ domains on C/M/R 7i instances, you can achieve up to a 51% price-performance gain compared to the past R5 instances on OpenSearch Service. As we discuss in this post, this launch offers improvements to your infrastructure total cost of ownership (TCO) and savings.

Accelerating generative AI applications with vectorization

Let’s understand how these technologies come together through the building of a simple generative AI application. First, you bring vector search online by using machine learning (ML) models to encode your content (such as text, image or audio) into vectors. You then index these vectors into an OpenSearch Service domain, enabling real-time content similarity search that can be scaled to search billions of vectors in milliseconds. These vector searches provide contextually relevant insights, which can be further enriched by AI for hyper-personalization and integrated with generative models to power chatbots.

Vector search use cases extend beyond generative AI applications. Use cases include image to semantic search, and recommendations such as the following real-world use case from Amazon Music. The Amazon Music application uses vectorization to encode 100 million songs into vectors that represent both music tracks and customer preferences. These vectors are then indexed in OpenSearch, which manages over a billion vectors and handles up to 7,100 vector queries per second to analyze user listening behavior and provide real-time recommendations.

The indexing and search processes are computationally intensive, requiring calculations between vectors that are typically represented as 128–2,048 dimensions (numerical values). The Intel Xeon Scalable processors found on the 7^th generation Intel instances use Intel AVX-512 to increase the speed and efficiency of vector operations through the following features:

• Data parallel processing – By processing 512 bits (twice the number of its predecessor) of data at once, Intel AVX-512 efficiently uses SIMD (single input multiple data) to run multiple operations simultaneously, which provides significant speed-up
• Pathlength reduction – The speed-up is due to a significant improvement in pathlength, which is a measure of the number of instructions required to perform a unit of work in workloads
• Power performance savings – You can lower power performance costs by processing more data and performing more operations in a shorter amount of time

Benchmarking vector search on OpenSearch

OpenSearch Services R7i Instances with Intel AVX-512 are an excellent choice for OpenSearch vector workloads. They offer a high CPU-to-memory ratio, which further maximizes the compute potential while providing ample memory.

To verify just how much faster the new R7i instances perform, you can run OpenSearch benchmarks firsthand. Using your OpenSearch 2.17 domain, create a k-NN index configured to use either the Lucene or FAISS engine. Use the OpenSearch Benchmark with the public Cohere 10M 768D dataset to replicate the benchmarks published in this post. Replicate these tests using the older R5 instances as the baseline.

In the following sections, we present the benchmarks that demonstrate the 51% price-performance gains between the R7i and the R5 instances.

Lucene engine results

In this post, we define price-performance as the number of documents that can be indexed or search queries executed given a fixed budget ($1), taking into account the instance cost. The following are results of price-performance with the Cohere 10M dataset.

Up to a 44% improvement in price-performance is observed when using the Lucene engine and upgrading from R5 to R7i instances. The difference between the blue and orange bars in the following graphs illustrates the gains contributed by AVX512 acceleration.

FAISS engine results

We also examine results from the same tests performed on k-NN indexes configured on the FAISS engine. Up to 51% price-performance gains is achieved on index performance simply by upgrading from r5 to r7i instances. Again, the difference between the blue and orange bar demonstrates the additional gains contributed by AVX512.

In addition to price-performance gains, search response times also improved by upgrading R5 to R7i instances with AVX512. P90 and P99 latencies were lower by 33% and 38%, respectively.

The FAISS engine has the added benefit of AVX-512 acceleration with FP16 quantized vectors. With FP16 quantization, vectors are compressed to half the size, reducing memory and storage requirements and in turn infrastructure costs. AVX-512 contributes to further price-performance gains.

Conclusion

If you’re looking to modernize search experiences on OpenSearch Service while potentially lowering costs, try out the OpenSearch vector engine on OpenSearch Service C7i, M7i, or R7i instances. Built on 4^th Gen Intel Xeon processors, the latest Intel instances provide advanced features like Intel AVX-512 accelerators, improved CPU performance, and higher memory bandwidth than the previous generation, which makes them an excellent choice for optimizing your vector search workloads on OpenSearch Service.

Credits to: Vesa Pehkonen, Noah Staveley, Assane Diop, Naveen Tatikonda

About the Authors

Mulugeta Mammo is a Senior Software Engineer, and currently leads the OpenSearch Optimization team at Intel.

Vamshi Vijay Nakkirtha is a software engineering manager working on the OpenSearch Project and Amazon OpenSearch Service. His primary interests include distributed systems.

Akash Shankaran is a Software Architect and Tech Lead in the Xeon software team at Intel working on OpenSearch. He works on pathfinding opportunities and enabling optimizations within databases, analytics, and data management domains.

Dylan Tong is a Senior Product Manager at Amazon Web Services. He leads the product initiatives for AI and machine learning (ML) on OpenSearch including OpenSearch’s vector database capabilities. Dylan has decades of experience working directly with customers and creating products and solutions in the database, analytics and AI/ML domain. Dylan holds a BSc and MEng degree in Computer Science from Cornell University.

Notices and disclaimers

Performance varies by use, configuration, and other factors. Learn more on the Performance Index website.
Your costs and results may vary.
Intel technologies may require enabled hardware, software, or service activation.