Post Syndicated from Dylan Tong original https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-launches-flow-builder-to-empower-rapid-ai-search-innovation/

You can now access the AI search flow builder on OpenSearch 2.19+ domains with Amazon OpenSearch Service and begin innovating AI search applications faster. Through a visual designer, you can configure custom AI search flows—a series of AI-driven data enrichments performed during ingestion and search. You can build and run these AI search flows on OpenSearch to power AI search applications on OpenSearch without you having to build and maintain custom middleware.

Applications are increasingly using AI and search to reinvent and improve user interactions, content discovery, and automation to uplift business outcomes. These innovations run AI search flows to uncover relevant information through semantic, cross-language, and content understanding; adapt information ranking to individual behaviors; and enable guided conversations to pinpoint answers. Nonetheless, search engines are limited in native AI-enhanced search support, so builders develop middleware to complement search engines to fill in functional gaps. This middleware consists of custom code that runs data flows to stitch data transformations, search queries, and AI enrichments in varying combinations tailored to use cases, datasets, and requirements.

With the new AI search flow builder for OpenSearch, you have a collaborative environment to design and run AI search flows on OpenSearch. You can find the visual designer within OpenSearch Dashboards under AI Search Flows, and get started quickly by launching preconfigured flow templates for popular use cases like semantic, multimodal or hybrid search, and retrieval augmented generation (RAG). Through configurations, you can create customize flows to enrich search and index processes through AI providers like Amazon Bedrock, Amazon SageMaker, Amazon Comprehend, OpenAI, DeepSeek, and Cohere. Flows can be programmatically exported, deployed, and scaled on any OpenSearch 2.19+ cluster through OpenSearch’s existing ingest, index, workflow and search APIs.

In the remainder of the post, we’ll walk through a couple of scenarios to demonstrate the flow builder. First, we’ll enable semantic search on your old keyword-based OpenSearch application without client-side code changes. Next, we’ll create a multi-modal RAG flow, to showcase how you can redefine image discovery within your applications.

AI search flow builder key concepts

Before we get started, let’s cover some key concepts. You can use the flow builder through APIs or a visual designer. The visual designer is recommended for helping you manage workflow projects. Each project contains at least one ingest or search flow. Flows are a pipeline of processor resources. Each processor applies a type of data transform such as encoding text into vector embeddings, or summarizing search results with a chatbot AI service.

Ingest flows are created to enrich data as it’s added to an index. They consist of:

1. A data sample of the documents you want to index.
2. A pipeline of processors that apply transforms on ingested documents.
3. An index constructed from the processed documents.

Search flows are created to dynamically enrich search request and results. They consist of:

1. A query interface based on the search API, defining how the flow is queried and ran.
2. A pipeline of processors that transform the request context or search results.

Generally, the path from prototype to production starts with deploying your AI connectors, designing flows from a data sample, then exporting your flows from a development cluster to a preproduction environment for testing at-scale.

Scenario 1: Enable semantic search on an OpenSearch application without client-side code changes

In this scenario, we have a product catalog that was built on OpenSearch a decade ago. We aim to improve its search quality, and in turn, uplift purchases. The catalog has search quality issues, for instance, a search for “NBA,” doesn’t surface basketball merchandise. The application is also untouched for a decade, so we aim to avoid changes to client-side code to reduce risk and implementation effort.

A solution requires the following:

• An ingest flow to generate text embeddings (vectors) from text in an existing index.
• A search flow that encodes search terms into text embeddings, and dynamically rewrites keyword-type match queries into a k-NN (vector) query to run a semantic search on the encoded terms. The rewrite allows your application to transparently run semantic-type queries through keyword-type queries.

We will also evaluate a second-stage reranking flow, which uses a cross-encoder to rerank results as it can potentially boost search quality.

We’ll accomplish our task through the flow builder. We begin by navigating to AI Search Flows in the OpenSearch Dashboard, and selecting Semantic Search from the template catalog.

This template requires us to select a text embedding model. We’ll use Amazon Bedrock Titan Text, which was deployed as a prerequisite. Once the template is configured, we enter the designer’s main interface. From the preview, we can see that the template consists of a preset ingestion and search flow.

The ingest flow requires us to provide a data sample. Our product catalog is currently served by an index containing the Amazon product dataset, so we import a data sample from this index.

The ingest flow includes a ML Inference Ingest Processor, which generates machine learning (ML) model outputs such as embeddings (vectors) as your data is ingested into OpenSearch. As previously configured, the processor is set to use Amazon Titan Text to generate text embeddings. We map the data field that holds our product descriptions to the model’s inputText field to enable embedding generation.

We can now run our ingest flow, which builds a new index containing our data sample embeddings. We can inspect the index’s contents to confirm that the embeddings were successfully generated.

Once we have an index, we can configure our search flow. We’ll start with updating the query interface, which is preset to a basic match query. The placeholder my_text has to be replaced with the product descriptions. With this update, our search flow can now respond to queries from our legacy application.

The search flow includes an ML Inference Search Processor. As previously configured, it’s set to use Amazon Titan Text. Since it’s added under Transform query, it’s applied to query requests. In this case, it will transform search terms into text embeddings (a query vector). The designer lists the variables from the query interface, allowing us to map the search terms (query.match.text.query), to the model’s inputText field. Text embeddings will now be generated from the search terms whenever our index is queried.

Next, we update the query rewrite configurations, which is preset to rewrite the match query into a k-NN query. We replace the placeholder my_embedding with the query field assigned to your embeddings. Note that we could rewrite this to another query type, including a hybrid query, which may improve search quality.

Let’s compare our semantic and keyword solutions from the search comparison tool. Both solutions are able to find basketball merchandise when we search for “basketball.”

But what happens if we search for “NBA?” Only our semantic search flow returns results because it detects the semantic similarities between “NBA” and “basketball.”

We’ve managed improvements, but we might be able to do better. Let’s see if reranking our search results with a cross-encoder helps. We’ll add a ML Inference Search Processor under Transform response, so that the processor applies to search results, and select Cohere Rerank. From the designer, we see that Cohere Rerank requires a list of documents and the query context as input. Data transformations are needed to package the search results into a format that can be processed by Cohere Rerank. So, we apply JSONPath expressions to extract the query context, flatten data structures, and pack the product descriptions from our documents into a list.

Let’s return to the search comparison tool to compare our flow variations. We don’t observe any meaningful difference in our previous search for “basketball” and “NBA.” However, improvements are observed when we search, “hot weather.” On the right, we see that the second and fifth search hit moved 32 and 62 spots up, and returned “sandals” that are well suited for “hot weather.”

We’re ready to proceed to production, so we export our flows from our development cluster into our preproduction environment, use the workflow APIs to integrate our flows into automations, and scale our test processes through the bulk, ingest and search APIs.

Scenario 2: Use generative AI to redefine and elevate image search

In this scenario, we have photos of millions of fashion designs. We’re looking for a low-maintenance image search solution. We will use generative multimodal AI to modernize image search, eliminating the need for labor to maintain image tags and other metadata.

Our solution requires the following:

• An ingest flow which uses a multimodal model like Amazon Titan Multimodal Embeddings G1 to generate image embeddings.
• A search flow which generates text embeddings with a multimodal model, runs a k-NN query for text to image matching, and sends matching images to a generative model like Anthropic’s Claude Sonnet 3.7 that can operate on text and images.

We’ll start from the RAG with Vector Retrieval template. With this template, we can quickly configure a basic RAG flow. The template requires an embedding and large language model (LLM) that can process text and image content. We use Amazon Bedrock Titan Multimodal G1 and Anthropic’s Claude Sonnet 3.7, respectively.

From the designer’s preview panel, we can see similarities between this template and the semantic search template. Again, we seed the ingest flow with a data sample. Like the previous example, we use the Amazon product dataset except we replace the production descriptions with base64 encoded images because our models require base64 images, and this solution doesn’t require text. We map the base64 image data to the corresponding Amazon Titan G1 inputs to generate embeddings. We then run our ingest flow and confirm that our index contains base64 images and corresponding embeddings.

The initial steps for configuring this search flow are similar to the previous scenario: we update the query interface, map the query text fields to the model inputs for the ML Inference Search Processor, and revise the query rewrite settings. The main difference with this flow is the additional response processor set to use Anthropic’s Claude Sonnet 3.7 to process images.

We need to configure an LLM prompt that includes the query context and instructions for the LLM to play the role of a fashion advisor, and provide commentary about the image payload.

Next, we map the prompt and the base64 image data field to the model’s inputs accordingly.

Let’s test our multimodal RAG flow by searching for “sunset colored dresses.” We observe the following results.

This beautiful dress perfectly captures the essence of a sunset with its gorgeous ombre color gradient. Starting with a soft peachy pink at the top, it transitions into a vibrant orange in the middle, and finishes with a bright yellow at the hem – just like the colors of the sky as the sun dips below the horizon. The dress features a simple, casual silhouette with short cap sleeves and appears to have a comfortable, relaxed fit that would flatter many body types. The tie-dye effect gives it a bohemian, carefree vibe that’s perfect for summer outings, beach vacations, or casual evening events. I recommend this sunset-colored dress because it’s eye-catching, versatile, and embodies that magical sunset glow that can brighten any occasion. The colors would complement warm skin tones particularly well, and you could easily dress it up with gold accessories or keep it casual with simple sandals.

This vibrant tie-dye dress perfectly captures the essence of a sunset with its beautiful gradient of colors. The dress features bands of golden yellow, fiery orange, and rich pink that blend together seamlessly, reminiscent of the sky at dusk. The strapless design showcases the shoulders elegantly, while the gathered waist creates a flattering silhouette that flows into a comfortable, knee-length skirt. This dress would be an excellent choice for summer events, beach vacations, or casual outings. The sunset color palette is not only on-trend but also versatile enough to pair with neutral accessories. I recommend this piece for its eye-catching colors, comfortable fit, and the way it embodies the warm, relaxed feeling of watching a beautiful sunset.

Without any image metadata, OpenSearch finds images of sunset-colored dresses, and responds with accurate and colorful commentary.

Conclusion

The AI search flow builder is available in all AWS Regions that support OpenSearch 2.19+ on OpenSearch Service. To learn more, refer to Building AI search workflows in OpenSearch Dashboards, and the available tutorials on GitHub, which demonstrate how to integrate various AI models from Amazon Bedrock, SageMaker, and other AWS and third-party AI services.

---

About the authors

Dylan Tong is a Senior Product Manager at Amazon Web Services. He leads the product initiatives for AI and machine learning (ML) on OpenSearch including OpenSearch’s vector database capabilities. Dylan has decades of experience working directly with customers and creating products and solutions in the database, analytics and AI/ML domain. Dylan holds a BSc and MEng degree in Computer Science from Cornell University.

Tyler Ohlsen is a software engineer at Amazon Web Services focusing mostly on the OpenSearch Anomaly Detection and Flow Framework plugins.

Mingshi Liu is a Machine Learning Engineer at OpenSearch, primarily contributing to OpenSearch, ML Commons and Search Processors repo. Her work focuses on developing and integrating machine learning features for search technologies and other open-source projects.

Ka Ming Leung (Ming) is a Senior UX designer at OpenSearch, focusing on ML-powered search developer experiences as well as designing observability and cluster management features.