Understanding span queries in Amazon OpenSearch Service

Post Syndicated from Jon Handler original https://aws.amazon.com/blogs/big-data/understanding-span-queries-in-amazon-opensearch-service/

Amazon OpenSearch Service is a fully managed service for deploying, scaling, and operating OpenSearch in the AWS Cloud. OpenSearch is an open source software suite for search, analytics, and observability, licensed under the Apache License V2.0. Among the OpenSearch query types, span queries give you precise control over where terms appear in a document and how they relate to one another. Most applications get good results from match and phrase queries, but certain workloads require exact term placement that simpler queries cannot express. These include legal search, regulatory compliance, scientific and patent search, and contract analysis. Span queries fill that gap.

In this post, you will learn how span queries work, when to use them instead of phrase queries, and how to compose them to express precise structural constraints on your text.

When phrase queries fall short

The phrase query matches a contiguous sequence of terms in order. The match_phrase query accepts a slop parameter that loosens this constraint by allowing some number of position swaps between terms. A user searching for “star wars iv” expects to find the movie “Star Wars: Episode IV — A New Hope.” However, the exact phrase doesn’t appear in the title because of the intervening word “Episode.” Setting slop to 1 allows OpenSearch to swap one term position and find the match.

To explore this, create an index with a single text field:

PUT term_order
{
  "mappings": {
    "properties": {
      "title": {"type": "text"}
    }
  }
}

Index two documents:

POST term_order/_doc/1
{"title": "Star Wars: Episode IV - A New Hope"}

POST term_order/_doc/2
{"title": "Hope for Star Wars: The Prequel Series"}

A match_phrase query for “star wars iv” with slop: 1 matches the first document. Change the query to “star wars hope” with slop: 4, and the query matches both documents:

GET term_order/_search
{
  "query": {
    "match_phrase": {
      "title": {
        "query": "star wars hope",
        "slop": 4
      }
    }
  }
}

The match against “Hope for Star Wars: The Prequel Series” is unexpected. The order of terms in that document is different from the query. Phrase slop performs an edit-distance match, swapping terms within the edit window regardless of order. When you need to enforce term order, position within a field, or structural relationships between terms, phrase queries lack the expressiveness.

What span queries provide

Span queries are compound queries that give you precise control over the position of terms relative to one another. Span queries only contain other span queries. You can’t mix them with non-span queries inside the same compound expression. OpenSearch supports several span query types:

span_term: Matches a single term. This is typically the leaf of a compound span query.
span_multi: Wraps a multi-term query (prefix, regex, wildcard) so it can be used inside a span query.
span_first: Matches spans at the start of a field, within a specified number of positions.
span_near: Matches a set of span clauses within a specified slop, similar to match_phrase but operating on spans.
span_not: Excludes specific terms from a span match.
span_or: Unions multiple span clauses.
span_field_masking: Uses a span_term from another field, masked to appear as the primary field.
span_containing: Matches spans that contain other spans.
span_within: Matches spans that are contained within other spans. span_within is the mirror of span_containing.

All span queries accept an in_order parameter that controls whether sub-spans must appear in the order of the query clauses.

Anchoring a term to a position

To find “A New Hope” in the index, you can require “star” at the beginning of the title and “hope” within 12 terms after it:

GET term_order/_search
{
  "query": {
    "span_near": {
      "clauses": [
        {
          "span_first": {
            "match": {"span_term": {"title": "star"}},
            "end": 2
          }
        },
        {"span_term": {"title": {"value": "hope"}}}
      ],
      "slop": 12,
      "in_order": true
    }
  }
}

This query matches only “Star Wars: Episode IV — A New Hope.” The span_first clause requires “star” within the first two positions of the title. The enclosing span_near then requires “hope” within 12 positions, in the specified order. The second document doesn’t match because “star” isn’t at the start of the title.

Spans within spans

With span_containing and span_within, you can express structural constraints by matching one span inside another. Both take a big and a little clause: the little span must be contained by the big span. The difference is which span the query returns. span_containing returns the big matches, and span_within returns the little matches.

For example, to find documents where the phrase “episode iv” appears within a span containing both “star” and “hope”:

GET term_order/_search
{
  "query": {
    "span_containing": {
      "little": {
        "span_near": {
          "clauses": [
            {"span_term": {"title": {"value": "episode"}}},
            {"span_term": {"title": {"value": "iv"}}}
          ],
          "slop": 0,
          "in_order": true
        }
      },
      "big": {
        "span_near": {
          "clauses": [
            {"span_term": {"title": {"value": "star"}}},
            {"span_term": {"title": {"value": "hope"}}}
          ],
          "slop": 12,
          "in_order": false
        }
      }
    }
  }
}

The little clause matches “episode iv” as an exact phrase (slop is 0, in_order is true). The big clause matches a span containing “star” and “hope” within 12 terms of one another, in either order. The span_containing query then requires the little phrase to fall within a big match. Only “Star Wars: Episode IV — A New Hope” satisfies both constraints.

When to use span queries

Span queries are useful any time you need to express constraints that phrase queries cannot. In legal and regulatory search, statutes and contracts often require specific phrases to appear in proximity. One example is finding documents where “force majeure” appears within five terms of “termination.” Span queries give you the precise control to express that requirement.

In patent and scientific search, claims and papers frequently require terms to appear in specific structural positions relative to one another. Compliance and audit workflows that must enforce exact term placement to satisfy regulatory definitions benefit from the same precision. More generally, when your text doesn’t have an explicit structured model but you need to express constraints about where terms appear within a field, span queries provide a query-time alternative to reshaping your data.

Conclusion

Span queries trade simplicity for precision. Span queries are more verbose than phrase queries and require careful composition. In return, you can use them to express constraints that phrase queries can’t: exact ordering, position within a field, and containment relationships. For applications where term placement carries meaning, span queries are the right tool.

For more information, see the OpenSearch span query documentation and the Amazon OpenSearch Service documentation.

Noise