How to use a search query to optimise category page filter performance

Overview

There are occasions where you may want to use filters to curate the initial search results that are loaded when a page is displayed. This is often the case with category or landing pages. The example is a below is a typical query which may be used to load a headphones category page.

{
  "q": "",
  "filter": "shopNavigationCategoryHierarchy ~ 'Electronics > Headphones'",
  "fields": ""
}

CODE

As the query above contains no search, all of the records in the collection need to be assessed to determine if they pass the filter criteria. This is highly inefficient from a query processing point of view.

A more efficient approach is to run a text search against the category name to generate a smaller results set and then apply the category filter to that smaller set of results

The guide below explains how to use a dedicated pipeline and a search query like the example below to search for "Electronics > Headphones" textual matches within the shopNavigationCategoryHierarchy field.

{
  "q": "Electronics > Headphones",
  "filter": "category2='Headphones'",
  "fields": ""
}

CODE

Configuration guide

The shopNavigationCategoryHierarchy field contains the values we need to match against our new query. In the record pipeline example below this field along with title, description etc are being assigned as index fields

- id: create-indexes
  params:
    fields:
      constant: title,description,brand,shopNavigationCategoryHierarchy

CODE

Usually data being ingested into the index will be modified through stemming and removal of stop words and punctuation before its stored as values within the index. In this example we want to keep the original values assigned to shopNavigationCategoryHierarchy so we use - id: copy-values to copy the unmodified values of to a new field in the Record pipeline called shopNavigationCategoryHierarchy_zxx

- id: copy-values
  params:
    fields:
      constant: shopNavigationCategoryHierarchy:shopNavigationCategoryHierarchy_zxx

CODE

We then want to use - id: create-indexes to add shopNavigationCategoryHierarchy_zxx to our index and assign lang a default value of zxx which means no language processing/modification will occur to the values this field is assigned during indexing.

- id: create-indexes
  params:
    fields:
      constant: shopNavigationCategoryHierarchy_zxx
    lang:
      defaultValue: zxx

CODE

We want shopNavigationCategoryHierarchy_zxx to be searchable, but as it's not needed in the search result we use - id: drop-values to remove it

- id: drop-values
  params:
    fields:
      constant: shopNavigationCategoryHierarchy_zxx

CODE

We now have unmodified values assigned to shopNavigationCategoryHierarchy_zxx which completes the work in the Record pipeline.

The next step is to create a new Query pipeline called categoryPages to display results for category pages. Within the pipeline we’ll use - id: set-param-value to assign zxx to lang so that this pipeline does not apply any language processing to the query text.

- id: set-param-value
  params:
    param:
      bind: lang
    value:
      constant: zxx
  condition: '!lang'

CODE

Our final configuration step. We want the categoryPages pipeline to look at the category being passed as search text with the query and only return records in the result set which have a matching value within shopNavigationCategoryHierarchy_zxx

To do this we add a single - id: index-text-index-boost to the categoryPages pipeline and assign the shopNavigationCategoryHierarchy_zxx field to it.

- id: index-text-index-boost
  params:
    field:
      constant: shopNavigationCategoryHierarchy_zxx
    score:
      constant: "0.2"
    text:
      bind: q

CODE

Querying the new categoryPages pipeline

To load category page results the category shopNavigationCategoryHierarchy value now just needs to be passed in as a search query, rather than just a filter.

This will increase the speed of the query response by around 5-10x.

{
  "q": "Electronics > Headphones",
  "filter": "category2='Headphones'",
  "fields": ""
}

CODE