Autcomplete, part 4 (Ngram and faceting)

In the previous parts of autocomplete series we presented two methods of autocomplete queries. Than we extended one of those with the ability to define returned information. In todays entry we are back to autocomplete with facet and ngram.

Requirements

Our autocomplete mechanism has the following requirements:

1. We return whole phrase, not just s single word
2. Returned phrase can be present multiple times in the index
3. We want to know the number of results for the returned phrase
4. Common phrases should be shown higher than the less common ones
5. Order of words entered by the user doesn’t matter

Solution

Solution given in the first part of the series will not met the requirements because of the first requirement. Of course we could change analysis type, but we wouldn’t return the whole phrase.

Solution to the above requirements is the modified faceting method. Instead of searching all the elements and narrowing results with facet.prefix parameter, we can search only for those elements that have the word fragment we are looking for. We don’t want wildcard query to be used (because of performance) we call ngram’s for the rescue. This means we need to write the ngrams into the index (of course Solr will do that for us). The obvious flaw is the index size growth, but in this case we can live with that.

Schema.xml

We define an additional type:

  <fieldType name="text_autocomplete" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="25" />
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
  </fieldType>

We also define additional fields: one which value we plan to return and one which will be used for searching:

<field name="tag_autocomplete" type="text_autocomplete" indexed="true" stored="true" omitNorms="true" omitTermFreqAndPositions="true"/>
<field name="tag" type="string" indexed="true" stored="true" />

And one copyField to make things easier:

<copyField source="tag" dest="tag_autocomplete"/>

Query

After indexation we are ready to test our queries:

1. We are narrowing results, only to those which have the interesting word fragment in the tag_autocomplete field, with: q=tag_autocomplete:(PHRASE)
2. We need all the fragments entered by the user, so we use AND as our logical operator: q.op=AND
3. We not interested in the actual query results, we will use data returned by faceting, so we say: rows=0
4. We need faceting: facet=true
5. We need faceting on the field where we store the original phrase: facet.field=tag
6. We are not interested in empty tags: facet.mincount=1
7. We are only interested in 5 autocomplete values: facet.limit=5

And the final query:

?q=tag_autocomplete:(PHRASE)&q.op=AND&rows=0&facet=true&facet.field=tag&facet.mincount=1&facet.limit=5

If we will configure out search handler to include all the constant parameters, we will have the following query:

?q=tag_autocomplete:(PHRASE)

At the end

The basic virtue of the presented method is the ability to use one field for searching and other for returning results. Because of that, we were able to return the whole phrase instead of a single word.