Autcomplete, part 4 (Ngram and faceting)

In the previous parts of autocomplete series we presented two methods of autocomplete queries. Than we extended one of those with the ability to define returned information. In todays entry we are back to autocomplete with facet and ngram.


Our autocomplete mechanism has the following requirements:

  1. We return whole phrase, not just s single word
  2. Returned phrase can be present multiple times in the index
  3. We want to know the number of results for the returned phrase
  4. Common phrases should be shown higher than the less common ones
  5. Order of words entered by the user doesn’t matter


Solution given in the first part of the series will not met the requirements because of the first requirement. Of course we could change analysis type, but we wouldn’t return the whole phrase.

Solution to the above requirements is the modified faceting method. Instead of searching all the elements and narrowing results with facet.prefix parameter, we can search only for those elements that have the word fragment we are looking for. We don’t want wildcard query to be used (because of performance) we call ngram’s for the rescue. This means we need to write the ngrams into the index (of course Solr will do that for us). The obvious flaw is the index size growth, but in this case we can live with that.


We define an additional type:

  <fieldType name="text_autocomplete" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="25" />
    <analyzer type="query">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>

We also define additional fields: one which value we plan to return and one which will be used for searching:

<field name="tag_autocomplete" type="text_autocomplete" indexed="true" stored="true" omitNorms="true" omitTermFreqAndPositions="true"/>
<field name="tag" type="string" indexed="true" stored="true" />

And one copyField to make things easier:

<copyField source="tag" dest="tag_autocomplete"/>


After indexation we are ready to test our queries:

  1. We are narrowing results, only to those which have the interesting word fragment in the tag_autocomplete field, with: q=tag_autocomplete:(PHRASE)
  2. We need all the fragments entered by the user, so we use AND as our logical operator: q.op=AND
  3. We not interested in the actual query results, we will use data returned by faceting, so we say: rows=0
  4. We need faceting: facet=true
  5. We need faceting on the field where we store the original phrase: facet.field=tag
  6. We are not interested in empty tags: facet.mincount=1
  7. We are only interested in 5 autocomplete values: facet.limit=5

And the final query:


If we will configure out search handler to include all the constant parameters, we will have the following query:


At the end

The basic virtue of the presented method is the ability to use one field for searching and other for returning results. Because of that, we were able to return the whole phrase instead of a single word.

This post is also available in: Polish

This entry was posted on Monday, May 28th, 2012 at 07:36 and is filed under Autocomplete. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

6 Responses to “Autcomplete, part 4 (Ngram and faceting)”

  1. jay Says:


    These posts look extremely useful – but your website is quite difficult to navigate. I have found this “part 4” from Google, but I want to go back to part 3,2,1 to find out how to do this from scratch.

    Clicking the “Autocomplete” tag results in “Page not found” errors.

    As I said, this content looks very useful & I’m thankful that somebody like you is posting this… but if there is any way that you can make the tutorial easily accessible, it would be very much appreciated.

    For example, links to the different parts from whichever page you are on would be fantastic.

  2. Cristofer Says:

    Hi jay

    Same problem here, but I found these links searching on Google:

  3. shahne Says:

    Thanks for posting this – we’ve been struggling with this issue for many months, and this is the most elegant solution we’ve come across.

    Great solution to a common need.

  4. Clifford Says:

    Thanks for your sharing your elegant solution. I tried it work and it works very well, except for Chinese characters it only returns phrases that contain the words as prefix.

    e.g If I search for “林業”, it will return “林業服務” but not “農業、林業和漁業”

    I replaced WhitespaceTokenizerFactory with CJKTokenizerFactory but it didn’t help.

    Thanks !

  5. essential Says:

    Good Article!

    A minor point of detail: in your example you’ve set the stored attribute for tag_autocomplete to “true”. This is unnecessary and just wastes disk space doesn’t it?

  6. muaiyed Alghamdi Says:

    Thanks for the great post and it is really helpful and helped me alot in my project .

    I would ask you one thing though, how to change the query to make it accept parameters that will represent the number of words to be gramed ?

    what i mean is , is there a way to pass for example the number “3” and the query will be configured to find the 3gram ( for three words) .

    Thanks & Regards,