Autcomplete, part 4 (Ngram and faceting)

In the previous parts of autocomplete series we presented two methods of autocomplete queries. Than we extended one of those with the ability to define returned information. In todays entry we are back to autocomplete with facet and ngram.

Requirements

Our autocomplete mechanism has the following requirements:

  1. We return whole phrase, not just s single word
  2. Returned phrase can be present multiple times in the index
  3. We want to know the number of results for the returned phrase
  4. Common phrases should be shown higher than the less common ones
  5. Order of words entered by the user doesn’t matter

Solution

Solution given in the first part of the series will not met the requirements because of the first requirement. Of course we could change analysis type, but we wouldn’t return the whole phrase.

Solution to the above requirements is the modified faceting method. Instead of searching all the elements and narrowing results with facet.prefix parameter, we can search only for those elements that have the word fragment we are looking for. We don’t want wildcard query to be used (because of performance) we call ngram’s for the rescue. This means we need to write the ngrams into the index (of course Solr will do that for us). The obvious flaw is the index size growth, but in this case we can live with that.

Schema.xml

We define an additional type:

We also define additional fields: one which value we plan to return and one which will be used for searching:

And one copyField to make things easier:

Query

After indexation we are ready to test our queries:

  1. We are narrowing results, only to those which have the interesting word fragment in the tag_autocomplete field, with: q=tag_autocomplete:(PHRASE)
  2. We need all the fragments entered by the user, so we use AND as our logical operator: q.op=AND
  3. We not interested in the actual query results, we will use data returned by faceting, so we say: rows=0
  4. We need faceting: facet=true
  5. We need faceting on the field where we store the original phrase: facet.field=tag
  6. We are not interested in empty tags: facet.mincount=1
  7. We are only interested in 5 autocomplete values: facet.limit=5

And the final query:

If we will configure out search handler to include all the constant parameters, we will have the following query:

At the end

The basic virtue of the presented method is the ability to use one field for searching and other for returning results. Because of that, we were able to return the whole phrase instead of a single word.

6 thoughts on “Autcomplete, part 4 (Ngram and faceting)

  • 29 May 2012 at 02:21
    Permalink

    Hi,

    These posts look extremely useful – but your website is quite difficult to navigate. I have found this “part 4” from Google, but I want to go back to part 3,2,1 to find out how to do this from scratch.

    Clicking the “Autocomplete” tag results in “Page not found” errors.

    As I said, this content looks very useful & I’m thankful that somebody like you is posting this… but if there is any way that you can make the tutorial easily accessible, it would be very much appreciated.

    For example, links to the different parts from whichever page you are on would be fantastic.

    Reply
  • 13 June 2012 at 19:58
    Permalink

    Thanks for posting this – we’ve been struggling with this issue for many months, and this is the most elegant solution we’ve come across.

    Great solution to a common need.

    Reply
  • 31 July 2012 at 22:18
    Permalink

    Thanks for your sharing your elegant solution. I tried it work and it works very well, except for Chinese characters it only returns phrases that contain the words as prefix.

    e.g If I search for “林業”, it will return “林業服務” but not “農業、林業和漁業”

    I replaced WhitespaceTokenizerFactory with CJKTokenizerFactory but it didn’t help.

    Thanks !

    Reply
  • 20 November 2012 at 15:03
    Permalink

    Good Article!

    A minor point of detail: in your example you’ve set the stored attribute for tag_autocomplete to “true”. This is unnecessary and just wastes disk space doesn’t it?

    Reply
  • 22 April 2015 at 08:14
    Permalink

    Thanks for the great post and it is really helpful and helped me alot in my project .

    I would ask you one thing though, how to change the query to make it accept parameters that will represent the number of words to be gramed ?

    what i mean is , is there a way to pass for example the number “3” and the query will be configured to find the 3gram ( for three words) .

    Thanks & Regards,

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.