“Car sale application” – SpellCheckComponent – did you really mean that ? (part 5)

The time has come to add another important functionality to our car sale application. It will be the spell checking mechanism with the ability to construct a new query from the suggestions. It has become the main functionality of every search engine so we will also make use of it.

Requirements specification

Our car database is so large that it contains many different names of makes and models. Some of that names could be really hard to spell/write:

    • make: Bugatti
    • model: Veyron
    • make: Daewoo
    • model: Lacetti
    • make: Cadillac
    • model: Brougham
    • make: Ford
    • model: Capri
    • make: Maserati
    • model: Coupe

The query examples, where misspelled words caused the query to provide no search results:

  1. ?q=bugati+weyron
  2. ?q=daewo+laceti
  3. ?q=cadilac+brogham
  4. ?q=ford+kapri
  5. ?q=maseratti+coupe

We would like to add the functionality, that in case of entering incorrect names will be able to suggest the phrase which probably was the intention of an application user. Then we will be able to make use of it to find the documents related to the proper phrase.

solrconfig.xml changes

The most important element, which should be added to the solrconfig.xml configuration file is the solr.SpellCheckComponent. Let’s try to add the simple standard configuration of this component and find out how it works:

<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
    <lst name="spellchecker">
      <str name="classname">solr.IndexBasedSpellChecker</str>
      <str name="spellcheckIndexDir">./spellchecker</str>
      <str name="field">content</str>
      <str name="buildOnCommit">true</str>
    </lst>
</searchComponent>

Let’s explain the attributes used in this component:

    • classname – the name of the class which is the implementation of our spellcheck mechanism. We are using the solr.IndexBasedSpellChecker class, which use a spelling dictionary that is based on the Solr/Lucene index.
    • spellcheckIndexDir – the directory name which holds the spellcheck index.
    • field – the name of the field defined in the schema.xml file, used as the source field to generate the spellcheck index. In our case it will be the “content” field (why? – it will be explained later).
    • buildOnCommit – if the value of this attribute is set to true, then the spellcheck index will be automatically build after the main solr index commit.

Now when we have our component defined, let’s add it to some handler to be able to make use of it. The best option is to add it to our standard, default handler, which would provide query results with the suggestions when hitting only one request. Before the changes, our default handler looked like this:

<requestHandler name="standard" class="solr.SearchHandler" default="true">
     <lst name="defaults">
       <str name="echoParams">explicit</str>
     </lst>
</requestHandler>

Po zmianie, wygląda tak:

<requestHandler name="standard" class="solr.SearchHandler" default="true">
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <str name="spellcheck">true</str>
       <str name="spellcheck.collate">true</str>
     </lst>
     <arr name="last-components">
       <str>spellcheck</str>
     </arr>
</requestHandler>

As we can see, we have added the spellcheck component and yet two another default values:

    • spellcheck – when set to true causes every request should also generate a spellcheck suggestion.
    • spellcheck.collate – when set to true causes the mechanism to choose the best suggestion for every word entered and to construct a new query containing proper words. If the spellchecker recognises a word to be correct, it leaves it unchanged.

schema.xml changes

The possible changes in the schema.xml configuration file would be to add the field which would be used by the solr.SpellCheckComponent component as the source of tokens used for spell checking. The field should contain the data which we would like to be used when creating the spellcheck index. The type of that field should ensure the proper data tokenization. It should be also out of any stemming/lametization filters that could affect the spellcheck results badly.

Our schema already contains the field which fulfil all those requirements – “content” field. Just to remind, it is the default search field used by our search engine. The current field and type definitions look like this:

<field name="content" type="text" indexed="true" stored="false" multiValued="true"/>
<fieldType name="text" positionIncrementGap="100">
 <analyzer>
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   <filter class="solr.PatternReplaceFilterFactory" pattern="'" replacement="" replace="all" />
   <filter class="solr.WordDelimiterFilterFactory"
    generateWordParts="1"
    generateNumberParts="1"
    catenateWords="1"
    stemEnglishPossessive="0"
  />
  <filter class="solr.LowerCaseFilterFactory"/>
 </analyzer>
</fieldType>

There are values of three fields copied to the “content” field: make, model and year:

<copyField source="make" dest="content"/>
<copyField source="model" dest="content"/>
<copyField source="year" dest="content"/>

Let’s create queries

Let’s take the no results queries from the requirements specification and add the spellcheck.q parameter which value will be the same as entered in the q parameter. Now, hitting only one query, we are able to get the search results wit the spellcheck suggestions:

  1. ?q=bugati+weyron&spellcheck.q=bugati+weyron
      <result name="response" numFound="0" start="0" />
      <lst name="spellcheck">
        <lst name="suggestions">
          <lst name="bugati">
            <int name="numFound">1</int>
            <int name="startOffset">0</int>
            <int name="endOffset">6</int>
            <arr name="suggestion">
              <str>bugatti</str>
            </arr>
          </lst>
          <lst name="weyron">
            <int name="numFound">1</int>
            <int name="startOffset">7</int>
            <int name="endOffset">13</int>
            <arr name="suggestion">
              <str>veyron</str>
            </arr>
          </lst>
            <str name="collation">bugatti veyron</str>
          </lst>
      </lst>

      The spellcheck mechanism has corrected the query tokens and the collation functionality has generated the proper phrase query, which now can be simply used in order to provide us the proper search results. Let’s check the rest of the queries:

  2. ?q=daewo+laceti&spellcheck.q=?q=daewo+laceti
      <result name="response" numFound="0" start="0" />
      <lst name="spellcheck">
        <lst name="suggestions">
          <lst name="daewo">
            <int name="numFound">1</int>
            <int name="startOffset">0</int>
            <int name="endOffset">5</int>
            <arr name="suggestion">
              <str>daewoo</str>
            </arr>
          </lst>
          <lst name="laceti">
            <int name="numFound">1</int>
            <int name="startOffset">6</int>
            <int name="endOffset">12</int>
            <arr name="suggestion">
              <str>lacetti</str>
            </arr>
          </lst>
            <str name="collation">daewoo lacetti</str>
          </lst>
      </lst>
  3. ?q=cadilac+brogham&spellcheck.q=cadilac+brogham
      <result name="response" numFound="0" start="0" />
      <lst name="spellcheck">
        <lst name="suggestions">
          <lst name="cadilac">
            <int name="numFound">1</int>
            <int name="startOffset">0</int>
            <int name="endOffset">7</int>
            <arr name="suggestion">
              <str>cadillac</str>
            </arr>
          </lst>
          <lst name="brogham">
            <int name="numFound">1</int>
            <int name="startOffset">8</int>
            <int name="endOffset">15</int>
            <arr name="suggestion">
              <str>brougham</str>
            </arr>
          </lst>
            <str name="collation">cadillac brougham</str>
          </lst>
      </lst>
  4. ?q=ford+kapri& spellcheck.q=?q=ford+kapri
      <result name="response" numFound="0" start="0" />
      <lst name="spellcheck">
        <lst name="suggestions">
          <lst name="kapri">
            <int name="numFound">1</int>
            <int name="startOffset">5</int>
            <int name="endOffset">10</int>
            <arr name="suggestion">
              <str>capri</str>
            </arr>
          </lst>
            <str name="collation">ford capri</str>
          </lst>
      </lst>
  5. ?q=maseratti+coupe&spellcheck.q=?q=maseratti+coupe
      <result name="response" numFound="0" start="0" />
      <lst name="spellcheck">
        <lst name="suggestions">
          <lst name="maseratti">
            <int name="numFound">1</int>
            <int name="startOffset">0</int>
            <int name="endOffset">9</int>
            <arr name="suggestion">
              <str>maserati</str>
            </arr>
          </lst>
            <str name="collation">maserati coupe</str>
          </lst>
      </lst>

The spellcheck mechanism has worked great, correcting all of the misspellings and generating the proper phrase queries. In the last two cases (4,5) we can see that the component has not corrected the properly entered words (4 – ford, 5 – coupe) but used them to construct the proper queries (collation).

The end

Our search engine has now yet another functionality. This time it was the spell checking mechanism. Now all we have to do is to wait for some comments … and maybe some improvements can be provided 🙂

This entry was posted on Monday, May 23rd, 2011 at 08:21 and is filed under About Solr. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

6 Responses to ““Car sale application” – SpellCheckComponent – did you really mean that ? (part 5)”

  1. Roy Says:

    Hello,

    What if the data looks like this:

    Bugattie Veyron
    Ford Weyron

    When a user types “Bugattie Wehron” SOLR returns: Did you mean “Bugattie Weyron”. I could use a keywordTokenizer but i also want spellchecking for “Bugatie”.

    Another problem that that i have:
    If a user enters a word that is in the dictionary, solr will never try to correct it. The only way around this is to use spellcheck.onlyMorePopular. The problem with this approach is “onlyMorePopular” causes the spellchecker to assume *every* word in the query is a misspelling and it won’t even consider the original terms in building collations. What is needed is a hybrid option that will try to build collations using combinations of original terms, corrected terms and “more popular” terms. To my knowledge, there is no way to get the spellchecker to do that currently.

  2. Selvam Says:

    Hi,

    Thanks for article, but I want to get proper case in results For eg “Bugattie Veyron” not “bugattie veyron”. But of course search should be case insensitive.

  3. gr0 Says:

    You can use two separate fields – one lowercased and one not. That should work.

  4. Sooraj Says:

    Hi I’m following the above steps to configure spell checker in solr, but its not working here. I’m attaching the code. Could you please suggest something…

    Schema.xml

    solrconfig.xml

    text_general

    default
    text
    solr.DirectSolrSpellChecker

    internal

    0.5

    2

    1

    5

    4

    0.01
    <!– uncomment this to require suggestions to occur in 1% of the documents
    .01
    –>

    text

    default
    wordbreak
    on
    true
    10
    5
    5
    true
    true
    10
    5

    spellcheck

  5. Diptee Says:

    How can i apply filters in spellcheck query. I do not want the suggestion which does not returns the results after appplying a filter

  6. gr0 Says:

    I’m afraid spellchecking doesn’t allow to use filter queries.