“Car sale application” – SpellCheckComponent – did you really mean that ? (part 5)

The time has come to add another important functionality to our car sale application. It will be the spell checking mechanism with the ability to construct a new query from the suggestions. It has become the main functionality of every search engine so we will also make use of it.

Requirements specification

Our car database is so large that it contains many different names of makes and models. Some of that names could be really hard to spell/write:

- make: Bugatti
- model: Veyron
- make: Daewoo
- model: Lacetti
- make: Cadillac
- model: Brougham
- make: Ford
- model: Capri
- make: Maserati
- model: Coupe

The query examples, where misspelled words caused the query to provide no search results:

?q=bugati+weyron
?q=daewo+laceti
?q=cadilac+brogham
?q=ford+kapri
?q=maseratti+coupe

We would like to add the functionality, that in case of entering incorrect names will be able to suggest the phrase which probably was the intention of an application user. Then we will be able to make use of it to find the documents related to the proper phrase.

solrconfig.xml changes

The most important element, which should be added to the solrconfig.xml configuration file is the solr.SpellCheckComponent. Let’s try to add the simple standard configuration of this component and find out how it works:

<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
    <lst name="spellchecker">
      <str name="classname">solr.IndexBasedSpellChecker</str>
      <str name="spellcheckIndexDir">./spellchecker</str>
      <str name="field">content</str>
      <str name="buildOnCommit">true</str>
    </lst>
</searchComponent>

Let’s explain the attributes used in this component:

- classname – the name of the class which is the implementation of our spellcheck mechanism. We are using the solr.IndexBasedSpellChecker class, which use a spelling dictionary that is based on the Solr/Lucene index.
- spellcheckIndexDir – the directory name which holds the spellcheck index.
- field – the name of the field defined in the schema.xml file, used as the source field to generate the spellcheck index. In our case it will be the “content” field (why? – it will be explained later).
- buildOnCommit – if the value of this attribute is set to true, then the spellcheck index will be automatically build after the main solr index commit.

Now when we have our component defined, let’s add it to some handler to be able to make use of it. The best option is to add it to our standard, default handler, which would provide query results with the suggestions when hitting only one request. Before the changes, our default handler looked like this:

<requestHandler name="standard" class="solr.SearchHandler" default="true">
     <lst name="defaults">
       <str name="echoParams">explicit</str>
     </lst>
</requestHandler>

Po zmianie, wygląda tak:

<requestHandler name="standard" class="solr.SearchHandler" default="true">
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <str name="spellcheck">true</str>
       <str name="spellcheck.collate">true</str>
     </lst>
     <arr name="last-components">
       <str>spellcheck</str>
     </arr>
</requestHandler>

As we can see, we have added the spellcheck component and yet two another default values:

- spellcheck – when set to true causes every request should also generate a spellcheck suggestion.
- spellcheck.collate – when set to true causes the mechanism to choose the best suggestion for every word entered and to construct a new query containing proper words. If the spellchecker recognises a word to be correct, it leaves it unchanged.

schema.xml changes

The possible changes in the schema.xml configuration file would be to add the field which would be used by the solr.SpellCheckComponent component as the source of tokens used for spell checking. The field should contain the data which we would like to be used when creating the spellcheck index. The type of that field should ensure the proper data tokenization. It should be also out of any stemming/lametization filters that could affect the spellcheck results badly.

Our schema already contains the field which fulfil all those requirements – “content” field. Just to remind, it is the default search field used by our search engine. The current field and type definitions look like this:

<field name="content" type="text" indexed="true" stored="false" multiValued="true"/>

<fieldType name="text" positionIncrementGap="100">
 <analyzer>
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   <filter class="solr.PatternReplaceFilterFactory" pattern="'" replacement="" replace="all" />
   <filter class="solr.WordDelimiterFilterFactory"
    generateWordParts="1"
    generateNumberParts="1"
    catenateWords="1"
    stemEnglishPossessive="0"
  />
  <filter class="solr.LowerCaseFilterFactory"/>
 </analyzer>
</fieldType>

There are values of three fields copied to the “content” field: make, model and year:

<copyField source="make" dest="content"/>
<copyField source="model" dest="content"/>
<copyField source="year" dest="content"/>

Let’s create queries

Let’s take the no results queries from the requirements specification and add the spellcheck.q parameter which value will be the same as entered in the q parameter. Now, hitting only one query, we are able to get the search results wit the spellcheck suggestions:

?q=bugati+weyron&spellcheck.q=bugati+weyron

<result name="response" numFound="0" start="0" />
<lst name="spellcheck">
  <lst name="suggestions">
    <lst name="bugati">
      <int name="numFound">1</int>
      <int name="startOffset">0</int>
      <int name="endOffset">6</int>
      <arr name="suggestion">
        <str>bugatti</str>
      </arr>
    </lst>
    <lst name="weyron">
      <int name="numFound">1</int>
      <int name="startOffset">7</int>
      <int name="endOffset">13</int>
      <arr name="suggestion">
        <str>veyron</str>
      </arr>
    </lst>
      <str name="collation">bugatti veyron</str>
    </lst>
</lst>

The spellcheck mechanism has corrected the query tokens and the collation functionality has generated the proper phrase query, which now can be simply used in order to provide us the proper search results. Let’s check the rest of the queries:

?q=daewo+laceti&spellcheck.q=?q=daewo+laceti

<result name="response" numFound="0" start="0" />
<lst name="spellcheck">
  <lst name="suggestions">
    <lst name="daewo">
      <int name="numFound">1</int>
      <int name="startOffset">0</int>
      <int name="endOffset">5</int>
      <arr name="suggestion">
        <str>daewoo</str>
      </arr>
    </lst>
    <lst name="laceti">
      <int name="numFound">1</int>
      <int name="startOffset">6</int>
      <int name="endOffset">12</int>
      <arr name="suggestion">
        <str>lacetti</str>
      </arr>
    </lst>
      <str name="collation">daewoo lacetti</str>
    </lst>
</lst>

?q=cadilac+brogham&spellcheck.q=cadilac+brogham

<result name="response" numFound="0" start="0" />
<lst name="spellcheck">
  <lst name="suggestions">
    <lst name="cadilac">
      <int name="numFound">1</int>
      <int name="startOffset">0</int>
      <int name="endOffset">7</int>
      <arr name="suggestion">
        <str>cadillac</str>
      </arr>
    </lst>
    <lst name="brogham">
      <int name="numFound">1</int>
      <int name="startOffset">8</int>
      <int name="endOffset">15</int>
      <arr name="suggestion">
        <str>brougham</str>
      </arr>
    </lst>
      <str name="collation">cadillac brougham</str>
    </lst>
</lst>

?q=ford+kapri& spellcheck.q=?q=ford+kapri

<result name="response" numFound="0" start="0" />
<lst name="spellcheck">
  <lst name="suggestions">
    <lst name="kapri">
      <int name="numFound">1</int>
      <int name="startOffset">5</int>
      <int name="endOffset">10</int>
      <arr name="suggestion">
        <str>capri</str>
      </arr>
    </lst>
      <str name="collation">ford capri</str>
    </lst>
</lst>

?q=maseratti+coupe&spellcheck.q=?q=maseratti+coupe

<result name="response" numFound="0" start="0" />
<lst name="spellcheck">
  <lst name="suggestions">
    <lst name="maseratti">
      <int name="numFound">1</int>
      <int name="startOffset">0</int>
      <int name="endOffset">9</int>
      <arr name="suggestion">
        <str>maserati</str>
      </arr>
    </lst>
      <str name="collation">maserati coupe</str>
    </lst>
</lst>

The spellcheck mechanism has worked great, correcting all of the misspellings and generating the proper phrase queries. In the last two cases (4,5) we can see that the component has not corrected the properly entered words (4 – ford, 5 – coupe) but used them to construct the proper queries (collation).

The end

Our search engine has now yet another functionality. This time it was the spell checking mechanism. Now all we have to do is to wait for some comments … and maybe some improvements can be provided 🙂

Solr.pl