solr.ReversedWildcardFilter

The solr.ReversedWildcardFilter filter provides us with new tokens, which in fact are reverses tokens, that are indexed to provide faster leading wildcard queries. The filter supports the following init arguments:

• withOriginal – if true, then produce both original and reversed tokens at the same positions. If false, then produce only reversed tokens.
• maxPosAsterisk – maximum position (1-based) of the asterisk wildcard (‘*’) that triggers the reversal of query term. Asterisk that occurs at positions higher than this value will not cause the reversal of query term.
• maxPosQuestion – maximum position (1-based) of the question mark wildcard (‘?’) that triggers the reversal of query term.
• maxFractionAsterisk – additional parameter that triggers the reversal if asterisk (‘*’) position is less than this fraction of the query token length.
• minTrailing – minimum number of trailing characters in query token after the last wildcard character. For good performance this should be set to a value larger than 1.

schema.xml changes

New filter is added to the “text” field type:

<fieldType name="text" class="solr.TextField"
	positionIncrementGap="100">
	<analyzer type="index">
		<tokenizer class="solr.WhitespaceTokenizerFactory" />
		<filter class="solr.PatternReplaceFilterFactory" pattern="'"
			replacement="" replace="all" />
		<filter class="solr.WordDelimiterFilterFactory"
			generateWordParts="1" generateNumberParts="1" catenateWords="1"
			stemEnglishPossessive="0" />
		<filter class="solr.LowerCaseFilterFactory" />
		<filter class="solr.ReversedWildcardFilterFactory" />
	</analyzer>
	<analyzer type="query">
		<tokenizer class="solr.WhitespaceTokenizerFactory" />
		<filter class="solr.PatternReplaceFilterFactory" pattern="'"
			replacement="" replace="all" />
		<filter class="solr.WordDelimiterFilterFactory"
			generateWordParts="1" generateNumberParts="1" catenateWords="1"
			stemEnglishPossessive="0" />
		<filter class="solr.LowerCaseFilterFactory" />
	</analyzer>
</fieldType>

solr.ReversedWildcardFilterFactory filter is added only to the index analyzer. We do not define any arguments in the filter definition, because we would like to use the default configuration, which is:

• withOriginal – „true”, we would like to produce original tokens
• maxPosAsterisk – 2
• maxPosQuestion – 1
• maxPosQuestion – 0.0f (disabled)
• maxPosQuestion – 2

Sample data

Let’s index some sample data:

<add>
  <doc>
    <field name="id">1</field>
    <field name="make">Lancia</field>
    <field name="model">Delta</field>
    ...
  </doc>
  <doc>
    <field name="id">2</field>
    <field name="make">Land Rover</field>
    <field name="model">Defender</field>
    ...
  </doc>
  <doc>
    <field name="id">3</field>
    <field name="make">Acura</field>
    <field name="model">MDX</field>
    ...
  </doc>
  <doc>
    <field name="id">4</field>
    <field name="make">Acura</field>
    <field name="model">RDX</field>
    ...
  </doc>
  <doc>
    <field name="id">5</field>
    <field name="make">Acura</field>
    <field name="model">RSX</field>
    ...
  </doc>
</add>

Let’s create queries

Let me remind you that the default search field is the “content” field, that among others contains “make” and “model” field. To analyse query results and solr.ReversedWildcardFilter filter behaviour, we will set the „stored” argument of the „content” field to “true”. We will also add the debugQuery query argument, which will allow us to find out, which tokens are used in the query processing (original or reversed).

1. ?q=lan*&fl=id,content&debugQuery=on

   <result name="response" numFound="2" start="0">
     <doc>
       <arr name="content">
         <str>Lancia</str>
         <str>Delta</str>
         <str>2002</str>
       </arr>
       <str name="id">1</str>
     </doc>
     <doc>
       <arr name="content">
         <str>Land Rover</str>
         <str>Defender</str>
         <str>2002</str>
       </arr>
       <str name="id">2</str>
     </doc>
   </result>
   <lst name="debug">
     <str name="rawquerystring">lan*</str>
     <str name="querystring">lan*</str>
     <str name="parsedquery">content:lan*</str>
     <str name="parsedquery_toString">content:lan*</str>
     ...
   </lst>

   We have used asterisk wildcard (‘*’) at the end of the query (position = 4), so the original tokens were used:

   <str name="parsedquery">content:lan*</str>

2. ?q=*dx&fl=id,content&debugQuery=on

   <result name="response" numFound="2" start="0">
     <doc>
       <arr name="content">
         <str>Acura</str>
         <str>MDX</str>
         <str>2002</str>
       </arr>
       <str name="id">3</str>
     </doc>
     <doc>
       <arr name="content">
         <str>Acura</str>
         <str>RDX</str>
         <str>2003</str>
       </arr>
       <str name="id">4</str>
     </doc>
   </result>
   <lst name="debug">
     <str name="rawquerystring">*dx</str>
     <str name="querystring">*dx</str>
     <str name="parsedquery">content:#1;xd*</str>
     <str name="parsedquery_toString">content:#1;xd*</str>
     ...
   </lst>

   We have used asterisk wildcard (‘*’) at the beginning of the query (position = 1) and additionally we have two trailing characters after the last wildcard. That’s why the revesed tokens were used:

   <str name="parsedquery">content:#1;xd*</str>

   As we can see, the reversed tokens have a special prefix in order to avoid collisions and false matches.

3. ?q=r?x&fl=id,content&debugQuery=on

   <result name="response" numFound="2" start="0">
     <doc>
       <arr name="content">
         <str>Acura</str>
         <str>RDX</str>
         <str>2003</str>
       </arr>
       <str name="id">4</str>
     </doc>
     <doc>
       <arr name="content">
         <str>Acura</str>
         <str>RSX</str>
         <str>2006</str>
       </arr>
       <str name="id">5</str>
     </doc>
   </result>
   <lst name="debug">
     <str name="rawquerystring">r?x</str>
     <str name="querystring">r?x</str>
     <str name="parsedquery">content:r?x</str>
     <str name="parsedquery_toString">content:r?x</str>
     ...
   </lst>

   We have used question mark wildcard (‘?’) on position number 2 and additionally we have only one trailing character after the wildcard. The original tokens were used:

   <str name="parsedquery">content:r?x<</str>

The end

Thanks to the solr.ReversedWildcardFilter filter, we have successfully optimized wildcard queries. “Car sale application” users can now effectively use them

Requirements specification

I would like to be able to create a grouping query, that will show me the number of generated groups and provide only one result within every group – the one with the lowest price in its year group.

New functionality request parameters description

What we need is:

• group.ngroups – boolean type parameter that allows us to include the number of generated groups
• group.sort – parameter describing how to sort documents within a group

Let’s create query

Using the query example from the previous post, let’s add two new parameters:

?q=audi+a4&group=true&group.field=year_group&group.limit=1&fl=id,mileage,make,model,year,price&group.ngroups=true&group.sort=price+asc

As you see, we’ve also set the group.limit parameter to 1 (in order to have the only one result within every group) and extended the value of fl parameter by adding the price field. As a result we have the response:

<lst name="grouped">
  <lst name="year_group">
    <int name="matches">5</int>
    <int name="ngroups">3</int>
    <arr name="groups">
      <lst>
        <str name="groupValue">2002</str>
        <result name="doclist" numFound="2" start="0">
          <doc>
            <str name="id">3</str>
            <str name="make">Audi</str>
            <int name="mileage">125000</int>
            <str name="model">A4</str>
            <float name="price">21300.0</float>
            <int name="year">2002</int>
          </doc>
        </result>
      </lst>
      <lst>
        <str name="groupValue">2003</str>
        <result name="doclist" numFound="2" start="0">
          <doc>
            <str name="id">2</str>
            <str name="make">Audi</str>
            <int name="mileage">220000</int>
            <str name="model">A4</str>
            <float name="price">27800.0</float>
            <int name="year">2003</int>
          </doc>
        </result>
      </lst>
      <lst>
        <str name="groupValue">2006</str>
        <result name="doclist" numFound="1" start="0">
          <doc>
            <str name="id">5</str>
            <str name="make">Audi</str>
            <int name="mileage">9900</int>
            <str name="model">A4</str>
            <float name="price">32100.0</float>
            <int name="year">2006</int>
          </doc>
        </result>
      </lst>
    </arr>
  </lst>
</lst>

As we see in the response, we have a new element that shows us how many groups are generated:

<int name="ngroups">3</int>

We also have only one document in each year group – the car with the lowest price in its year group. You don’t believe me ? Look at the query responses from the previous post and compare the prices

The end

It was a fast review of yet two another grouping parameters. Big thanks to David Martin who gave me the subject by asking some questions in the previous post

Requirements specification

Our car database is so large that it contains many different names of makes and models. Some of that names could be really hard to spell/write:

1. • make: Bugatti
   • model: Veyron
2. • make: Daewoo
   • model: Lacetti
3. • make: Cadillac
   • model: Brougham
4. • make: Ford
   • model: Capri
5. • make: Maserati
   • model: Coupe

The query examples, where misspelled words caused the query to provide no search results:

1. ?q=bugati+weyron
2. ?q=daewo+laceti
3. ?q=cadilac+brogham
4. ?q=ford+kapri
5. ?q=maseratti+coupe

We would like to add the functionality, that in case of entering incorrect names will be able to suggest the phrase which probably was the intention of an application user. Then we will be able to make use of it to find the documents related to the proper phrase.

solrconfig.xml changes

The most important element, which should be added to the solrconfig.xml configuration file is the solr.SpellCheckComponent. Let’s try to add the simple standard configuration of this component and find out how it works:

<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
    <lst name="spellchecker">
      <str name="classname">solr.IndexBasedSpellChecker</str>
      <str name="spellcheckIndexDir">./spellchecker</str>
      <str name="field">content</str>
      <str name="buildOnCommit">true</str>
    </lst>
</searchComponent>

Let’s explain the attributes used in this component:

1. • classname – the name of the class which is the implementation of our spellcheck mechanism. We are using the solr.IndexBasedSpellChecker class, which use a spelling dictionary that is based on the Solr/Lucene index.
   • spellcheckIndexDir – the directory name which holds the spellcheck index.
   • field – the name of the field defined in the schema.xml file, used as the source field to generate the spellcheck index. In our case it will be the “content” field (why? – it will be explained later).
   • buildOnCommit – if the value of this attribute is set to true, then the spellcheck index will be automatically build after the main solr index commit.

Now when we have our component defined, let’s add it to some handler to be able to make use of it. The best option is to add it to our standard, default handler, which would provide query results with the suggestions when hitting only one request. Before the changes, our default handler looked like this:

<requestHandler name="standard" class="solr.SearchHandler" default="true">
     <lst name="defaults">
       <str name="echoParams">explicit</str>
     </lst>
</requestHandler>

Po zmianie, wygląda tak:

<requestHandler name="standard" class="solr.SearchHandler" default="true">
     <lst name="defaults">
       <str name="echoParams">explicit</str>
       <str name="spellcheck">true</str>
       <str name="spellcheck.collate">true</str>
     </lst>
     <arr name="last-components">
       <str>spellcheck</str>
     </arr>
</requestHandler>

As we can see, we have added the spellcheck component and yet two another default values:

1. • spellcheck – when set to true causes every request should also generate a spellcheck suggestion.
   • spellcheck.collate – when set to true causes the mechanism to choose the best suggestion for every word entered and to construct a new query containing proper words. If the spellchecker recognises a word to be correct, it leaves it unchanged.

schema.xml changes

The possible changes in the schema.xml configuration file would be to add the field which would be used by the solr.SpellCheckComponent component as the source of tokens used for spell checking. The field should contain the data which we would like to be used when creating the spellcheck index. The type of that field should ensure the proper data tokenization. It should be also out of any stemming/lametization filters that could affect the spellcheck results badly.

Our schema already contains the field which fulfil all those requirements – “content” field. Just to remind, it is the default search field used by our search engine. The current field and type definitions look like this:

<field name="content" type="text" indexed="true" stored="false" multiValued="true"/>

<fieldType name="text" positionIncrementGap="100">
 <analyzer>
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   <filter class="solr.PatternReplaceFilterFactory" pattern="'" replacement="" replace="all" />
   <filter class="solr.WordDelimiterFilterFactory"
    generateWordParts="1"
    generateNumberParts="1"
    catenateWords="1"
    stemEnglishPossessive="0"
  />
  <filter class="solr.LowerCaseFilterFactory"/>
 </analyzer>
</fieldType>

There are values of three fields copied to the “content” field: make, model and year:

<copyField source="make" dest="content"/>
<copyField source="model" dest="content"/>
<copyField source="year" dest="content"/>

Let’s create queries

Let’s take the no results queries from the requirements specification and add the spellcheck.q parameter which value will be the same as entered in the q parameter. Now, hitting only one query, we are able to get the search results wit the spellcheck suggestions:

1. ?q=bugati+weyron&spellcheck.q=bugati+weyron

   <result name="response" numFound="0" start="0" />
   <lst name="spellcheck">
     <lst name="suggestions">
       <lst name="bugati">
         <int name="numFound">1</int>
         <int name="startOffset">0</int>
         <int name="endOffset">6</int>
         <arr name="suggestion">
           <str>bugatti</str>
         </arr>
       </lst>
       <lst name="weyron">
         <int name="numFound">1</int>
         <int name="startOffset">7</int>
         <int name="endOffset">13</int>
         <arr name="suggestion">
           <str>veyron</str>
         </arr>
       </lst>
         <str name="collation">bugatti veyron</str>
       </lst>
   </lst>

   The spellcheck mechanism has corrected the query tokens and the collation functionality has generated the proper phrase query, which now can be simply used in order to provide us the proper search results. Let’s check the rest of the queries:

2. ?q=daewo+laceti&spellcheck.q=?q=daewo+laceti

   <result name="response" numFound="0" start="0" />
   <lst name="spellcheck">
     <lst name="suggestions">
       <lst name="daewo">
         <int name="numFound">1</int>
         <int name="startOffset">0</int>
         <int name="endOffset">5</int>
         <arr name="suggestion">
           <str>daewoo</str>
         </arr>
       </lst>
       <lst name="laceti">
         <int name="numFound">1</int>
         <int name="startOffset">6</int>
         <int name="endOffset">12</int>
         <arr name="suggestion">
           <str>lacetti</str>
         </arr>
       </lst>
         <str name="collation">daewoo lacetti</str>
       </lst>
   </lst>

3. ?q=cadilac+brogham&spellcheck.q=cadilac+brogham

   <result name="response" numFound="0" start="0" />
   <lst name="spellcheck">
     <lst name="suggestions">
       <lst name="cadilac">
         <int name="numFound">1</int>
         <int name="startOffset">0</int>
         <int name="endOffset">7</int>
         <arr name="suggestion">
           <str>cadillac</str>
         </arr>
       </lst>
       <lst name="brogham">
         <int name="numFound">1</int>
         <int name="startOffset">8</int>
         <int name="endOffset">15</int>
         <arr name="suggestion">
           <str>brougham</str>
         </arr>
       </lst>
         <str name="collation">cadillac brougham</str>
       </lst>
   </lst>

4. ?q=ford+kapri& spellcheck.q=?q=ford+kapri

   <result name="response" numFound="0" start="0" />
   <lst name="spellcheck">
     <lst name="suggestions">
       <lst name="kapri">
         <int name="numFound">1</int>
         <int name="startOffset">5</int>
         <int name="endOffset">10</int>
         <arr name="suggestion">
           <str>capri</str>
         </arr>
       </lst>
         <str name="collation">ford capri</str>
       </lst>
   </lst>

5. ?q=maseratti+coupe&spellcheck.q=?q=maseratti+coupe

   <result name="response" numFound="0" start="0" />
   <lst name="spellcheck">
     <lst name="suggestions">
       <lst name="maseratti">
         <int name="numFound">1</int>
         <int name="startOffset">0</int>
         <int name="endOffset">9</int>
         <arr name="suggestion">
           <str>maserati</str>
         </arr>
       </lst>
         <str name="collation">maserati coupe</str>
       </lst>
   </lst>

The spellcheck mechanism has worked great, correcting all of the misspellings and generating the proper phrase queries. In the last two cases (4,5) we can see that the component has not corrected the properly entered words (4 – ford, 5 – coupe) but used them to construct the proper queries (collation).

The end

Our search engine has now yet another functionality. This time it was the spell checking mechanism. Now all we have to do is to wait for some comments … and maybe some improvements can be provided

<field name="city_sort" type="lowercase" indexed="true" stored="false" />
...
<copyField source="city" dest="city_sort"/>

It turned out, that sorting using the city_sort field did not work as we expected. All because of the polish signs appearing in the city names. What should we do with it ?

Requirements specification

Let’s check if the „city_sort” field sorting does really not working well in conjunction with the polish signs. When we enter the query:

q=*:*&fl=city&sort=city_sort+asc

we have the result:

<result name="response" numFound="6" start="0">
   <doc>
      <str name="city">Białystok</str>
   </doc>
   <doc>
      <str name="city">Koszalin</str>
   </doc>
   <doc>
      <str name="city">Szczecin</str>
   </doc>
   <doc>
      <str name="city">Warszawa</str>
   </doc>
   <doc>
      <str name="city">Świdnik</str>
   </doc>
   <doc>
      <str name="city">Łowicz</str>
   </doc>
</result>

That’s really not what we expect. We would like to have:

<result name="response" numFound="6" start="0">
   <doc>
      <str name="city">Białystok</str>
   </doc>
   <doc>
      <str name="city">Koszalin</str>
   </doc>
   <doc>
      <str name="city">Łowicz</str>
   </doc>
   <doc>
      <str name="city">Szczecin</str>
   </doc>
   <doc>
      <str name="city">Świdnik</str>
   </doc>
   <doc>
      <str name="city">Warszawa</str>
   </doc>
</result>

To make the sorting functionality work well, we will use the „solr.CollationKeyFilter” filter.

solr.CollationKeyFilter

The filter called solr.CollationKeyFilter is used at index time, indexing special “sort keys” into the sort field. It allows us to choose the collator related to wanted country and language. We can also choose the strength of the collation which determines the minimum level of difference considered significant during comparison. For example:

<filter class="solr.CollationKeyFilterFactory" language="es" country=”ES” strength="primary" />

The given example shows us the configuration of the solr.CollationKeyFilterFactory, where we want to handle the spanish language with the primary strength.

Schema.xml changes

1. New field types definitions:

   <fieldType name="polishLowercase" positionIncrementGap="100">
     <analyzer>
       <tokenizer class="solr.KeywordTokenizerFactory"/>
       <filter class="solr.LowerCaseFilterFactory" />
       <filter class="solr.TrimFilterFactory" />
       <filter class="solr.CollationKeyFilterFactory"  language="pl" country=”PL” strength="primary" />
     </analyzer>
   </fieldType>

   As we may notice, it’s the definition of the currently existing „lowercase” type, where we added the solr.CollationKeyFilter, handling the polish language. The type will be used for the fields, where the data contains polish signs.

2. New „city_sort” field definition:
   • let’s change the type for the „city_sort” field to „polishLowercase”:

   <field name="city_sort" type="polishLowercase" indexed="true" stored="false" />

Functional tests

Before we check if the given field type change is just what we need, we must remember that the solr.CollationKeyFilter is used at index time, so we need to re-index all of the data.

Now let’s check our test query result:

q=*:*&fl=city&sort=city_sort+asc

It appears that the result is correct:

<result name="response" numFound="6" start="0">
   <doc>
      <str name="city">Białystok</str>
   </doc>
   <doc>
      <str name="city">Koszalin</str>
   </doc>
   <doc>
      <str name="city">Łowicz</str>
   </doc>
   <doc>
      <str name="city">Szczecin</str>
   </doc>
   <doc>
      <str name="city">Świdnik</str>
   </doc>
   <doc>
      <str name="city">Warszawa</str>
   </doc>
</result>

The end

Yet another reported problem has been solved successfully. We have improved the quality of the sorting mechanism, where we must handle the polish signs, by adding the solr.CollationKeyFilter which entirely fulfilled our needs. Now we can only wait for another notifications and improvements