The Idea Behind Rank Query Parser

The idea behind the Rank Query Parser is that it provides the functionality of using the information from the document to modify the score of the resulting documents. It provides a subset of what the Function Query Parser already provided, but it can also be used with the BlockMax-WAND algorithm for improved query performance. 

The RankField

Using RankField is very simple. We need to define the appropriate field type, a field using that field type, and of course, populate it with data. Let’s assume we have the following document structure:

{
  "id" : 1,
  "name": "RankField and RankQueryParser",
  "type": "post",
  "views": 1000 
}

We have the document identifier, the name of the document, its type, and the number of views. We will be interested in the last field. In addition to using it for display purposes, we would also like to use it for ranking. Our schema could look as follows:

<field name="id" type="string" />
<field name="name" type="text_ws" />
<field name="type" type="string" />
<field name="views" type="rank" />

We also need to define the rank type, which could look as follows:

<fieldType name="rank" class="solr.RankField" />

That is everything we need – we are ready to go.

Using the Rank Query Parser

To simply use the RankQueryParser and include the views field in the scoring calculation we could run a query similar to the following one:

q=_query_:{!rank f='views' function='log'}

Knowing that we have two documents that look as follows:

[
  {
    "id" : 1,
    "name": "RankField and RankQueryParser",
    "type": "post",
    "views": 1000 
  },
  {
    "id" : 2,
    "name": "Lucene and Solr 8.6.1 were released",
    "type": "announcement",
    "views": 10
  }
]

Our results would look like this:

{
  "responseHeader":{
    "zkConnected":true,
    "status":0,
    "QTime":3,
    "params":{
      "q":"_query_:{!rank f='views' function='log'}",
      "fl":"score,*"}},
  "response":{"numFound":2,"start":0,"maxScore":6.908755,"numFoundExact":true,"docs":[
      {
        "id":"1",
        "name":"RankField and RankQueryParser",
        "type":"post",
        "_version_":1678886835690930176,
        "score":6.908755},
      {
        "id":"2",
        "name": "Lucene and Solr 8.6.1 were released",
        "type":"announcement",
        "_version_":1678886835758039040,
        "score":2.3978953}]
  }}

You can see that even though we’ve run the match all query that gives a score of 1.0 to all matching documents, the score in our case is different. Solr took the log function and applied it to all matching results.

Performance

Of course, the above behavior can be easily achieved by using a standard Function Query Parser, but the key point with the Rank Query Parser is that we can use the BlockMax-WAND algorithm to improve the performance of our query. To do this we need to include the minExactCount parameter to our query to define how many accurate hits need to be present in the results. After that, Solr may skip documents that do not enter the top N results matching the query.

The response from Solr when minExactCount parameter is used look as follows:

{
  "responseHeader":{
    "zkConnected":true,
    "status":0,
    "QTime":1,
    "params":{
      "q":"_query_:{!rank f='views' function='log'}",
      "fl":"score,*",
      "minExactCount":"1"}},
  "response":{"numFound":2,"start":0,"maxScore":6.908755,"numFoundExact":true,"docs":[
      {
        "id":"1",
        "name":"RankField and RankQueryParser",
        "type":"post",
        "_version_":1678886835690930176,
        "score":6.908755},
      {
        "id":"2",
        "name":"Lucene and Solr 8.6.1 were released",
        "type":"announcement",
        "_version_":1678886835758039040,
        "score":2.3978953}]
  }}

You can see an additional numFoundExact attribute in the response header. We will talk about the BlockMax-WAND algorithm in Solr in the next few weeks in a dedicated blog post, so stay tuned if you would like to read about it. There are some pros and cons to it that I think is worth discussing. 

Available Functions

At the moment of writing the blog post there are three functions available that we can use with the Rank Query Parser:

• log – the logarithmic function, which accepts weight and scalingFactor attributes
• satu – the saturation function accepting the pivot and weight attributes
• sigm – the sigmoid function accepting the pivot, weight, and exponent attributes

You can use one of those functions to scale the scoring factor and adjust how the rank field value affects the scoring.

Conclusions

Though we already had the ability to include the function query in our queries and use the field value from it we can now also use the BlockMax-WAND algorithm. This allows improving the query performance in situations where we don’t need the exact number of rows and we are happy with only top N results. Something worth considering.

Let’s start with the list

By using the defType parameter during query time, we can specify which query parser should be used as the default one during query execution. However, as you’ll see in the examples this is not the only way – we can also use local params. However, before short description of each of the query parsers available in Solr we would like to list them all. In Solr 4.4 the following query parsers can be used:

• lucene
• lucenePlusSort
• func
• prefix
• boost
• dismax
• edismax
• field
• raw
• term
• query
• frange
• geofilt
• bbox
• join
• surround
• switch
• maxscore

Lucene Query Parser – lucene

Default query parser used by Solr that allows us to use Lucene query syntax to make queries (with some small differences that are mentioned on Solr wiki: http://wiki.apache.org/solr/SolrQuerySyntax). An example query made with this parser can look like this: q=+title:harry +category:books

Old Lucene Query Parser – lucenePlusSort

Old Lucene query parser, that is currently rarely used. It allowed after query sorting specification, that is marked as deprecated, for example: q={!lucenePlusSort}title:harry;price asc

Function Query Parser – func

Query parser that allows us to use function queries (http://wiki.apache.org/solr/FunctionQuery) in our queries. For example we can make calculations during queries – q={!func}add($value1,$value2)&value1=max(price,100)&value2=1.0.

Prefix Query Parser – prefix

Parser that is not field type aware, so no analysis is done on the input text. It allows us to make prefix queries. For example a query like q={!prefix}har is more of less equal to q=har* in Lucene query parser syntax.

Boost Query Parser – boost

Boost query parser allows us to create boosting queries from the input value. When using local params we have our query that should be boosted and the b parameter – the query that will be used for boosting. For example {!boost b=recip(ms(NOW,published),3.16e-11,1,1)}harry will cause the harry query to be boosted on the basis of publication date.

Dismax Query Parser – dismax

Query parser that supports simplified version of Lucene query syntax (for example field names are not allowed) and creates disjunction max queries from each term from the user input against all the fields provided in the qf parameter. It allows specifying how much terms should match (the mm parameter) and how lower scoring query elements should be treated (tie parameter). All the parameters supported by dismax query parser (and its overview) can be found at: http://wiki.apache.org/solr/DisMaxQParserPlugin.

Extended Dismax Query Parser – edismax

Extended version of the mentioned dismax query parser, which allows us to use field names in the main query. It was designed to provide users with the possibility of specifying more advanced queries in comparison to dismax query parser. All the parameters supported by extended dismax query parser (and its overview) can be found at: http://wiki.apache.org/solr/ExtendedDisMax.

Field Query Parser – field

Field query parser allows us to make a query against a field, applying text analysis and constructing a phrase query if appropriate, for example: {!field f=title}harry potter. This is more or less equivalent to Lucene query parser expression of title:”harry potter”.

Raw Query Parser – raw

Query parser which allows us to pass a term, which will not be analysed or transformed in any way. For example, the {!raw f=title}harry will create a term query against the title field with the exact value of harry, without any analysis done.

Term Query Parser – term

Query parser useful for making filters with human readable terms, like the ones for faceting or terms component, for example {!term f=category}book. Please remember, that the term given for the term query parser will not be analyzed or transformed.

Nested Query Parser – query

A query parser that allows us to specify a nested query, with the ability to change its type in the nested query. For example, the following query {!query defType=func v=$query}&$query=max(price,100) would result in function query defined in the query parameter, however if we would change the query parameter to $query={!term f=category}book, the term query parser would be used, so the function query parser would be overwritten. This query parser can be useful in situations, when we allow users to reference values stored in handlers configurations.

Function Range Query Parser – frange

Query parser that allows us to create a range query over a function value (we’ve wrote about it long time ago – http://solr.pl/en/2011/05/30/quick-look-frange/). It exposes four parameters l (lower bound), u (upper bound), incl (include lower bound), incu (include upper bound) which allows us to configure its behavior. An example query using this query parser could look like this: {!frange l=10 u=12 incl=true incu=true}sum(price,rate).

Spatial Filter Query Parser – geofilt

Spatial filter query parser allows us to make queries which results will be narrowed to documents that are in the distance from a given point. For example, by running a query like q=*:*&fq={!geofilt pt=52.14,21.10 sfield=location d=50} we will find documents that are more or less in Warsaw, Poland You can find more about geofilt query parser on Solr wiki – http://wiki.apache.org/solr/SpatialSearch#geofilt_-_The_distance_filter.

Spatial Box Query Parser – bbox

Similar to geofilt query parser, the spatial box query parser allows us to narrow down query results to only those documents, that are in the location of interest, but because of approximations it is less demanding, when it comes to processing power (however you may get slightly more results when comparing to geofilt). An example query would be very similar to the one shown in geofilt example: q=*:*&fq={!bbox pt=52.14,21.10 sfield=location d=50}. You can find more about it on Solr wiki – http://wiki.apache.org/solr/SpatialSearch#bbox_-_Bounding-box_filter.

Join Query Parser – join

Query parser that allows us to make pseudo joins between documents, both coming from the same core and ones that are between different cores, for example: q={!join from=parent to=id}color:Yellow. You can see an example and read more about this query parser in http://solr.pl/en/2011/02/21/waiting-for-4-0-solr-2272-solr-and-join-functionality/.

Surround Query Parser – surround

Contrib query parser, that allows us to use span query in Solr. You can read more about this query parser at: http://wiki.apache.org/solr/SurroundQueryParser. We will try to write a blog post about how span queries work both in Lucene and Solr – in general it allows searching for terms or phrases that use position information.

Switch Query Parser – switch

The logic of switch query parser is quite simple – allow processing of a simple logic on the Solr side and add it as a sub-query. You can find an example usage and read more about this query parser at: http://solr.pl/en/2013/06/03/switch-query-parser-quick-look/.

Max Score Query Parser – maxscore

Very similar to Lucene query parser (accept all of its parameters), but the score of the documents is not the sum of all scoring elements, but the score of the maximum scoring element. It allows to use the tie parameter which defines how the final document score will be calculated (similar to how tie parameter works in dismax query parser and extended dismax query parser). For tie=0.0 only the score of the maximum scoring element will be used, for tie=1.0 a sum of all scoring elements will be given to a document. An example query using this parser can look like this: q=potter {!maxscore v=’harry’}. You can read more about tie parameter at http://solr.pl/en/2012/02/06/what-can-we-use-dismax-tie-parameter-for/.

Small summary

As you can see Solr provides a wide range of query parsers that can be included in our queries. With that we can make complicated queries, that will leverage different functionalities of Solr. However there is one thing to remember – the more complicated the query is, the more processing power it will require. Keep that in mind when constructing your queries.

Logic behind the parser

The logic of the SwitchQueryParser is quite simple – allow processing a simple logic on the Solr side and add it as a sub-query. For example let’s say that we have an application that understand the following four values of the priceRange field:

• cheap – when the price of the product (indexed in the price field) is lower than 10$,
• average – when the price is between 10 and 30$,
• expensive – when the product price is higher than  30$,
• all – in case we want to return all the documents without looking at the price.

We want to have this logic stored in Solr somehow in order not to change our application or its configuration every time we want to change the above ranges. For this purpose we will use the SwitchQueryParser.

Our query

Let’s assume that our application will be able to send the following query:

http://localhost:8983/solr/collection1/price?q=*:*&priceRange=cheap

We want the above query to return all the documents (q=*:*) narrowed to only those that are cheap, which in our case it will mean that they have price lower than 10$ (priceRange=cheap parameter).

Solr configuration

Of course we don’t want to send the price range in the query, because that wouldn’t make much sense for us. Because of that we decided to alter the solrconfig.xml file and add a new SearchHandler with the name of /price, which is configured as follows:

<requestHandler name="/price">
 <lst name="defaults">
  <str name="priceRange">all</str>
 </lst>
 <lst name="appends">
  <str name="fq">{!switch case.all='price:[* TO *]' case.cheap='price:[0 TO 10]' case.average='price:[10 TO 30]' case.expensive='price:[30 TO *]' v=$priceRange}</str>
 </lst>
</requestHandler>

As you can see the configuration of our SearchHandler consist of two elements. In the defaults section we defined the default value for the priceRange parameter, which is all. In addition to that, we’ve defined filter (fq) which is using the SwitchQueryParser (!switch). For each of the possible values the priceRange parameter can take (v=$priceRange) we defined a filter on the price field using the following expression – case.priceRangeValue=filter. So, when the value of the priceRange parameter in the query will be equal to cheap than Solr will use the filter defined by the case.cheap part of the filter definition, when the priceRange parameter value will be equal to expensive than Solr will use the filter defined by the case.expensive and so on.

What to remember about

There is one crucial thing to remember when using the described parser. In our case, if we would priceRange parameter value different than the four mentioned it will result in Solr error.

Summary

In my opinion the SwitchQueryParser is a nice addition, although it is rather a feature that will be used by the minority of the users. However taking into consideration that is can help implementing some very basic logic and because it is simple (and thus not hungry for resources) there will be users which will find this nice query parser useful