The problem

The deep paging problem is quite easy to define. To return search results Solr must prepare an in-memory structure and return part of it. Returning the part of the structure is simple, if that part comes from the beginning of the structure. However, if we want to return page number 10.000 (where we return 20 results per page) Solr needs to prepare a structure containing minimum of 200.000 elements (10.000 * 20). You see that it not only takes time, but also memory.

The good thing is, that with the release of Solr 4.7 the situation had changed – the cursor has been introduced. Cursor is a logic structure, that doesn’t require its state to be stored on the server side. Cursor contains information about storing and lest document returned in the results. Because of that, Solr doesn’t need to start search from beginning each time we want to get next page of results. It results in drastic performance improvement when using cursor and going deep into results.

Usage

Cursor usage is very simple. To tell Solr to return cursor, in the first query we need to pass an additional parameter – cursorMark=*. In result, apart from documents, we will get a cursor identifier returned in the nextCursorMark parameter. Let’s look at the example.

The query

Let’s start with a very simple query:

curl 'localhost:8983/solr/select?q=*:*&rows=1&sort=score+desc,id+asc&cursorMark=*'

There are four things here that we are interested in. First of off, we either omit the start parameter or we set it to 0. The rows parameter can take values we need, there is no limitation on it. Of course, we passed the cursorMark=* parameter, to tell Solr that we want the cursor to be used. The final thing we did is sorting definition. We need to define sorting for cursor to be working, one that will tell cursor how to behave. That’s why we needed to overwrite default sorting and include sorting not only by score, by also by document identifier.

Search results

Our query returns the following search results:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">33</int>
  <lst name="params">
   <str name="sort">score desc,id asc</str>
   <str name="start">0</str>
   <str name="q">*:*</str>
   <str name="cursorMark">*</str>
   <str name="rows">1</str>
  </lst>
 </lst>
<result name="response" numFound="32" start="0">
 <doc>
  <str name="id">0579B002</str>
  <str name="name">Canon PIXMA MP500 All-In-One Photo Printer</str>
  <str name="manu">Canon Inc.</str>
  <str name="manu_id_s">canon</str>
  <arr name="cat">
   <str>electronics</str>
   <str>multifunction printer</str>
   <str>printer</str>
   <str>scanner</str>
   <str>copier</str>
  </arr>
  <arr name="features">
   <str>Multifunction ink-jet color photo printer</str>
   <str>Flatbed scanner, optical scan resolution of 1,200 x 2,400 dpi</str>
   <str>2.5" color LCD preview screen</str>
   <str>Duplex Copying</str>
   <str>Printing speed up to 29ppm black, 19ppm color</str>
   <str>Hi-Speed USB</str>
   <str>memory card: CompactFlash, Micro Drive, SmartMedia, Memory Stick, Memory Stick Pro, SD Card, and MultiMediaCard</str>
  </arr>
  <float name="weight">352.0</float>
  <float name="price">179.99</float>
  <str name="price_c">179.99,USD</str>
  <int name="popularity">6</int>
  <bool name="inStock">true</bool>
  <str name="store">45.19214,-93.89941</str>
  <long name="_version_">1461375031699308544</long></doc>
 </result>
 <str name="nextCursorMark">AoIIP4AAACgwNTc5QjAwMg==</str>
</response>

As we can see, in addition to standard search results, we got the cursor identifier in the nextCursorMark section. Now, to get the next results bound to that cursor, we need to pass that identifier using the cursorMark parameter.

Next query

Our next query looks as follows (note the cursorMark parameter value):

curl 'localhost:8983/solr/select?q=*:*&rows=1&sort=score+desc,id+asc&cursorMark=AoIIP4AAACgwNTc5QjAwMg=='

The results were as follows:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">2</int>
  <lst name="params">
   <str name="sort">score desc,id asc</str>
   <str name="indent">true</str>
   <str name="q">*:*</str>
   <str name="cursorMark">AoIIP4AAACgwNTc5QjAwMg==</str>
   <str name="rows">1</str>
  </lst>
 </lst>
<result name="response" numFound="32" start="0">
 <doc>
  <str name="id">100-435805</str>
  <str name="name">ATI Radeon X1900 XTX 512 MB PCIE Video Card</str>
  <str name="manu">ATI Technologies</str>
  <str name="manu_id_s">ati</str>
  <arr name="cat">
   <str>electronics</str>
   <str>graphics card</str>
  </arr>
  <arr name="features">
   <str>ATI RADEON X1900 GPU/VPU clocked at 650MHz</str>
   <str>512MB GDDR3 SDRAM clocked at 1.55GHz</str>
   <str>PCI Express x16</str>
   <str>dual DVI, HDTV, svideo, composite out</str>
   <str>OpenGL 2.0, DirectX 9.0</str>
  </arr>
  <float name="weight">48.0</float>
  <float name="price">649.99</float>
  <str name="price_c">649.99,USD</str>
  <int name="popularity">7</int>
  <bool name="inStock">false</bool>
  <date name="manufacturedate_dt">2006-02-13T00:00:00Z</date>
  <str name="store">40.7143,-74.006</str>
  <long name="_version_">1461375031846109184</long></doc>
 </result>
 <str name="nextCursorMark">AoIIP4AAACoxMDAtNDM1ODA1</str>
</response>

As we can see, the returned nextCursorMark was different again.

Further queries

Logic for further queries is simple – we use the cursorMark parameter with the value returned with the previous search results. So again, our next query would look as follows:

curl 'localhost:8983/solr/select?q=*:*&rows=1&sort=score+desc,id+asc&cursorMark=AoIIP4AAACoxMDAtNDM1ODA1'

Summary

Simple API and massive gain on performance in case of deep paging. That’s how I think the cursor introduced in Solr 4.7 could be summarized. I decided not do re-do performance tests, there are ones already done by Chris Hostetter in his entry about this functionality. If you are interested please look at: http://searchhub.org/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/.

A few numbers at the beginning

Let’s start with the query and statistics. Imagine that we have the following query, which is sent to Solr to get the five hundredth page of search results:

q=*:*&sort=price+asc&rows=100&start=50000

What Solr must do Solr to retrieve and render the results list ? Of course, read the documents from Lucene index. There is of course the question of how many documents to be read from the index? Is it 100? Unfortunately, no. Solr must collect 50.100 sorted documents from the Lucene index, due to the fact that we want 100 documents starting from 50.000-th. Kinda scary. Now let’s look at the comparison of how long it takes for Solr download the first page of search results (the query q=*:*&sort=price+asc&rows=100&start=0) and how long it takes to render the last page of search results (ie the query q=*:*&sort = price+asc&rows=100&start=999900). The test was performed on an index containing a million documents, consisting of four fields: id (string), name (text), description (text), price (long). Before every test iteration Solr was started the query was run and Solr was truned off. These steps were repeated 100 times for each of the queries, and times is seen in the table are the arithmetic mean of the query time execution. Here are the test results:

Typical solutions

Of course we can try to set the cache or the size of queryResultWindowSize, but there will be a problem of how to set the size, there may be a situation where it will be insufficient or not relevant entry in the memory of Solr and then waiting time for the n-th search page will be very long. We can also try adding warming queries, but we won’t be able to prepare all the combinations, but even if we could the the cache would have to be big. So we won’t be able to achieve the desired results with any of these solutions.

Filters, filters and filter again

This behavior Solr (and other applications based on Lucene too) is caused by the queue of documents, called. priority queue, which in order to display the last page of results must download all documents matching the query and return the ones we want located in the desired page. Of course, in our case, if we want the first page of search results queue will have 100 entries. However, if we want the last page will have Solr search in the queue to put one million documents. Of course what I told is in big simplification.

The idea is to limit the number of documents Lucene must put in the queue. How to do it ? We will use filters to help us, so in Solr we will use the fq parameter. Using a filter will limit the number of search results. The ideal size of the queue would be the one that is passed in the rows parameter of query. However, this situation is ideal and not very possible in most situations. An additional problem is that asking a query with a filter we can not determine the number of results, because we do not know how much documents will the filter return. The solution to this problem is the making two queries instead of just one – the first one to see how fimiting is our filter thus using rows=0 and start=0, and the second is already adequately calculated (example below).

The maximum price of the product in the test data is 10,000, and the minimum is 0. So to the first query we will add the following bracket: <0; 1000>, and to the second query, we add the following bracket: <9000; 10000>.

Disadvantages of solution based on filters

There is one minus the filter-based solution and it is quite large. It may happen that the number of results to which the filter is limiting the query is too small. What then? We should increase the choosen bracket for the filter. Of course we can calculate the optimal brackets on the basis of our data, but it depends on the data and queries and why I won’t be talking about this at this point.

What is the performance after the change?

So let’s repeat the tests, but now let’s implement the filter based approach. So the first will just return the first page of results (the query q=*:*&sort=price+asc&rows=100&start=0&fq=price:[0+TO+1000]). The second query (actually two queries) will will be used the check the number of results and then fetch those results (those two queries: q=*:*&sort=price+asc&rows=0&start=0&fq=price:[9000+TO+10,000] and q=*:*&sort=price+asc&rows=100&start=100037&fq=price:[9000+TO+10000]). It is worth noting about the changed start parameter in the query, due to the fact that we get fewer search results (this is caused by the fq parameter). This test was was carried out in similar way to the previous one – start Solr, run a query (or queries), and shutdown Solr. The number seen in the table are the arithmetic mean of query time execution.

As you can see, the query performance changed. We can therefore conclude that we succeeded. Of course, we could be tempted by further optimizations, but for now let’s say that we are satisfied with the results. I suspect however that you can ask the following question:

How is this handled by other search engines ?

Good question, but the answer is trivial in total – one of the methods is to prevent a very deep paging. This method shall include Google. Try to type the word “ask” for the search and go further than 91 page of search results. Google didn’t let me

Conclusions

As you can see deep paging performance after our changes has increased significantly. We can now allow the user to search results without worrying that it will kill our Solr and the machine on which he works. Of course, this method is not without flaws, due to the fact that we need to know certain parameters (eg in our case, the maximum price for descending sort), but this is a solution that allows you to provide search results in a relatively low latency time when compared to the pre-optimization method.