Running a query

I’m sure you are familiar on how to run queries to Solr in the standard, old fashioned way, right? So, to get documents that match the solr term in the _text_ field we would just run a command like this:

curl -XGET 'localhost:8983/solr/gettingstarted/select?q=_text_:solr&indent=true'

Just for the record, I’m using the schemaless example provided with Solr and I indexed the data from the docs directory.

We can do the same using the JSON API of Solr. For example, a JSON structured query that would return exactly the same results as the one shown above would look as follows:

curl -XGET 'localhost:8983/solr/gettingstarted/query' -d '{
 "query":"_text_:solr"
}'

As you can see, we just specify the query using a JSON object and the query property in it and we send it as a request body to the /query handler. We could also send the query to the /select handler, but the /query one uses JSON response by default, so it seems more appropriate for JSON queries

We can also use the standard parameter names and pass them in the body of the request, just like this:

curl -XGET 'localhost:8983/solr/gettingstarted/query' -d 'q=_text_:solr'

So, this time we didn’t have the JSON structure in the request body, we just pass the parameters there and separate them with & character, just like when sending a query using URI request.

Paging

Of course, we can also page through the results when using JSON API. We do that by using the limit and and offset parameters. The limit parameter specifies the maximum number of returned documents and the offset parameter is responsible for telling Solr from which document the results should be returned. For example, to get 20 documents starting from the 11th one, we would run the following query:

curl -XGET 'localhost:8983/solr/gettingstarted/query' -d '{
 "query":"_text_:solr",
 "limit":20,
 "offset":10
}'

Sorting

The next thing is the ability to sort the results and we can do that using Solr JSON API as well. To sort our results on the basis of the id field, in the descending order we would run the following query:

curl -XGET 'localhost:8983/solr/gettingstarted/query' -d '{
 "query":"_text_:solr",
 "sort":"id desc"
}'

It is very similar to what we are used to when using the standard URI request queries, but we are using the sort property.

Filtering

Finally, we have filtering. We can filter our data by using the filter property. For example, to narrow down our data to only the ones that have text/html value in the content_type field we would run the following query:

curl -XGET 'localhost:8983/solr/gettingstarted/query' -d '{
 "query":"_text_:solr",
 "filter":"content_type:text/html"
}'

What’s next

As you can see the JSON API of Solr is very easy to use and allows us to use more structured queries when talking to Solr. In the next entry we will discuss the JSON facets API, how to use it and what are other differences when it comes to JSON facets and the traditional implementation, because making the request is not the only difference when we compare those two.

Let’s start

The new handler (JsonUpdateRequestHandler) allows us to transfer data in the JSON format which in theory should translate into a smaller amount of data sent over the network and the speedup of indexing, as the JSON parser is theoretically faster than XML parsers. But let’s leave the performance for now.

Configuration

Let’s start by defining a handler. To do that add the following definition to the solrconfig.xml file (if you use the default solrconfig.xml file provided with Solr 3.1 than this handler is already defined):

<requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler" startup="lazy" />

The entry above defines a new handler that will be initialized when used for the first time (startup=”lazy”).

Indexing

The next step is to prepare the data – of course in JSON format. Here’s an example showing two documents in one file called data.json:

{

"add": {
  "doc": {
    "id" : "123456788",
    "region" : ["abc","def"],
    "name" : "ABCDEF",
  }
}

,
"add": {
  "doc": {
    "id" : "123456789",
    "region" : ["abc","def"],
    "name" : "XYZMN",
  }
}

}

Such prepared file can be sent to the /update/json address and thus be indexed. Remember to send a commit command to the appropriate address (standard /update) in order to tell Solr to open a new index searcher.

Performance

At the end I left myself what I’m really most interested in – the performance of the new handler. According to the information stored in JIRA system we can be expect that JsonUpdateRequestHandler will be faster than its counterpart processor of XML format. To examine this, I prepared the files of 10.000, 100.000 and 1 million documents. Every document contained an identifier (string field), two regions (String field, multivalued) and the name (text field). One file was saved in the JSON format, the second one was saved in XML format, the third one was saved in CSV format. All files were then indexed separately. Here is an outcome of this simple test:

The conclusions suggest themselves. First, XML data is relatively larger than the one written in JSON format (the difference is about 35%). However, a file stored in JSON format, is larger (which might be expected) than the one written in the CSV. If you send data not on the local network, the size is relevant – the difference in file size is significant enough that it is worth thinking about changing the XML to any of the formats that require less space.

Indexation time

Another thing is the indexing time. Leaning on the results of this simple test we can think that JsonUpdateRequestHandler is slightly (about 7 – 9%) faster than the XmlUpdateRequestHandler. As you can see, the difference is similar for JsonUpdateRequestHandler and CSVRequestHandler, where the handler operates on files in CSV format is faster than its counterpart that operates in JSON format by about 7 to 9%. Let’s hope that when the noggit library comes out of Apache Labs, its performance will be even greater, and thus we will see even faster JsonUpdateRequestHandler.