The Beginning

We decided to write about index backups even though this functionality is fairly simple. We noticed that many people tend to forget about this functionality, not only when it comes to Apache Solr. We hope that this blog entry, will help you remember about backup creation functionality, when you need it. But now, lets start from the beginning – before we started the tests, we looked at the directory where Solr keeps its indices and this is what we saw:

drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:17 index
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:16 spellchecker

Manual Backup

In order to create a backup of your index with the use of HTTP API you have to have replication handler configured. If you have it, then you need to send the command parameter with backup value to the master server replication handler, for example like this:

curl 'http://localhost:8983/solr/replication?command=backup'

The above will tell Solr to create a new backup of the current index. Lets now look how the directory where indices live looks like after running the above command:

drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:18 index
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:19 snapshot.20120812201917
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:16 spellchecker

As you can see, there is a new directory created – snapshot.20120812201917. We can assume, that we got what we wanted

Automatic Backup

In addition to manual backup creation, you can also configure Solr to create indices after commit or optimize operation. Please remember though, that if your index is changing rapidly it is usually a bad idea to create backup after each commit operation.  But lets get back to automatic backups. In order to configure Solr to create backups for us, you need to add the following line to replication handler configuration:

<str name="backupAfter">commit</str>

So, the full replication handler configuration (on the master server) would look like this:

<requestHandler name="/replication" >
 <lst name="master">
  <str name="replicateAfter">commit</str>
  <str name="replicateAfter">startup</str>
  <str name="confFiles">schema.xml,stopwords.txt</str>
  <str name="backupAfter">commit</str>
 </lst>
</requestHandler>

After sending two commit operation our dictionary with indices looks like this:

drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 21:12 index
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 21:12 snapshot.20120812211203
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 21:12 snapshot.20120812211216
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:16 spellchecker

As you can see, Solr did what we wanted to be done.

Keeping Order

It is possible to control the maximum amount of backups that should be stored on disk. In order to configure that number you need to add the following line to your replication handler configuration:

<str name="maxNumberOfBackups">10</str>

The above configuration value tells Solr to keep maximum of ten backups of your index. Of course you can delete created backups (manually for example) if you don’t need them anymore.

Let’s start

The new handler (JsonUpdateRequestHandler) allows us to transfer data in the JSON format which in theory should translate into a smaller amount of data sent over the network and the speedup of indexing, as the JSON parser is theoretically faster than XML parsers. But let’s leave the performance for now.

Configuration

Let’s start by defining a handler. To do that add the following definition to the solrconfig.xml file (if you use the default solrconfig.xml file provided with Solr 3.1 than this handler is already defined):

<requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler" startup="lazy" />

The entry above defines a new handler that will be initialized when used for the first time (startup=”lazy”).

Indexing

The next step is to prepare the data – of course in JSON format. Here’s an example showing two documents in one file called data.json:

{

"add": {
  "doc": {
    "id" : "123456788",
    "region" : ["abc","def"],
    "name" : "ABCDEF",
  }
}

,
"add": {
  "doc": {
    "id" : "123456789",
    "region" : ["abc","def"],
    "name" : "XYZMN",
  }
}

}

Such prepared file can be sent to the /update/json address and thus be indexed. Remember to send a commit command to the appropriate address (standard /update) in order to tell Solr to open a new index searcher.

Performance

At the end I left myself what I’m really most interested in – the performance of the new handler. According to the information stored in JIRA system we can be expect that JsonUpdateRequestHandler will be faster than its counterpart processor of XML format. To examine this, I prepared the files of 10.000, 100.000 and 1 million documents. Every document contained an identifier (string field), two regions (String field, multivalued) and the name (text field). One file was saved in the JSON format, the second one was saved in XML format, the third one was saved in CSV format. All files were then indexed separately. Here is an outcome of this simple test:

The conclusions suggest themselves. First, XML data is relatively larger than the one written in JSON format (the difference is about 35%). However, a file stored in JSON format, is larger (which might be expected) than the one written in the CSV. If you send data not on the local network, the size is relevant – the difference in file size is significant enough that it is worth thinking about changing the XML to any of the formats that require less space.

Indexation time

Another thing is the indexing time. Leaning on the results of this simple test we can think that JsonUpdateRequestHandler is slightly (about 7 – 9%) faster than the XmlUpdateRequestHandler. As you can see, the difference is similar for JsonUpdateRequestHandler and CSVRequestHandler, where the handler operates on files in CSV format is faster than its counterpart that operates in JSON format by about 7 to 9%. Let’s hope that when the noggit library comes out of Apache Labs, its performance will be even greater, and thus we will see even faster JsonUpdateRequestHandler.