To show you how replication throttling works we will compare two use cases – one will copy 2GB index without any limits and the other will copy the same index, but with throttling enabled. To illustrate that we will use SolrCloud deployment, with very simple configuration. Please note, that replication throttling is not limited to SolrCloud deployments and can be used successfully in old master – slave architecture.

Replication without throttling

To show you how network bandwidth has been used we will use the following, standard replication handler configuration:

<requestHandler name="/replication" class="solr.ReplicationHandler">
</requestHandler>

Network usage in that case looked as follows:

Replication with throttling enabled

Now let’s compare it to replication that is setup to use throttling:

<requestHandler name="/replication" class="solr.ReplicationHandler">
 <lst name="defaults">
  <str name="maxWriteMBPerSec">0.1</str>
 </lst>
</requestHandler>

Network usage in this case looks as follows:

As we can see the difference is large and throttling works as intended

Short summary

As we can see the configuration itself is very simple and it just works. Again, it does work in master – slave and in SolrCloud deployments, so no matter which way you go, you can use replication throttling. The only thing that is missing now, at least for me, is dedicated API that I could use to change the replication configuration without core reload. Maybe in the future?

The Beginning

We decided to write about index backups even though this functionality is fairly simple. We noticed that many people tend to forget about this functionality, not only when it comes to Apache Solr. We hope that this blog entry, will help you remember about backup creation functionality, when you need it. But now, lets start from the beginning – before we started the tests, we looked at the directory where Solr keeps its indices and this is what we saw:

drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:17 index
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:16 spellchecker

Manual Backup

In order to create a backup of your index with the use of HTTP API you have to have replication handler configured. If you have it, then you need to send the command parameter with backup value to the master server replication handler, for example like this:

curl 'http://localhost:8983/solr/replication?command=backup'

The above will tell Solr to create a new backup of the current index. Lets now look how the directory where indices live looks like after running the above command:

drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:18 index
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:19 snapshot.20120812201917
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:16 spellchecker

As you can see, there is a new directory created – snapshot.20120812201917. We can assume, that we got what we wanted

Automatic Backup

In addition to manual backup creation, you can also configure Solr to create indices after commit or optimize operation. Please remember though, that if your index is changing rapidly it is usually a bad idea to create backup after each commit operation.  But lets get back to automatic backups. In order to configure Solr to create backups for us, you need to add the following line to replication handler configuration:

<str name="backupAfter">commit</str>

So, the full replication handler configuration (on the master server) would look like this:

<requestHandler name="/replication" >
 <lst name="master">
  <str name="replicateAfter">commit</str>
  <str name="replicateAfter">startup</str>
  <str name="confFiles">schema.xml,stopwords.txt</str>
  <str name="backupAfter">commit</str>
 </lst>
</requestHandler>

After sending two commit operation our dictionary with indices looks like this:

drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 21:12 index
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 21:12 snapshot.20120812211203
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 21:12 snapshot.20120812211216
drwxrwxr-x 2 gr0 gr0 4096 2012-08-12 20:16 spellchecker

As you can see, Solr did what we wanted to be done.

Keeping Order

It is possible to control the maximum amount of backups that should be stored on disk. In order to configure that number you need to add the following line to your replication handler configuration:

<str name="maxNumberOfBackups">10</str>

The above configuration value tells Solr to keep maximum of ten backups of your index. Of course you can delete created backups (manually for example) if you don’t need them anymore.