Looking back at the old post

In the SolrCloud – What happens when ZooKeeper fails? blog post, we’ve shown that Solr can handle querying without any issues when connection to ZooKeeper has been lost (which can be caused by different reasons). Of course this is true until we change the cluster topology. Unfortunately, in case of indexing or cluster change operations, we can’t change the cluster state or index documents when ZooKeeper connection is not working or ZooKeeper failed to read/write the data we want.

Why we can run queries?

The situation is quite simple – querying is not an operation that needs to alter SolrCloud cluster state. The only thing Solr needs to do is accept the query, run it against known shards/replicas and gather the results. Of course cluster topology is not retrieved with each query, so when there is no active ZooKeeper connection (or ZooKeeper failed) we don’t have a problem with running queries.

There is also one important and not widely know feature of SolrCloud – the ability to return partial results. By adding the shards.tolerant=true parameter to our queries we inform Solr, that we can live with partial results and it should ignore shards that are not available. This means that Solr will return results even if some of the shards from our collection is not available. By default, when this parameter is not present or set to false, Solr will just return error when running a query against collection that doesn’t have all the shards available.

Why we can’t index data?

So, we can’t we index data, when ZooKeeper connection is not available or when ZooKeeper doesn’t have a quorum? Because there is potentially not enough information about the cluster state to process the indexing operation. Solr just may not have the fresh information about all the shards, replicas, etc. Because of that, indexing operation may be pointed to incorrect shard (like not to the current leader), which can lead to data corruption. And because of that indexing (or cluster change) operation is jus not possible.

It is generally worth remembering, that all operations that can lead to cluster state update or collections update won’t be possible when ZooKeeper quorum is not visible by Solr (in our test case, it will be a lack of connectivity of a single ZooKeeper server).

Of course, we could leave you with what we wrote above, but let’s check if all that is true.

Running ZooKeeper

A very simple step. For the purpose of the test we will only need a single ZooKeeper instance which is run using the following command from ZooKeeper installation directory:

bin/zkServer.sh start

We should see the following information on the console:

JMX enabled by default
Using config: /Users/gro/Solry/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

And that means that we have a running ZooKeeper server.

Starting two Solr instances

To run the test we’ve used the newest available Solr version – the 5.2.1 when this blog post was published. To run two Solr instances we’ve used the following command:

bin/solr start -e cloud -z localhost:2181

Solr asked us a few questions when it was starting and the answers where the following:

• number of instances: 2
• collection name: gettingstarted
• number of shards: 2
• replication count: 1
• configuration name: data_driven_schema_configs

Cluster topology after Solr started was as follows:

Let’s index a few documents

To see that Solr is really running, we’ve indexed a few documents by running the following command:

bin/post -c gettingstarted docs/

If everything went well, after running the following command:

curl -XGET 'localhost:8983/solr/gettingstarted/select?indent=true&q=*:*&rows=0'

we should see Solr responding with similar XML:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">38</int>
  <lst name="params">
   <str name="q">*:*</str>
   <str name="indent">true</str>
   <str name="rows">0</str>
  </lst>
 </lst>
 <result name="response" numFound="3577" start="0" maxScore="1.0">
 </result>
</response>

We’ve indexed our documents, we have Solr running.

Let’s stop ZooKeeper and index data

To stop ZooKeeper server we will just run the following command in the ZooKeeper installation directory:

bin/zkServer.sh stop

And now, let’s again try to index our data:

bin/post -c gettingstarted docs/

This time, instead of data being written into the collection we will get an error response similar to the following one:

POSTing file index.html (text/html) to [base]/extract
SimplePostTool: WARNING: Solr returned an error #503 (Service Unavailable) for url: http://localhost:8983/solr/gettingstarted/update/extract?resource.name=%2FUsers%2Fgro%2FSolry%2F5.2.1%2Fdocs%2Findex.html&literal.id=%2FUsers%2Fgro%2FSolry%2F5.2.1%2Fdocs%2Findex.html
SimplePostTool: WARNING: Response: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">503</int><int name="QTime">3</int></lst><lst name="error"><str name="msg">Cannot talk to ZooKeeper - Updates are disabled.</str><int name="code">503</int></lst>
</response>

As we can see, the lack of ZooKeeper connectivity resulted in Solr not being able to index data. Of course querying still works. Turning on ZooKeeper again and retrying indexing will be successful, because Solr will automatically reconnect to ZooKeeper and will start working again.

Short summary

Of course this and the previous blog post related to ZooKeeper and SolrCloud are only touching the surface of what is happening when ZooKeeper connection is not available. A very good test that shows us data consistency related information can be found at http://lucidworks.com/blog/call-maybe-solrcloud-jepsen-flaky-networks/. I really recommend it if you would like to know what will happen with SolrCloud in various emergency situations.

Test environment

The test environment was very simple:

• A single virtual machine running under Linux operating system
• A single instance of ZooKeeper (which will be suitable for our test)
• Two Solr instances with a single collection deployed
• Solr 4.6

In order to create our test collection I’ve uploaded the configuration to ZooKeeper and used the following command:

curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=collection1&numShards=2&replicationFactor=1'

The cloud view of the example cluster was as follows:

Test data indexing

The next step in our test will be indexing. We will index a few example documents that are provided with Solr in the exampledocs directory. The following commands were used to index the data:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @mem.xml -H 'Content-type:application/xml'
curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @monitor.xml -H 'Content-type:application/xml'
curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @monitor2.xml -H 'Content-type:application/xml'

After executing the above commands we get the following number of documents:

• The whole collection holds 5 documents
• Shard located on Solr running on port 8983 host 1 document
• Shard located on Solr running on port 7983 has 4 documents

Querying with ZooKeeper not present

Now we go to the next step – we shutdown our ZooKeeper instance and we try to run a simple query by sending the following command:

curl 'localhost:8983/solr/collection1/select?q=*:*&indent=true'

In result we get the following response:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">16</int>
  <lst name="params">
   <str name="indent">true</str>
   <str name="q">*:*</str>
  </lst>
 </lst>
<result name="response" numFound="5" start="0" maxScore="1.0">
<doc>
 <str name="id">TWINX2048-3200PRO</str> 
 <str name="name">CORSAIR  XMS 2GB (2 x 1GB) 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) Dual Channel Kit System Memory - Retail</str>
 <str name="manu">Corsair Microsystems Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <arr name="features">
  <str>CAS latency 2,    2-3-3-6 timing, 2.75v, unbuffered, heat-spreader</str>
 </arr>
 <float name="price">185.0</float>
 <str name="price_c">185,USD</str>
 <int name="popularity">5</int>
 <bool name="inStock">true</bool>
 <str name="store">37.7752,-122.4232</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|6.0 memory|3.0</str>
 <long name="_version_">1453219034197655552</long>
</doc>
<doc>
 <str name="id">VS1GB400C3</str>
 <str name="name">CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail</str>
 <str name="manu">Corsair Microsystems Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <float name="price">74.99</float>
 <str name="price_c">74.99,USD</str>
 <int name="popularity">7</int>
 <bool name="inStock">true</bool>
 <str name="store">37.7752,-100.0232</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|4.0 memory|2.0</str>
 <long name="_version_">1453219034252181504</long>
</doc>
<doc>
 <str name="id">VDBDB1A16</str>
 <str name="name">A-DATA V-Series 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - OEM</str>
 <str name="manu">A-DATA Technology Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <arr name="features">
  <str>CAS latency 3,     2.7v</str>
 </arr>
 <int name="popularity">0</int>
 <bool name="inStock">true</bool>
 <str name="store">45.18414,-93.88141</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|0.9 memory|0.1</str>
 <long name="_version_">1453219034255327232</long>
</doc>
<doc>
 <str name="id">3007WFP</str>
 <str name="name">Dell Widescreen UltraSharp 3007WFP</str>
 <str name="manu">Dell, Inc.</str>
 <str name="manu_id_s">dell</str>
 <arr name="cat">
  <str>electronics</str>
  <str>monitor</str>
 </arr>
 <arr name="features">
  <str>30" TFT active matrix LCD, 2560 x 1600, .25mm dot pitch, 700:1 contrast</str>
 </arr>
 <str name="includes">USB cable</str>
 <float name="weight">401.6</float>
 <float name="price">2199.0</float>
 <str name="price_c">2199,USD</str>
 <int name="popularity">6</int>
 <bool name="inStock">true</bool>
 <str name="store">43.17614,-90.57341</str>
 <long name="_version_">1453219041357332480</long>
</doc>
<doc>
 <str name="id">VA902B</str>
 <str name="name">ViewSonic VA902B - flat panel display - TFT - 19"</str>
 <str name="manu">ViewSonic Corp.</str>
 <str name="manu_id_s">viewsonic</str>
 <arr name="cat">
  <str>electronics</str>
  <str>monitor</str>
 </arr>
 <arr name="features">
  <str>19" TFT active matrix LCD, 8ms response time, 1280 x 1024 native resolution</str>
 </arr>
 <float name="weight">190.4</float>
 <float name="price">279.95</float>
 <str name="price_c">279.95,USD</str>
 <int name="popularity">6</int>
 <bool name="inStock">true</bool>
 <str name="store">45.18814,-93.88541</str>
 <long name="_version_">1453219045997281280</long></doc>
</result>
</response>

As we can see Solr responded correctly. This is because Solr already has the clusterstate.json file cached. To search Solr doesn’t need to update that file, so search should and is working as we could see.

Indexing with failed ZooKeeper

Without turning on our ZooKeeper instance we try to run the following command:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @hd.xml -H 'Content-type:application/xml'

The above command should result in indexing the contents of the hd.xml file. After a longer period of time Solr responds with the following information:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">503</int><int name="QTime">15096</int></lst><lst name="error"><str name="msg">Cannot talk to ZooKeeper - Updates are disabled.</str><int name="code">503</int></lst>
</response>

So as you can see we are not able to index data without working ZooKeeper ensemble.

Starting ZooKeeper again

So let’s see what will happen when we start our ZooKeeper instance again without restarting Solr nodes. After starting ZooKeeper we try to run the same indexing command, we just did, once again:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @hd.xml -H 'Content-type:application/xml'

And this time the response is different:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">118</int></lst>
</response>

As we can see the indexing request was successful this time. This allows us to assume that the connection to ZooKeeper was re-established by Solr. We can see that in Solr and ZooKeeper logs.

Short summary

As you can see, our short test allowed to see what happens when our ZooKeeper ensemble fails and what we can expect from Solr in such rare cases. I hope this blog entry will help you with some doubts about SolrCloud and its usefulnesses.

Please also remember that during the test, the cluster state did not change – all shards were accessible and working. We will see what will be happening when shards or replicas fails when ZooKeeeper is down in the next blog entry about SolrCloud.

Test environment

The test environment was very simple:

• A single virtual machine running under Linux operating system
• A single instance of ZooKeeper (which will be suitable for our test)
• Two Solr instances with a single collection deployed
• Solr 4.6

In order to create our test collection I’ve uploaded the configuration to ZooKeeper and used the following command:

curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=collection1&numShards=2&replicationFactor=1'

The cloud view of the example cluster was as follows:

Test data indexing

The next step in our test will be indexing. We will index a few example documents that are provided with Solr in the exampledocs directory. The following commands were used to index the data:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @mem.xml -H 'Content-type:application/xml'
curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @monitor.xml -H 'Content-type:application/xml'
curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @monitor2.xml -H 'Content-type:application/xml'

After executing the above commands we get the following number of documents:

• The whole collection holds 5 documents
• Shard located on Solr running on port 8983 host 1 document
• Shard located on Solr running on port 7983 has 4 documents

Querying with ZooKeeper not present

Now we go to the next step – we shutdown our ZooKeeper instance and we try to run a simple query by sending the following command:

curl 'localhost:8983/solr/collection1/select?q=*:*&indent=true'

In result we get the following response:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">16</int>
  <lst name="params">
   <str name="indent">true</str>
   <str name="q">*:*</str>
  </lst>
 </lst>
<result name="response" numFound="5" start="0" maxScore="1.0">
<doc>
 <str name="id">TWINX2048-3200PRO</str> 
 <str name="name">CORSAIR  XMS 2GB (2 x 1GB) 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) Dual Channel Kit System Memory - Retail</str>
 <str name="manu">Corsair Microsystems Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <arr name="features">
  <str>CAS latency 2,    2-3-3-6 timing, 2.75v, unbuffered, heat-spreader</str>
 </arr>
 <float name="price">185.0</float>
 <str name="price_c">185,USD</str>
 <int name="popularity">5</int>
 <bool name="inStock">true</bool>
 <str name="store">37.7752,-122.4232</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|6.0 memory|3.0</str>
 <long name="_version_">1453219034197655552</long>
</doc>
<doc>
 <str name="id">VS1GB400C3</str>
 <str name="name">CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail</str>
 <str name="manu">Corsair Microsystems Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <float name="price">74.99</float>
 <str name="price_c">74.99,USD</str>
 <int name="popularity">7</int>
 <bool name="inStock">true</bool>
 <str name="store">37.7752,-100.0232</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|4.0 memory|2.0</str>
 <long name="_version_">1453219034252181504</long>
</doc>
<doc>
 <str name="id">VDBDB1A16</str>
 <str name="name">A-DATA V-Series 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - OEM</str>
 <str name="manu">A-DATA Technology Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <arr name="features">
  <str>CAS latency 3,     2.7v</str>
 </arr>
 <int name="popularity">0</int>
 <bool name="inStock">true</bool>
 <str name="store">45.18414,-93.88141</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|0.9 memory|0.1</str>
 <long name="_version_">1453219034255327232</long>
</doc>
<doc>
 <str name="id">3007WFP</str>
 <str name="name">Dell Widescreen UltraSharp 3007WFP</str>
 <str name="manu">Dell, Inc.</str>
 <str name="manu_id_s">dell</str>
 <arr name="cat">
  <str>electronics</str>
  <str>monitor</str>
 </arr>
 <arr name="features">
  <str>30" TFT active matrix LCD, 2560 x 1600, .25mm dot pitch, 700:1 contrast</str>
 </arr>
 <str name="includes">USB cable</str>
 <float name="weight">401.6</float>
 <float name="price">2199.0</float>
 <str name="price_c">2199,USD</str>
 <int name="popularity">6</int>
 <bool name="inStock">true</bool>
 <str name="store">43.17614,-90.57341</str>
 <long name="_version_">1453219041357332480</long>
</doc>
<doc>
 <str name="id">VA902B</str>
 <str name="name">ViewSonic VA902B - flat panel display - TFT - 19"</str>
 <str name="manu">ViewSonic Corp.</str>
 <str name="manu_id_s">viewsonic</str>
 <arr name="cat">
  <str>electronics</str>
  <str>monitor</str>
 </arr>
 <arr name="features">
  <str>19" TFT active matrix LCD, 8ms response time, 1280 x 1024 native resolution</str>
 </arr>
 <float name="weight">190.4</float>
 <float name="price">279.95</float>
 <str name="price_c">279.95,USD</str>
 <int name="popularity">6</int>
 <bool name="inStock">true</bool>
 <str name="store">45.18814,-93.88541</str>
 <long name="_version_">1453219045997281280</long></doc>
</result>
</response>

As we can see Solr responded correctly. This is because Solr already has the clusterstate.json file cached. To search Solr doesn’t need to update that file, so search should and is working as we could see.

Indexing with failed ZooKeeper

Without turning on our ZooKeeper instance we try to run the following command:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @hd.xml -H 'Content-type:application/xml'

The above command should result in indexing the contents of the hd.xml file. After a longer period of time Solr responds with the following information:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">503</int><int name="QTime">15096</int></lst><lst name="error"><str name="msg">Cannot talk to ZooKeeper - Updates are disabled.</str><int name="code">503</int></lst>
</response>

So as you can see we are not able to index data without working ZooKeeper ensemble.

Starting ZooKeeper again

So let’s see what will happen when we start our ZooKeeper instance again without restarting Solr nodes. After starting ZooKeeper we try to run the same indexing command, we just did, once again:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @hd.xml -H 'Content-type:application/xml'

And this time the response is different:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">118</int></lst>
</response>

As we can see the indexing request was successful this time. This allows us to assume that the connection to ZooKeeper was re-established by Solr. We can see that in Solr and ZooKeeper logs.

Short summary

As you can see, our short test allowed to see what happens when our ZooKeeper ensemble fails and what we can expect from Solr in such rare cases. I hope this blog entry will help you with some doubts about SolrCloud and its usefulnesses.

Please also remember that during the test, the cluster state did not change – all shards were accessible and working. We will see what will be happening when shards or replicas fails when ZooKeeeper is down in the next blog entry about SolrCloud.

We will try to write something about these new functionality as soon as possible