Some of the changes introduced in Solr 9.8:

• Solr cross data center feature graduated into a main Solr feature!
• A give request may now be limited when it comes to the amount of memory it can use using the memAllowed parameter.
• The lib tags in solrconfig.xml are now silently ignored unless you include the SOLR_CONFIG_LIB_ENABLED environment variable set to true.
• A new parser called knn_text_to_vector was added allowing to calculate text embeddings using external LLMs.

We encourage you to read the whole list of changes at: https://solr.apache.org/docs/9_8_0/changes/Changes.html.

Apache Solr 9.8 can be downloaded from https://dlcdn.apache.org/solr/.

Similar to the previous edition of cookbook, we took the time to rebuild the book and all recipes were updated, half of the previous content has been thrown away and new content was added. The very important thing in our minds is that Solr Cookbook Third Edition covers Solr 4.x version (basing on the newest 4.10.3 version of Solr) and Solr 5.0 which should be released very soon.

The book is targeting beginners and intermediate users working with Apache Solr. You’ll find recipes that should make your life easier when you take the first steps with Solr and when you are encountering common problems that intermediate users tend to struggle with. However I don’t recommend the book for those of you who knows everything about Solr – you may find parts of the book interesting, but this book is not directed to you.

The list of chapters from the book is as follows:

1. Apache Solr Configuration
2. Indexing Your Data
3. Analyzing Your Text Data
4. Querying Solr
5. Faceting
6. Improving Solr Performance
7. In the Cloud
8. Using Additional Solr Functionalities
9. Dealing with Problems
10. Real-life Situations

More information about the book itself with a free chapter (which will be available after official publication of the book) can be found on Packt Publishing web page dedicated to the book – https://www.packtpub.com/big-data-and-business-intelligence/solr-cookbook-third-edition.

All the information about the competition can be found at http://elasticsearchserverbook.com/win-elasticsearch-server-second-editon-ebook/.

Test environment

The test environment was very simple:

• A single virtual machine running under Linux operating system
• A single instance of ZooKeeper (which will be suitable for our test)
• Two Solr instances with a single collection deployed
• Solr 4.6

In order to create our test collection I’ve uploaded the configuration to ZooKeeper and used the following command:

curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=collection1&numShards=2&replicationFactor=1'

The cloud view of the example cluster was as follows:

Test data indexing

The next step in our test will be indexing. We will index a few example documents that are provided with Solr in the exampledocs directory. The following commands were used to index the data:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @mem.xml -H 'Content-type:application/xml'
curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @monitor.xml -H 'Content-type:application/xml'
curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @monitor2.xml -H 'Content-type:application/xml'

After executing the above commands we get the following number of documents:

• The whole collection holds 5 documents
• Shard located on Solr running on port 8983 host 1 document
• Shard located on Solr running on port 7983 has 4 documents

Querying with ZooKeeper not present

Now we go to the next step – we shutdown our ZooKeeper instance and we try to run a simple query by sending the following command:

curl 'localhost:8983/solr/collection1/select?q=*:*&indent=true'

In result we get the following response:

<?xml version="1.0" encoding="UTF-8"?>
<response>
 <lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">16</int>
  <lst name="params">
   <str name="indent">true</str>
   <str name="q">*:*</str>
  </lst>
 </lst>
<result name="response" numFound="5" start="0" maxScore="1.0">
<doc>
 <str name="id">TWINX2048-3200PRO</str> 
 <str name="name">CORSAIR  XMS 2GB (2 x 1GB) 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) Dual Channel Kit System Memory - Retail</str>
 <str name="manu">Corsair Microsystems Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <arr name="features">
  <str>CAS latency 2,    2-3-3-6 timing, 2.75v, unbuffered, heat-spreader</str>
 </arr>
 <float name="price">185.0</float>
 <str name="price_c">185,USD</str>
 <int name="popularity">5</int>
 <bool name="inStock">true</bool>
 <str name="store">37.7752,-122.4232</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|6.0 memory|3.0</str>
 <long name="_version_">1453219034197655552</long>
</doc>
<doc>
 <str name="id">VS1GB400C3</str>
 <str name="name">CORSAIR ValueSelect 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - Retail</str>
 <str name="manu">Corsair Microsystems Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <float name="price">74.99</float>
 <str name="price_c">74.99,USD</str>
 <int name="popularity">7</int>
 <bool name="inStock">true</bool>
 <str name="store">37.7752,-100.0232</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|4.0 memory|2.0</str>
 <long name="_version_">1453219034252181504</long>
</doc>
<doc>
 <str name="id">VDBDB1A16</str>
 <str name="name">A-DATA V-Series 1GB 184-Pin DDR SDRAM Unbuffered DDR 400 (PC 3200) System Memory - OEM</str>
 <str name="manu">A-DATA Technology Inc.</str>
 <str name="manu_id_s">corsair</str>
 <arr name="cat">
  <str>electronics</str>
  <str>memory</str>
 </arr>
 <arr name="features">
  <str>CAS latency 3,     2.7v</str>
 </arr>
 <int name="popularity">0</int>
 <bool name="inStock">true</bool>
 <str name="store">45.18414,-93.88141</str>
 <date name="manufacturedate_dt">2006-02-13T15:26:37Z</date>
 <str name="payloads">electronics|0.9 memory|0.1</str>
 <long name="_version_">1453219034255327232</long>
</doc>
<doc>
 <str name="id">3007WFP</str>
 <str name="name">Dell Widescreen UltraSharp 3007WFP</str>
 <str name="manu">Dell, Inc.</str>
 <str name="manu_id_s">dell</str>
 <arr name="cat">
  <str>electronics</str>
  <str>monitor</str>
 </arr>
 <arr name="features">
  <str>30" TFT active matrix LCD, 2560 x 1600, .25mm dot pitch, 700:1 contrast</str>
 </arr>
 <str name="includes">USB cable</str>
 <float name="weight">401.6</float>
 <float name="price">2199.0</float>
 <str name="price_c">2199,USD</str>
 <int name="popularity">6</int>
 <bool name="inStock">true</bool>
 <str name="store">43.17614,-90.57341</str>
 <long name="_version_">1453219041357332480</long>
</doc>
<doc>
 <str name="id">VA902B</str>
 <str name="name">ViewSonic VA902B - flat panel display - TFT - 19"</str>
 <str name="manu">ViewSonic Corp.</str>
 <str name="manu_id_s">viewsonic</str>
 <arr name="cat">
  <str>electronics</str>
  <str>monitor</str>
 </arr>
 <arr name="features">
  <str>19" TFT active matrix LCD, 8ms response time, 1280 x 1024 native resolution</str>
 </arr>
 <float name="weight">190.4</float>
 <float name="price">279.95</float>
 <str name="price_c">279.95,USD</str>
 <int name="popularity">6</int>
 <bool name="inStock">true</bool>
 <str name="store">45.18814,-93.88541</str>
 <long name="_version_">1453219045997281280</long></doc>
</result>
</response>

As we can see Solr responded correctly. This is because Solr already has the clusterstate.json file cached. To search Solr doesn’t need to update that file, so search should and is working as we could see.

Indexing with failed ZooKeeper

Without turning on our ZooKeeper instance we try to run the following command:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @hd.xml -H 'Content-type:application/xml'

The above command should result in indexing the contents of the hd.xml file. After a longer period of time Solr responds with the following information:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">503</int><int name="QTime">15096</int></lst><lst name="error"><str name="msg">Cannot talk to ZooKeeper - Updates are disabled.</str><int name="code">503</int></lst>
</response>

So as you can see we are not able to index data without working ZooKeeper ensemble.

Starting ZooKeeper again

So let’s see what will happen when we start our ZooKeeper instance again without restarting Solr nodes. After starting ZooKeeper we try to run the same indexing command, we just did, once again:

curl 'localhost:8983/solr/collection1/update?commit=true' --data-binary @hd.xml -H 'Content-type:application/xml'

And this time the response is different:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">118</int></lst>
</response>

As we can see the indexing request was successful this time. This allows us to assume that the connection to ZooKeeper was re-established by Solr. We can see that in Solr and ZooKeeper logs.

Short summary

As you can see, our short test allowed to see what happens when our ZooKeeper ensemble fails and what we can expect from Solr in such rare cases. I hope this blog entry will help you with some doubts about SolrCloud and its usefulnesses.

Please also remember that during the test, the cluster state did not change – all shards were accessible and working. We will see what will be happening when shards or replicas fails when ZooKeeeper is down in the next blog entry about SolrCloud.

Documents

Our documents contain information about participants of the competition – their id, name (as the author field) and email.. For example one record looks like that:

<doc>
  <field name="id">1</id>
  <field name="author">Solr.pl author</field>
  <field name="email">blog(at)solr.pl</field>
</doc>

Our very big data contains 19 records, maybe we should have used map/reduce ? :).

Schema

The schema.xml file describing the structure of the index is also very simple. In our case it contains the following fields:

<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="author" type="text_general" indexed="true" stored="true"/>
<field name="email" type="text_general" indexed="true" stored="true"/>

Additional configuration

Now we need to make sure that the schema.xml file contains the following definition type and field definitions:

<fieldType name="random" class="solr.RandomSortField" indexed="true" />
<dynamicField name="random_*" type="random" />

In the example schema.xml file provided with the standard Solr distribution package this type and dynamic field is available by default. We will need those to randomize our result set.

Running a query with random sorting

Running a query with a random sorting is a little bit tricky. We build a query like we usually do except for the sorting. For the sort parameter we will use the previously defined dynamic field with the random prefix. For example:

localhost:8983/solr/competition/select?q=*:*&sort=random_12939291%20desc

How it works ?

Solr will calculate ordering of the documents basing on the name of the random field and the index version. This means that every time you use the same field name and the same index (which was not changed between queries), you will get results that are ordered exactly the same way. This is disadvantage of this method, but sometimes this may be quite handy, like when doing paging (we don’t want to have different results ordering for each page, right ?). Because of this you have to generate the field name in your application that runs queries to Solr.

And now – Giveaway results !

We’ve used the above-mentioned query. Number used in sort field is absolutely random, that was randomized by saying: “Dad, tell me some random numbers” :). So the whole query we’ve used was:

localhost:8983/solr/collection1/select?q=*:*&indent=true&rows=2&sort=random_3721117253841%20desc

This above query gave the following results:

<result name="response" numFound="19" start="0">
  <doc>
    <str name="id">9</str>
    <str name="author">Rajeev Srivastava</str>
    <str name="email">[CENSORED]</str>
    <long name="_version_">1431017731370516481</long></doc>
  <doc>
    <str name="id">8</str>
    <str name="author">Evgeny</str>
    <str name="email">[CENSORED]</str>
    <long name="_version_">1431017731370516480</long></doc>
</result>

And the winners are

• Rajeev
• Evgeny

Congratulations ! We will contact you in the very near future with further information about how to receive your awards.  Once again congratulations! Also, to all the other participants, thanks for participations and your comments !

Let’s start with a little reminder about the book:

• Learn how to make Apache Solr search faster, more complete, and comprehensively scalable
• Solve performance, setup, configuration, analysis, and query problems in no time
• Get to grips with, and master, the new exciting features of Apache Solr 4

Read more about this book and download free Sample Chapter.

How to Enter ?

All you need to do is head on over to the book page (Apache Solr 4 Cookbook) and look through the product description of the book and drop a line via the comments below this post to let us know what interests you the most about this book. It’s that simple.

Product Description: http://www.packtpub.com/apache-solr-4-cookbook/book

Deadline

The contest will close on 28.03.2013. Winners will be contacted by email, so be sure to use your real email address when you comment!

Who Will Win ?

The winners will be chosen by the Solr.pl team randomly from readers entering the competition that replied with on topic comment.

If you want to increase your chances of winning, write a small review of the book using the sample chapter on Amazon.com and also forward the same post to bhavins@packtpub.com.

Book Format

The free copies will be provided in eBook format.

Update

The contents is now officially closed. Thank you to all the participants. The winners will be announced in a dedicated blog post right after Easter, on Tuesday 2nd of April.

ElasticSearch server book describes the most important and commonly used features of ElasticSearch (at least from our perspective). Example of topics discussed:

• ElasticSearch installation and configuration
• Static and dynamic index structure creation
• Querying ElasticSearch with Query DSL explained
• Using filters
• Faceting
• Routing
• Indexing data that is not flat

We also talk about:

• Autocomplete and how to implement it using ElasticSearch
• Percolator – what is it and how to use it
• ElasticSearch monitoring and being a fireman
• And much, much more

Eventhough we work with ElasticSearch everyday, we realized how big it was after digging into all the functionalities and cases. Because of that we were not able to describe all the features, but we hope we were able to choose the ones that are the most interested and needed.

Do you think that we should start writing about new enterprise search engine on solr.pl ?

Because of the fact that is was rather quietly on solr.pl lately, we would like to show you one of the reasons of this situation.

We are pleased to inform you, that the updated version of cookbook – “Solr 4.0 Cookbook” will be available in March 2013. The book is focused on the latest available version of Solr server – the 4.0.As before, the content is divided into ten thematic chapters and the book is again maintained in the cookbook conversion, which means that each recipe is focused on solving one particular problem.

From our point of view its worth to notice that the book was rewritten in the majority. More than half of the old recipes were removed and new ones came for them, for example ones describing Solr Cloud.

If you are interested, please refer to the Packt Publishing page: http://www.packtpub.com/apache-solr-4-cookbook/book, more details soon.

Options

As I wrote a few lines above we have two options – the first one is sending the binaries to search engine and use ExtractingRequestHandler (information about integrating Solr with Apache Tika can be found here) in Solr case, so it will make all the work for us. The second option is to use the same functionality (almost the same) to parse binary documents and get their contents before sending them to Solr. Of course there is a third option, not possible in most cases – get the documents you want to index in a format understandable by Solr

Processing on the Search Server Side

The simplest approach is to process your “rich documents” on the search server side. Lets assume its Apache Solr. We configure the ExtractingRequestHandler in the way we want it to work and we forget about everything else. But its not the right approach every time. You can imagine a situation when your indexing server is almost 100% utilized. If you would add another source of generating load you would probably suffer from performance problems. In such cases you will probably want to do it the other way.

Processing Outside of the Search Server

If the amount of rich documents is huge or your indexing server is almost completely utilized than it may be a good idea to process your binary files before sending them to your indexing server. Using Apache Tika for example we are able to build (quite easily) a good and reliable solution to process rich documents in your application. Of course such approach require a bit of knowledge about Java (or any other language you will use for content extraction). Such approach can save us from a situation where our indexing server is overloaded and because of the amount of data we can’t do anything with it.

A Few Words at the End

Once every few weeks we will be publishing posts that don’t cover one of the Apache Solr functionalities, but instead discuss some overall search problem or describe architecture of system with search as their part. We hope that such posts will allow us and you to look a bit wider on search topics than only from Apache Solr point of view.