Documents

Our documents contain information about participants of the competition – their id, name (as the author field) and email.. For example one record looks like that:

<doc>
  <field name="id">1</id>
  <field name="author">Solr.pl author</field>
  <field name="email">blog(at)solr.pl</field>
</doc>

Our very big data contains 19 records, maybe we should have used map/reduce ? :).

Schema

The schema.xml file describing the structure of the index is also very simple. In our case it contains the following fields:

<field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" />
<field name="author" type="text_general" indexed="true" stored="true"/>
<field name="email" type="text_general" indexed="true" stored="true"/>

Additional configuration

Now we need to make sure that the schema.xml file contains the following definition type and field definitions:

<fieldType name="random" class="solr.RandomSortField" indexed="true" />
<dynamicField name="random_*" type="random" />

In the example schema.xml file provided with the standard Solr distribution package this type and dynamic field is available by default. We will need those to randomize our result set.

Running a query with random sorting

Running a query with a random sorting is a little bit tricky. We build a query like we usually do except for the sorting. For the sort parameter we will use the previously defined dynamic field with the random prefix. For example:

localhost:8983/solr/competition/select?q=*:*&sort=random_12939291%20desc

How it works ?

Solr will calculate ordering of the documents basing on the name of the random field and the index version. This means that every time you use the same field name and the same index (which was not changed between queries), you will get results that are ordered exactly the same way. This is disadvantage of this method, but sometimes this may be quite handy, like when doing paging (we don’t want to have different results ordering for each page, right ?). Because of this you have to generate the field name in your application that runs queries to Solr.

And now – Giveaway results !

We’ve used the above-mentioned query. Number used in sort field is absolutely random, that was randomized by saying: “Dad, tell me some random numbers” :). So the whole query we’ve used was:

localhost:8983/solr/collection1/select?q=*:*&indent=true&rows=2&sort=random_3721117253841%20desc

This above query gave the following results:

<result name="response" numFound="19" start="0">
  <doc>
    <str name="id">9</str>
    <str name="author">Rajeev Srivastava</str>
    <str name="email">[CENSORED]</str>
    <long name="_version_">1431017731370516481</long></doc>
  <doc>
    <str name="id">8</str>
    <str name="author">Evgeny</str>
    <str name="email">[CENSORED]</str>
    <long name="_version_">1431017731370516480</long></doc>
</result>

And the winners are

• Rajeev
• Evgeny

Congratulations ! We will contact you in the very near future with further information about how to receive your awards.  Once again congratulations! Also, to all the other participants, thanks for participations and your comments !

Solr Cloud changed this behavior. In this article we will setup a new SolrCloud cluster from the scratch and we will see how it work.

Our example cluster

In our example we will use three Solr servers. Every server in the cluster is capable of handling both the index and the query requests. This is the main difference from the old-fashioned Solr architecture with single master and multiple slave servers. In the new architecture there is one additional element present: Zookeeper, which is responsible for holding configuration of the cluster and for synchronization of its work. It is crucial to understand that Solr relies on information stored in Zookeeper – if Zookeeper will fail, the whole cluster is useless. Because of this it is very important to have a fault tolerant Zookeeper ensemble and because of this we use three independent instances of Zookeeper that will form the ensemble.

Zookeeper installation

As we said previously, Zookeeper is a vital part of SolrCloud cluster. Although we can use embedded Zookeeper, this is only handy for testing. For production you definitely want your Zookeeper to be installed independently from Solr and run in a different Java virtual machine process to avoid those two interrupting each other and influencing each others work.

The installation of Apache Zookeeper is straight forward and may be described by the following steps:

1. Download Zookeeper archive from: http://www.apache.org/dyn/closer.cgi/zookeeper/
2. Unpack downloaded archive and copy conf/zoo_sample.cfg to conf/zoo.cfg
3. Modify zoo.cfg:
   1. Change dataDir to directory where you want to hold all cluster configuration data
   2. Add information about all Zookeeper servers (see below)

After mentioned changes my zoo.cfg looks like the following one:

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper/data
clientPort=2181
server.1=zk1:2888:3888
server.2=zk2:2888:3888
server.3=zk3:2888:3888

1. Copy this archive to the all servers, where Zookeeper service should be run
2. Create file /var/zookeeper/data/myid with server identifier. This identifier is different for each instance (for example on zk2 this file should contain 2 number)
3. Start all instances using “bin/zkServer.sh start-foreground” and verify validity of the installation
4. Add “bin/zkServer.sh start” to starting scripts and make sure that operation system monitors that Zookeeper service is available.

Solr installation

The installation of Solr is the following:

1. Download Solr archive from: http://www.apache.org/dyn/closer.cgi/lucene/solr/4.1.0
2. Unpack downloaded archive
3. In this tutorial we will use the ready Solr installation from the example directory and all changes are made to this example installation
4. Copy archive to all servers which are the part of the cluster
5. Install to Zookeeper configuration data, which will be used by the Solr cluster. For this run the first instance with:

   java -Dbootstrap_confdir=./solr/collection1/conf -Dcollection.configName=solr1 -DzkHost=zk1:2181 -DnumShards=2 -jar start.jar

This should be run only once. The next run will use configuration from Zookeeper cluster and local configuration files are not needed.

1. Run all instances using

   java –DzkHost=zk1:2181 –jar start.jar

Verify the installation

Go into administration panel on any Solr instance. For our deployment the URL should be like http://solr1:8983/solr. When you click on cloud tab, and graph, you should see something similar to the following screen shot:

Collection

Our first collection – the collection1 is divided into two shards (shard1 and shard2). Each of those shards is placed on two Solr instances (OK, on the picture you see that every Solr is placed on the same host – I have currently only one physical server available for tests – any volunteers for donation? ;)). You can see that type of the dot tell us if it is a primary shard or replica.

Summary

I hope this is the first note about solrCloud. I know it is very short and skips details and information about shards, replicas and architecture of this solution. Treat this as a simple checklist for basic, (but real) configuration of your cloud.

ElasticSearch server book describes the most important and commonly used features of ElasticSearch (at least from our perspective). Example of topics discussed:

• ElasticSearch installation and configuration
• Static and dynamic index structure creation
• Querying ElasticSearch with Query DSL explained
• Using filters
• Faceting
• Routing
• Indexing data that is not flat

We also talk about:

• Autocomplete and how to implement it using ElasticSearch
• Percolator – what is it and how to use it
• ElasticSearch monitoring and being a fireman
• And much, much more

Eventhough we work with ElasticSearch everyday, we realized how big it was after digging into all the functionalities and cases. Because of that we were not able to describe all the features, but we hope we were able to choose the ones that are the most interested and needed.

Do you think that we should start writing about new enterprise search engine on solr.pl ?

Full list of changes in Apache Lucene 4.0 beta, compared to 4.0 alpha version, can be found at the following address: http://wiki.apache.org/lucene-java/ReleaseNote40beta. The full list of changes in Apache Solr 4.0 can be found at the Solr wiki as well: http://wiki.apache.org/solr/ReleaseNote40beta.

In order to download Apache Lucene 4.0 beta library please go to: http://www.apache.org/dyn/closer.cgi/lucene/java/. Apache Solr 4.0 beta can be downloaded at the following address: http://www.apache.org/dyn/closer.cgi/lucene/solr/.

Please note that not all mirrors have to be updated yet.

Requirements

Our autocomplete mechanism has the following requirements:

1. We return whole phrase, not just s single word
2. Returned phrase can be present multiple times in the index
3. We want to know the number of results for the returned phrase
4. Common phrases should be shown higher than the less common ones
5. Order of words entered by the user doesn’t matter

Solution

Solution given in the first part of the series will not met the requirements because of the first requirement. Of course we could change analysis type, but we wouldn’t return the whole phrase.

Solution to the above requirements is the modified faceting method. Instead of searching all the elements and narrowing results with facet.prefix parameter, we can search only for those elements that have the word fragment we are looking for. We don’t want wildcard query to be used (because of performance) we call ngram’s for the rescue. This means we need to write the ngrams into the index (of course Solr will do that for us). The obvious flaw is the index size growth, but in this case we can live with that.

Schema.xml

We define an additional type:

  <fieldType name="text_autocomplete" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.EdgeNGramFilterFactory" minGramSize="1" maxGramSize="25" />
    </analyzer>
    <analyzer type="query">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
    </analyzer>
  </fieldType>

We also define additional fields: one which value we plan to return and one which will be used for searching:

<field name="tag_autocomplete" type="text_autocomplete" indexed="true" stored="true" omitNorms="true" omitTermFreqAndPositions="true"/>
<field name="tag" type="string" indexed="true" stored="true" />

And one copyField to make things easier:

<copyField source="tag" dest="tag_autocomplete"/>

Query

After indexation we are ready to test our queries:

1. We are narrowing results, only to those which have the interesting word fragment in the tag_autocomplete field, with: q=tag_autocomplete:(PHRASE)
2. We need all the fragments entered by the user, so we use AND as our logical operator: q.op=AND
3. We not interested in the actual query results, we will use data returned by faceting, so we say: rows=0
4. We need faceting: facet=true
5. We need faceting on the field where we store the original phrase: facet.field=tag
6. We are not interested in empty tags: facet.mincount=1
7. We are only interested in 5 autocomplete values: facet.limit=5

And the final query:

?q=tag_autocomplete:(PHRASE)&q.op=AND&rows=0&facet=true&facet.field=tag&facet.mincount=1&facet.limit=5

If we will configure out search handler to include all the constant parameters, we will have the following query:

?q=tag_autocomplete:(PHRASE)

At the end

The basic virtue of the presented method is the ability to use one field for searching and other for returning results. Because of that, we were able to return the whole phrase instead of a single word.

• bugfixes
• initial support for dla Solr 4.0
• support for ruby 1.9

Source code is available on our  GitHub.

To refresh Your memory: it is a tool to visualize Solr search results and get the following information:

• why a document was placed in the results list,
• why a document wasn’t placed in the results list.

We decided to release the code, even though there are many things left that we would like to change and make better. On the other hand, there will always be places to make code better

As for the application – it was developed with the use of Ruby On Rails and even though You may see that the code was developed by people using other programming languages, we hope that You will find the application useful.

Source code is available under Apache license (2.0), but it would be very happy to hear from You if You plan to use it. We also encourage to send us corrections and report issues.

You can find the source code on Github: https://github.com/solrpl/explain

We noticed the following:

• Explain tool have goaded some attention, but most to the explains are marked as private.
• In relatively large amount of explain only query was submit instead of the whose response.
• Often the results don’t have diagnostic information (debugQuery=true).
• Unreleased Apache Solr 4.0 is quite popular
• Explain parser did a good job (of course without counting the obvious Solr 4.0 explain parsing problems).

Today new version of explain.solr.pl was deployed. The changes are as follows:

• Simple analysis of query performance was added.
• Displayed messages are now more user friendly.
• Explain parser was corrected according to text files generated by Solr 3.5 unit test cases.
• Minor GUI enhancements.

• rebuilding Lucene explain information analyzer so it is ready for the changes in Lucene 4.0 (per field similarity and flexible similarity)
• user interface rebuild along with some changes to the usability
• code cleaning and preparation for publishing sources

Fortunately autumn and winter evenings are just perfect for work We will try to make http://explain.solr.pl available as soon as possible.

For us the most important was the second day and the presentation of the results of our work: Understanding & Visualising Solr ‘explain’ Information. The presentation consisted of two parts. First, the theoretical, described what Solr presents about the validity of the returned document. This information, unfortunately, do not belong to the most readable, especially when the search is done on many fields, often using dedicated query parser (the screenshot contains explain fragment describing a hit in only one document (!)).

The situation gets worse because the questions from the client about the documents positioning are one of the most time consuming ones during the project.

The second part of the presentation showed what we decided to do with this problem. The idea came after seeing explain.depesz.com – a similar tool for visualization of information provided by the postgreSQL database.

In retrospect, our explain seems to be a good idea. Although its not finished we are using the tool almost everyday. What’s more – the Lucene Eurocon presentation showed us that there is also considerable interest in the community for a tool like this. Thank you for your kind words and the promises of sending us your own modules.

The current version explain.solr.pl focuses on reading the information generated by the solr 3.x. We are currently working on going public, and (somewhat later) the opensourcing the code.

To sum up the plans for the nearest future are quite ambitious:)

1. Firstly – the chronicler’s duty – we will try to describe how Lucene Eurocon 2011 looked from our perspective. We plan to publish two entries – each of which will be dedicated to one day of the conference. That is something to read before they appear on the official videos from the conference.
2. Opening explain.solr.pl for a wide audience:)
3. Publication of explain.solr.pl code (github)
4. Back to more regular posts about Solr.

Keep your fingers crossed:)