I made a promise and here it is – the entry on the most common mistakes when designing Solr index, which is when You create or modify the schema.xml file for Your system implementation. Feel free to read on 😉
Each of us knows what is schema.xml file and what is (if not, I invite you to read the entry located at: http://solr.pl/2010/08/16/what-is-schema-xml/?lang=en). What are the most frequently commit errors creating or updating this file? I personally met with the following:
1. Trash in the configuration
I confess that the first principle is to keep the file schema.xml in the simplest possible form. Linked to this is a very important issue – this file should not be synonymous with chaos. In other word, do not stick with unnecessary comments, unwanted types, fields and so on. Order in the structure of the schema.xml file not only helps us to maintain this file and its modifications with ease, but also assures us that no information that is unnecessary will be stored in Solr index.
2. Cosmetic changes to the default configuration
How many of those who use Solr in their daily work took the default schema.xml file supplied in the example implementation Solr and only slightly modified the contents – for example, changing only the names of the fields ? I should raise my hand too, because I did it once. This is a pretty big mistake. Someone may ask why. Are you sure You need English stemming when implementing search for content written in Polish ? I think not. The same applies to field and type attributes like term vectors.
3. No updates
Sometimes I find the implementation of search based application, where update of Solr does not mean an update of schema.xml file. If it is a conscious decision, dictated by such costly or even impossible re-indexing of all data, I understand the situation. But there are cases where an upgrade would bring only benefits, and where costs of such upgrade would be minimal (eg less expensive re-index or slight changes in the application). Do not be afraid to update the schema.xml file – whether it is to update the fields, update types, whether the addition of newer stuff. A good example is the migration from Solr 1.3 to version 1.4 – newer version introduced significant changes associated with numeric types, where migration to the new types would result in great increase in query performance using those types (such as queries using value ranges).
4. “I`ll use it one day”
Adding new types, not removing unnecessary now, the same in the case of fields, or copyField definition. Most of us think – that old definition can be useful in the future, but remember that each type is some extra portion of memory needed by Solr, each field is a place in the index. My small advice – if you stop to use the type, field, or whatever else you have in your configuration file (not only in the schema.xml), simply remove it from this file. Applying this principle throughout the life cycle of the applications using Solr will ensure You that the index is in optimal condition, and after a few months since another feature implementation You will not need to be puzzled and as a result You will not need to dig into the application code to determine if the field is used in some forgotten code fragment.
5. Attributes, attributes and again attributes
Preservation of original values, adding term vectors and its properties are just examples of things we don`t need in every implementation. Sometimes we have more than required by the application index. A larger index, lower productivity, at least in some cases (eg, indexing). It is worth considering if you really need all this information, which we say to Solr to calculate and store. Removing some unnecessary, of course, from our point of view of information, may surprise us. Sometimes it is worth a try;)
Feel free to comment, because I will read eagerly, for what else we should pay attention to when modifying schema.xml file.
Finally, I think that it is worth to mention the article “The Seven Deadly Sins of Solr” LucidImagination published on the website at: http://www.lucidimagination.com/blog/2010/01/21/the-seven-deadly-sins-of-solr. It describes bad practices when working with Solr. In my opinion, interesting reading. I highly recommend it.
This post is also available in: Polish