Possibilities

Let’s look at the new API by example.

Getting information in XML format

Many Solr users are used to getting their data in the XML format, at least when using Solr HTTP API. However, the schema API uses JSON as the default format. In order to get the data in the XML format in all the below examples, you’ll need to appeng the wt=xml parameter to the call, for example like that:

$curl 'http://localhost:8983/solr/collection1/schema/fieldtypes?wt=xml'

Defined fields information

Let’s start by looking at how to fetch information about the fields that are defined in Solr. In order to do that we have the following possibilities:

1. Get information about all the fields defined in the index
2. Get information for a one, explicitly defined field

In the first case we should use the following command:

$curl 'http://localhost:8983/solr/collection1/schema/fields'

In second case we should add the / character and the field name to the above command. For example in order to get the information about the author field we should use the following command:

$curl 'http://localhost:8983/solr/collection1/schema/fields/author'

Solr response for the first command will be similar to the following one:

{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "fields":[{
      "name":"_version_",
      "type":"long",
      "indexed":true,
      "stored":true},
    {
      "name":"author",
      "type":"text_general",
      "indexed":true,
      "stored":true},
    {
      "name":"cat",
      "type":"string",
      "multiValued":true,
      "indexed":true,
      "stored":true},
    {
      "name":"category",
      "type":"text_general",
      "indexed":true,
      "stored":true},
    {
      "name":"id",
      "type":"string",
      "multiValued":false,
      "indexed":true,
      "required":true,
      "stored":true,
      "uniqueKey":true},
    {
      "name":"url",
      "type":"text_general",
      "indexed":true,
      "stored":true},
    {
      "name":"weight",
      "type":"float",
      "indexed":true,
      "stored":true}]}

On the other hand the response for the second command would be as follows:

{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "field":{
    "name":"author",
    "type":"text_general",
    "indexed":true,
    "stored":true}}

Getting information about defined dynamic fields

Similar to what information we can get about the fields defined in the schema.xml we can get the information about dynamic fields. Again we have to options:

1. Get information about all dynamic fields
2. Get information about specific dynamic field pattern

In order to get all the information about dynamic fields we should use the following command:

$curl 'http://localhost:8983/solr/collection1/schema/dynamicfields'

In order to get information about a specific pattern we append the / character followed by the pattern, for example like this:

$curl 'http://localhost:8983/solr/collection1/schema/dynamicfields/random_*'

Solr will return the following response for the first query:

{
  "responseHeader":{
    "status":0,
    "QTime":2},
  "dynamicfields":[{
      "name":"*_coordinate",
      "type":"tdouble",
      "indexed":true,
      "stored":false},
    {
      "name":"ignored_*",
      "type":"ignored",
      "multiValued":true},
    {
      "name":"random_*",
      "type":"random"},
    {
      "name":"*_p",
      "type":"location",
      "indexed":true,
      "stored":true},
    {
      "name":"*_c",
      "type":"currency",
      "indexed":true,
      "stored":true}]}

And the following response will be returned for the second command:

{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "dynamicfield":{
    "name":"random_*",
    "type":"random"}}

Getting field types

As you probably guess, in a way similar to the above describes examples, we can also get the information about the field types defined in our schema.xml files. We can fetch the following information:

1. All the field types defined in the schema.xml file
2. A single type

To get all the defined field types we should run the following command:

$curl 'http://localhost:8983/solr/collection1/schema/fieldtypes'

The get information about a single type we should again add the / character and append the field type name to it, for example like this:

$curl 'http://localhost:8983/solr/collection1/schema/fieldtypes/text_gl'

Solr will return the following information in response to the first command:

{
  "responseHeader":{
    "status":0,
    "QTime":3},
  "fieldTypes":[{
      "name":"alphaOnlySort",
      "class":"solr.TextField",
      "sortMissingLast":true,
      "omitNorms":true,
      "analyzer":{
        "class":"solr.TokenizerChain",
        "tokenizer":{
          "class":"solr.KeywordTokenizerFactory"},
        "filters":[{
            "class":"solr.LowerCaseFilterFactory"},
          {
            "class":"solr.TrimFilterFactory"},
          {
            "class":"solr.PatternReplaceFilterFactory",
            "replace":"all",
            "replacement":"",
            "pattern":"([^a-z])"}]},
      "fields":[],
      "dynamicFields":[]},
    {
      "name":"boolean",
      "class":"solr.BoolField",
      "sortMissingLast":true,
      "fields":["inStock"],
      "dynamicFields":["*_bs",
        "*_b"]},
    {
      "name":"text_gl",
      "class":"solr.TextField",
      "positionIncrementGap":"100",
      "analyzer":{
        "class":"solr.TokenizerChain",
        "tokenizer":{
          "class":"solr.StandardTokenizerFactory"},
        "filters":[{
            "class":"solr.LowerCaseFilterFactory"},
          {
            "class":"solr.StopFilterFactory",
            "words":"lang/stopwords_gl.txt",
            "ignoreCase":"true",
            "enablePositionIncrements":"true"},
          {
            "class":"solr.GalicianStemFilterFactory"}]},
      "fields":[],
      "dynamicFields":[]},
    {
      "name":"tlong",
      "class":"solr.TrieLongField",
      "precisionStep":"8",
      "positionIncrementGap":"0",
      "fields":[],
      "dynamicFields":["*_tl"]}]}

In response to the second command Solr will return the following:

{
  "responseHeader":{
    "status":0,
    "QTime":2},
  "fieldType":{
    "name":"text_gl",
    "class":"solr.TextField",
    "positionIncrementGap":"100",
    "analyzer":{
      "class":"solr.TokenizerChain",
      "tokenizer":{
        "class":"solr.StandardTokenizerFactory"},
      "filters":[{
          "class":"solr.LowerCaseFilterFactory"},
        {
          "class":"solr.StopFilterFactory",
          "words":"lang/stopwords_gl.txt",
          "ignoreCase":"true",
          "enablePositionIncrements":"true"},
        {
          "class":"solr.GalicianStemFilterFactory"}]},
    "fields":[],
    "dynamicFields":[]}}

As you can see, the amount information is nice as we are getting all the information about the field types and in addition to that the information which field are using give field (both dynamic and non-dynamic.

Retrieving information about copyFields

In addition to what we’ve discussed so far we are able to get information about copyFields section from the schema.xml. In order to do that one should run the following command:

$curl 'http://localhost:8983/solr/collection1/schema/copyfields'

And in response we will get the following data:

{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "copyfields":[{
      "source":"author",
      "dest":"text"},
    {
      "source":"cat",
      "dest":"text"},
    {
      "source":"content",
      "dest":"text"},
    {
      "source":"content_type",
      "dest":"text"},
    {
      "source":"description",
      "dest":"text"},
    {
      "source":"features",
      "dest":"text"},
    {
      "source":"author",
      "dest":"author_s",
      "destDynamicBase":"*_s"}]}

The future

In Solr 4.3 the described API was improved and is now being prepared to enable not only reading of the index structure, but also writing modifications to it with the use of HTTP requests. We can expect that feature in one of the upcoming versions of Apache Solr, so its worth waiting in my opinion, at least by those who needs it.

W Solr 4.3 opisywane API zostało usprawnione oraz jest przygotowywane do umożliwienia zmian w strukturze indeksu za pomocą protokołu HTTP. Możemy zatem spodziewać się, iż w jednej z kolejnych wersji serwera wyszukiwania Solr otrzymamy możliwość łatwej zmiany struktury indeksu, przynajmniej takich, które nie będą powodować konfliktów z już zaindeksowanymi danymi.

Requirements specification

We need to analyze the data, that is indexed in the “content” field. Let’s examine the sample data, that will be used for helping to create the new “text” type configuration:

• Make: Audi
  Model: 80, 90, A6, A8, TT

• Make: BMW
  Model: M3, M5, Series 7, Series 8, X1, X3

• Make: Chevrolet
  Model: TrailBlazer

• Make: Citroen
  Model: C-Crosser, C3 Pluriel, C4 Picasso

• Make: Ford
  Model: C-MAX, S-MAX

• Make: Honda
  Model: Accord, CR-V, FR-V, HR-V

• Make: Kia
  Model: Cee’d

• Make: Suzuki
  Model: Alto/Maruti

Make names are simple words, that are easily handled by the current configuration (WhitespaceTokenizer + LowerCaseFilter). The problem is with the model names, as they contain additional characters and separators, that we often ignore when entering the search phrase. Let’s try to put the sample date into some groups, that will help us with the incoming configuration:

1. Model names, that do not need to be processed by any additional filters (the current “text” type configuration is sufficient) – 80, 90, TT, Series 7, Series 8, Accord
2. Model names, which contain letters and numbers, where we want to split on letter-number transitions – A6, A8, M3, M5, X1, X3, C3 Pluriel, C4 Picasso. We would like to be able to find those models when entering only a letter, only a number and whole model name too.
3. Models, which have the case transitions in the name – TrailBlazer. We would like to find the document with this name when entering “trail”, “blazer”, “trailBlazer”, “trailblazer”.
4. Model names, that contain intra-word delimiters, which we want to ignore or split on them – C-Crosser, C-MAX, S-MAX, CR-V, FR-V, HR-V, Alto/Maruti.
   Example: we would like to find the document with the model name “C-MAX” entering the phrases “c”, “max”, “c-max” “cmax”.
5. We intentionally omitted the “Cee’d” model name in the 4th point as we would like to treat this example a little differently. We don’t want to be able to find this model when entering the “cee” and “d” phrases. We treat the name only as the whole word – “cee’d” or “ceed”.

WordDelimiterFilter configuration

With the given configuration we’ve described above, we are going to add proper values to the WordDelimiterFilter attributes in order to satisfy our needs:

1. WordDelimiterFilter is needless in this case, as the current “text” type configuration (WhitespaceTokenizer + LowerCaseFilter) is sufficient.
2. In order to face the 2nd point requirements we need to set the proper values of the following attributes:
   • generateWordParts=”1″ – the value must be set to “1” if we want to be able to generate parts of words
   • generateNumberParts=”1″ – the value must be set to “1” if we want to be able to generate parts of number words
   • splitOnNumerics=”1″ – the value must be set to “1” if we want to be able to generate a new parts from alphabet => number transitions
3. In order to face the 3rd point requirements we need to set the proper values of the following attributes:
   • generateWordParts=”1″
   • splitOnCaseChange=”1″ – the value must be set to “1” if we want to be able to split on lowercase => uppercase transitions
4. In order to face the 4th point requirements we need to set the proper values of the following attributes:
   • generateWordParts=”1″
   • catenateWords=”1″ – the value must be set to “1” if we want to be able to ignore the intra-word delimiters by joining the subwords

So let’s take a look at our WordDelimiterFilter configuration:

<filter class="solr.WordDelimiterFilterFactory"
 splitOnNumerics="1"
 splitOnNumerics="1"
 generateWordParts="1"
 generateNumberParts="1"
 catenateWords="1"
/>

Additionaly we may notice that the default value of the “splitOnNumerics” and “splitOnNumerics” attributes is “1”. The rest of the WordDelimiterFilter’s attributes (except the “stemEnglishPossessive”) have the default value set to “0”. So our configuration can be reduced to:

<filter class="solr.WordDelimiterFilterFactory"
 generateWordParts="1"
 generateNumberParts="1"
 catenateWords="1"
 stemEnglishPossessive="0"
/>

What about the 5th point of our data specification ? As we have stated, we wouldn’t like to treat the “‘” sign as the intra-word delimiter. So maybe we could use the protected=”protwords.txt” option of the WordDelimiterFilter which will keep the word “Cee’d” unchanged ? Ok, but we would also like to be able to find this model when entering the “ceed” phrase, so this option is not good for us. The best solution would be to take care of this case in the separate filter and leave the WordDelimiterFilter with nothing to do.

PatternReplaceFilter configuration

we are going to put the PatternReplaceFilter before the WordDelimiterFilter. Using the PatternReplaceFilter we will be able to ignore the ” ‘ ” sign by replacing it with the empty sign. Configuring the filter this way, the WordDelimiterFilter will receive the “Ceed” token and will not modify this value. The configuration of the filters will be the same for indexing and searching, so a user will be able to find the offer with the “Cee’d” model when entering the phrases “cee’d” and “ceed”:

<filter class="solr.PatternReplaceFilterFactory" pattern="'" replacement="" replace="all" />

New “text” type configuration visualization

Let’s take a look at our new “text” type:

<fieldType name="text" positionIncrementGap="100">
 <analyzer>
  <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   <filter class="solr.PatternReplaceFilterFactory" pattern="'" replacement="" replace="all" />
   <filter class="solr.WordDelimiterFilterFactory"
    generateWordParts="1"
    generateNumberParts="1"
    catenateWords="1"
    stemEnglishPossessive="0"
  />
  <filter class="solr.LowerCaseFilterFactory"/>
 </analyzer>
</fieldType>

We are going to use the solr’s administration panel to find out if the configuration we’ve created is correct:

1. (Model: “80”) As we’ve expected, our new filters don’t influence the data typical for the 1st point.
2. (Model: “A8”) WordDelimiterFilter did the split on letter-number transitions.
3. (Model: “TrailBlazer”) WordDelimiterFilter did the case transition generating “trail” and “Blazer” tokens. Additionaly we have the opportunity to enter the “trailblazer” phrase. Superb!
4. (Model: “CR-V”) WordDelimiterFilter ignored the intra-word delimiters by generating subwords(“cr” and “v”) and joining the subwords additionaly (“crv”).
5. (Model: “Cee’d”) PatternReplaceFilter have replaced the “Cee’d” word to “Ceed” and the WordDelimiterFilter have only passed the value. That’s what we needed.

The end

In this post we’ve showed how to configure two new filters in order to improve the search results quality – WordDelimiterFilter and PatternReplaceFilter. Our website users are satisfied … for now.

Requirements specification

Imagine we would like to use solr to provide our car sale website with a search engine. The functional part of our website is at the beginning rather primitive and takes the advantage of only the small piece of every car information:

• make
• model
• year of production
• price
• engine size
• mileage
• colour
• damaged

We would like to design a simple configuration schema file, which will make possible to index data from the given fields. But before we open the schema.xml file and start typing, let’s answer the seven fundamental questions related to our fields:

1. What is the field type ?

Let’s determine the type of every field:

• make – text field
• model – text field
• year of production – integer field
• price – float field
• engine size – integer field
• mileage – integer field
• colour – textual field
• damaged – logical field

So what ?

So we will need some basic type definitions like string, boolean, int, float.

2. Is it the field used in search process ?

We would like to use the data from some fields in order to enable our search engine to find the proper documents (car sale announcements). To accomplish that we are going to use 3 fields: make, model, year of production.

So what ?

So we will need to create another field type, which will contain some filters to make finding the documents easy and efficient. We will create another field of the newly created type, where we will put all the data from make, model and year of production fields.

3. Is it the field used in faceting or sorting operation ?

In our website we would like to sort search results using 4 fields: model, year of production, price and mileage. We would also like to be able to to use facet operation on fields: make, model, year of production and colour.

So what ?

When we want to create a field type for fields used for sorting/faceting, then we need to know that this type cannot contain tokenizers and filters which can tokenize values in those fields. But still we want the values to be lowercased, so the letters size does not influence the sorting/faceting results. So that’s the kind of another field type we will need to create.

4. Is it the field used to filter search results?

We would like to have the possibility to filter search results using ranges on fields: year of production, price, engine size and mileage.

So what ?

So let’s use the field types which will accelerate range queries.

5. Are there any fields which are not mentioned in the questions number 2, 3 or 4 ?

There is a field “damaged” which is not supposed to be involved in any of the mentioned operations.

So what?

So we will set the value of the “indexed” attribute to “false”.

6. Is the field required ?

We assume that there are 3 fields which are supposed to be required: make, model and year. We don’t want to have documents in index (car sale announcements available in the search process), which do not have values in those fields.

So what ?

So we will set the value of the “required” attribute to “true”.

7. Do we need to retrieve the information from the field in the original state?

We would like to retrieve the information from all of the fields mentioned in the requirements specification and present them directly on the website.

So what?

So we will set the value of the “stored” attribute to “true”.

Let’s add field type definitions

We’ve answered our questions, we’ve come to some conclusions so let’s add field types to the schema file:

We add the solr.StrField type, which is not analysed and can be used for example as the type for the unique document key.

<fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>

Add the boolean type:

<fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/>

Now the numerical types. Remember that we need types that can help us to accelerate range queries. So let’s use the tint and tfloat types:

    <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>

Now let’s create the textual type, which will be a definition type for the catch-all field used for searching. For now, the type with the whitespace tokenizer and the lowercase filter will be just fine:

    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

And last, but not least, the type for the sortable/facetable fields. What we need is the type that lowercases the entire field value, keeping it as a single token. KeywordTokenizer does no actual tokenizing, so it is the ideal tokenizer for our need. The TrimFilterFactory removes any leading or trailing whitespace:

    <fieldType name="lowercase" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.KeywordTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.TrimFilterFactory" />
      </analyzer>
    </fieldType>

Time to add field definitions

Document id:

   <field name="id" type="string" indexed="true" stored="true" required="true" />

Make and model:

   <field name="make" type="text" indexed="false" stored="true" required="true" />
   <field name="model" type="text" indexed="false" stored="true" required="true" />

Now why is the value of the “indexed” attribute set to “false” ? As far as we know, we need those fields to search, sort and facet operations. That’s true … but we need to notice that for the searching purposes we will copy the data from those fields to one catch-all field:

   <field name="content" type="text" indexed="true" stored="false" multiValued="true"/>

and for the sorting/faceting purposes we will copy the data yet to other fields of the type “lowercase”:

   <field name="make_sort" type="lowercase" indexed="true" stored="false" />
   <field name="model_sort" type="lowercase" indexed="true" stored="false" />

So the fields make and model will not take part in the operations itself and we can set the “indexed” attribute to “false” for best index size.

The rest of the fields:

   <field name="year" type="tint" indexed="true" stored="true" required="true" />
   <field name="price" type="tfloat" indexed="true" stored="true" />
   <field name="engine_size" type="tint" indexed="true" stored="true" />
   <field name="mileage" type="tint" indexed="true" stored="true" />
   <field name="colour" type="lowercase" indexed="true" stored="true" />

Remember about the “false” value of the “indexed” attribute of the “damaged” field:

 <field name="damaged" type="boolean" indexed="false" stored="true" />

copyField – let’s index the same data differently

We have mentioned the field values copying several times already so now let’s define copy fields.

Fields used for searching are copied to catch-all “content” field. There is more than one source field, that’s why the “content” field definition contains the multiValued attribute set to “true”:

 <copyField source="make" dest="content"/>
 <copyField source="model" dest="content"/>
 <copyField source="year" dest="content"/>

Copying the sortable/facetable fields:

 <copyField source="make" dest="make_sort"/>
 <copyField source="model" dest="model_sort"/>

Anything else ?

We shall add 3 more elements to the schema:

The unique key of the document:

 <uniqueKey>id</uniqueKey>

Default search field:

 <defaultSearchField>content</defaultSearchField>

Default query parser operator. Let’s set it to “AND”.

 <solrQueryParser defaultOperator="AND"/>

It’s done! The schema.xml configuration file is ready and looks like this:

<?xml version="1.0" encoding="UTF-8" ?>

<schema name="carsale" version="1.2">

  <types>
    <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>

    <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/>

     <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>
    <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" omitNorms="true" positionIncrementGap="0"/>

    <fieldType name="text" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.WhitespaceTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory"/>
      </analyzer>
    </fieldType>

    <fieldType name="lowercase" class="solr.TextField" positionIncrementGap="100">
      <analyzer>
        <tokenizer class="solr.KeywordTokenizerFactory"/>
        <filter class="solr.LowerCaseFilterFactory" />
        <filter class="solr.TrimFilterFactory" />
      </analyzer>
    </fieldType>

 </types>

 <fields>
   <field name="id" type="string" indexed="true" stored="true" required="true" />
   <field name="make" type="text" indexed="false" stored="true" required="true" />
   <field name="model" type="text" indexed="false" stored="true" required="true" />
   <field name="make_sort" type="lowercase" indexed="true" stored="false" />
   <field name="model_sort" type="lowercase" indexed="true" stored="false" />
   <field name="year" type="tint" indexed="true" stored="true" required="true" />
   <field name="price" type="tfloat" indexed="true" stored="true" />
   <field name="engine_size" type="tint" indexed="true" stored="true" />
   <field name="mileage" type="tint" indexed="true" stored="true" />
   <field name="colour" type="lowercase" indexed="true" stored="true" />
   <field name="damaged" type="boolean" indexed="false" stored="true" />
   <field name="content" type="text" indexed="true" stored="false" multiValued="true"/>

 </fields>

 <uniqueKey>id</uniqueKey>

 <defaultSearchField>content</defaultSearchField>

 <solrQueryParser defaultOperator="AND"/>

 <copyField source="make" dest="content"/>
 <copyField source="model" dest="content"/>
 <copyField source="make" dest="make_sort"/>
 <copyField source="model" dest="model_sort"/>
 <copyField source="year" dest="content"/>

</schema>

The end

In today’s post we have created the simple schema.xml file, which allows us to index data, so that we are able to face our car sale website search functionalities. But still we want to develop our website which will surely affects the schema … and not only the schema. In the next “car sale” related post we will try to face some new requirements and provide next modifications.

Each of us knows what is schema.xml file and what is (if not, I invite you to read the entry located at: http://solr.pl/2010/08/16/what-is-schema-xml/?lang=en). What are the most frequently commit errors creating or updating this file? I personally met with the following:

1. Trash in the configuration

I confess that the first principle is to keep the file schema.xml in the simplest possible form. Linked to this is a very important issue – this file should not be synonymous with chaos. In other word, do not stick with unnecessary comments, unwanted types, fields and so on. Order in the structure of the schema.xml file not only helps us to maintain this file and its modifications with ease, but also assures us that no information that is unnecessary will be stored in Solr index.

2. Cosmetic changes to the default configuration

How many of those who use Solr in their daily work took the default schema.xml file supplied in the example implementation Solr and only slightly modified the contents – for example, changing only the names of the fields ? I should raise my hand too, because I did it once. This is a pretty big mistake. Someone may ask why. Are you sure You need English stemming when implementing search for content written in Polish ? I think not. The same applies to field and type attributes like term vectors.

3. No updates

Sometimes I find the implementation of search based application, where update of Solr does not mean an update of schema.xml file. If it is a conscious decision, dictated by such costly or even impossible re-indexing of all data, I understand the situation. But there are cases where an upgrade would bring only benefits, and where costs of such upgrade would be minimal (eg less expensive re-index or slight changes in the application). Do not be afraid to update the schema.xml file – whether it is to update the fields, update types, whether the addition of newer stuff. A good example is the migration from Solr 1.3 to version 1.4 – newer version introduced significant changes associated with numeric types, where migration to the new types would result in great increase in query performance using those types (such as queries using value ranges).

4. “I`ll use it one day”

Adding new types, not removing unnecessary now, the same in the case of fields, or copyField definition. Most of us think – that old definition can be useful in the future, but remember that each type is some extra portion of memory needed by Solr, each field is a place in the index. My small advice – if you stop to use the type, field, or whatever else you have in your configuration file (not only in the schema.xml), simply remove it from this file. Applying this principle throughout the life cycle of the applications using Solr will ensure You that the index is in optimal condition, and after a few months since another feature implementation You will not need to be puzzled and as a result You will not need to dig into the application code to determine if the field is used in some forgotten code fragment.

5. Attributes, attributes and again attributes

Preservation of original values, adding term vectors and its properties are just examples of things we don`t need in every implementation. Sometimes we have more than required by the application index. A larger index, lower productivity, at least in some cases (eg, indexing). It is worth considering if you really need all this information, which we say to Solr to calculate and store. Removing some unnecessary, of course, from our point of view of information, may surprise us. Sometimes it is worth a try;)

Feel free to comment, because I will read eagerly, for what else we should pay attention to when modifying schema.xml file.

Finally, I think that it is worth to mention the article “The Seven Deadly Sins of Solr” LucidImagination published on the website at: http://www.lucidimagination.com/blog/2010/01/21/the-seven-deadly-sins-of-solr. It describes bad practices when working with Solr. In my opinion, interesting reading. I highly recommend it.

Schema.xml file consists of several parts:

• version,
• type definitions,
• field definitions,
• copyField section,
• additional definitions.

Version

The first thing we come across in the schema.xml file is the version. This is the information for Solr how to treat some of the attributes in schema.xml file. The definition is as follows:

<schema name="example" version="1.3">

Please note that this is not the definition of the version from the perspective of your project. At this point Solr supports four versions of a schema.xml file:

• 1.0 – multiValued attribute does not exist, all fields are multivalued by default.
• 1.1 – introduced multiValued attribute, the default attribute value is false.
• 1.2 – introduced omitTermFreqAndPositions attribute, the default value is true for all fields, besides text fields.
• 1.3 – removed the possibility of an optional compression of fields.

Type definitions

Type definitions can be logically divided into two separate sections – the simple types and complex types. Simple types as opposed to the complex types do not have a defined filters and tokenizer.

Simple types

First thing we see in the schema.xml file after version are types definition. Each type is described as a number of attributes defining the behavior of that type. First, some attributes that describe each type and are mandatory:

• name – name of the type (required attribute).
• class – class that is responsible for the implementation. Please note that classes are delivered from standard Solr packaged will have names with ‘solr’ prefix.

Besides the two mentioned above, types can have the following optional attributes:

• sortMissingLast – attribute specifying how values in a field based on this type should be treated in case of sorting. When set to true documents without value in a field of this type will always be at the end of the results list regardless of sort order. The default attribute value is false. Attribute can be used only for types that are considered by Lucene as a string.
• sortMissingFirst – attribute specifying how values in a field based on this type should be treated in case of sorting. When set to true documents without value in a field of this type will always be at the first positions of the results list regardles of sort order. The default attribute value is false. Attribute can be used only for types that are considered by Lucene as a string.
• omitNorms – attribute specifying whether field normalization should take place.
• omitTermFreqAndPositions – attribute specifying whether term frequency and term positions should be calculated.
• indexed – attribute specifying whether the field based on this type will keep their original values.
• positionIncrementGap – attribute specifying how many position Lucene should skip.

It is worth remembering that in the default settings sortMissingLast and sortMissingFirst attributes Lucene will apply behavior of placing a document with blank field values at the beginning of the ascending sort, and at the end of the list of results for descending sorting.

One more options for simple types, but only those based on Trie*Field classes:

• precisionStep – attribute specifying the number of bits of precision. The greater the number of bits, the faster the queries based on numerical ranges. This however, also increases the size of the index, as more values are indexed. Set attribute value to 0 to disable the functionality of indexing at various precisions.

An example of a simple type defined:

<fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/>

Complex types

In addition to simple types, schema.xml file may include types consisting of a tokenizer and filters. Tokenizer is responsible for dividing the contents of the field in the tokens, while the filters are responsible for further token analysis. For example, the type that is responsible for dealing with the texts in Polish, would consist of a tokenizer in charge of the division of words based on whitespace, commas and periods. Filters for that type could be responsible for bringing generated tokens to lowercase, further division of tokens (for example on the basis of dashes), and then bringing tokens to the basic form.

Complex types, like simple types, have their name (name attribute) and the class which is responsible for implementation (class attribute). They can also be characterized by other attributes as described in the case of simple types (on the same basis). In addition, however, complex types can have a definition of tokenizer and filters to be used at the stage of indexing, and at the stage of query. As most of you know, for a given phase (indexing, or query) there can can be many filters defined but only one tokenizer. For example, just looks like a text type definition look like in the example provided with Solr:

<fieldType name="text" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
   <analyzer type="index">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
      <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
   </analyzer>
   <analyzer type="query">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
      <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
      <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
      <filter class="solr.LowerCaseFilterFactory"/>
      <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
      <filter class="solr.PorterStemFilterFactory"/>
   </analyzer>
</fieldType>

It is worth noting that there is an additional attribute for the text field type:

• autoGeneratePhraseQueries

This attribute is responsible for telling filters how to behave when dividing tokens. Some filters (such as WordDelimiterFilter) can divide tokens into a set of tokens. Setting the attribute to true (default value) will automatically generate phrase queries. This means that WordDelimiterFilter will divide the word “wi-fi” into two tokens “wi” and “fi”. With autoGeneratePhraseQueries set to true query sent to Lucene will look like "field:wi fi", while with set to false Lucene query will look like field:wi OR field:fi. However, please note, that this attribute only behaves well with tokenizers based on white spaces.

Returning to the type definition. As you can see, I gave an example which has two main sections:

<analyzer type="index">

and

<analyzer type="query">

The first section is responsible for the definition of the type, which will be used for indexing documents, the second section is responsible for the definition of type used for queries to fields based on this type. Note that if you want to use the same definitions for indexing and query phase, you can opt out of the two sections. Then our definition will look like this:

<fieldType name="text" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
   <tokenizer class="solr.WhitespaceTokenizerFactory"/>
   <filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" enablePositionIncrements="true" />
   <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
   <filter class="solr.LowerCaseFilterFactory"/>
   <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
   <filter class="solr.PorterStemFilterFactory"/>
</fieldType>

As I mentioned in the definition of each complex type there is a tokenizer and a series of filters (though not necessarily). I will not describe each filter and tokenizer available in Solr. This information is available at the following address: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

At the end I wanted to add an important thing. Starting from 1.4 Solr tokenizer does not need to be the first mechanism that deals with the analysis of the field. Solr 1.4 introduced new filters – CharFilters that operate on the field before tokenizer and transmit the result to the tokenizer. It is worth to know because it might come in useful.

Multi-dimensional types

At the end I left myself a little addition – a novelty in Solr 1.4 – multi-dimensional fields – fields consisting of a number of other fields. Generally speaking, the assumption of this type of field was simple – to store in Solr pairs of values, triples or more related data, such as georaphical point coordinates. In practice this is realized by means of dynamic fields, but let me not get into the implementation details. The sample type definition that will consist two fields:

<fieldType name="location" class="solr.PointType" dimension="2" subFieldSuffix="_d"/>

In addition to standard attributes: name and class there are two others:

• dimension – the number of dimensions (used by the class attribute solr.PointType).
• subFieldSuffix – suffix, which will be added to the dynamic fields created by that type. It is important to remember that the field based on the presented type will create three fields in the index – the actual field (for example named mylocation and two additional dynamic fields).

Field Definitions

Definitions of the fields is another section in the schema.xml file, the section, which in theory should be of interest to us the most during the design of Solr index. As a rule, we find here two kinds of field definitions:

1. Static Fields
2. Dynamic Fields

These fields are treated differently by the Solr. The first type of fields, are fields that are available under one name. Dynamic fields are fields that are available under many names – actually their name are a simple regular expression (name starting or ending with a ‘*’ sign). Please note that Solr first selects the static field, then the dynamic field. In addition, if the field name matches more than one definition, Solr will select a field with a longer name pattern.

Returning to the definition of the fields (both static and dynamic), they consist of the following attributes:

• name – the name of the field (required attribute).
• type – type of field, which is one of the pre-defined types (required attribute).
• indexed – if a field is to be indexed (set to true, if you want to search or sort on this field).
• stored – whether you want to store the original values (set to true, if we want to retrieve the original value of the field).
• omitNorms – whether you want norms to be ignored (set to true for the fields for which You will apply the full-text search).
• termVectors – set to true in the case when we want to keep so called term vectors. The default parameter value is false. Some features require setting this parameter to true (eg MoreLikeThis or FastVectorHighlighting).
• termPositions – set to true, if You want to keep term positions with the term vector. Setting to true will cause the index to expand its size.
• termOffsets – set to true, if You want to keep term offsets together with term vector. Setting to true will cause the index to expand its size.
• default – the default value to be given to the field when the document was not given any value in this field.

The following examples of definitions of fields:

<field name="id" type="string" indexed="true" stored="true" required="true" />
<field name="includes" type="text" indexed="true" stored="true" termVectors="true" termPositions="true" termOffsets="true" />
<field name="timestamp" type="date" indexed="true" stored="true" default="NOW" multiValued="false"/>
<dynamicField name="*_i" type="int" indexed="true" stored="true"/>

And finally, additional information to remember. In addition to the attributes listed above in the fields definition, we can overwrite the attributes that have been defined for type (eg whether a field is to be multiValued – the above example for a field called timestamp). Sometimes, this functionality can be useful if you need a specific field whose type is slightly different from other types (as in the example – only multiValued attribute). Of course, keep in mind the limitations imposed on the individual attributes associated with types.

CopyField section

In short, this section is responsible for copying the contents of fields to other fields. We define the field which value should be copied, and the destination field. Please note that copying takes place before the field value is analyzed. Example copyField definition:

<copyField source="category" dest="text"/>

For the sake of accuracy, occurring attributes mean:

• source – the source field,
• dest – the destination field.

Additional definitions

1. Unique key definition

The definition of a unique key that makes possible to unambiguously identify the document. Defining a unique key is not necessary, but is recommended. Sample definition:

<uniqueKey>id</uniqueKey>

2. Default search field definition

The Section is responsible for defining a default search field, which Solr use in case You have not given any field. Sample definition:

<defaultSearchField>content</defaultSearchField>

3. Default logical operator definition

This section is responsible for the definition of default logical operator that will be used. Sample definition looks as follows:

<solrQueryParser defaultOperator="OR" />

Possible values are: OR and AND.

4. Defining similarity

Finally we define the similarity that we will use. It is rather a topic for another post, but you must know that if necessary You can change the default similarity (currently in Solr trunk there are already two classes of similarity). The sample definition is as follows:

<similarity class="pl.solr.similarity.CustomSimilarity" />

A few words at the end

Information presented above should give some insight on what schema.xml file is and what correspond to the different sections in this file. Soon I will try to write what You should avoid when designing the index.