field – Solr.pl

Category Routed Aliases

Rafał Kuć — Mon, 21 Oct 2019 14:12:01 +0000

Through the lifetime of Solr we were given the possibility to work with cores, then collections and finally aliases – the alternative names for collections. Aliases allow the user to give your collection a new, virtual name and group multiple collections under that single virtual name. This allows isolation of the real collection name from the name that the client application is using. That allows changing the collection in the background without the need of bringing down the whole cluster and make your application or product unavailable. In Solr we have the option to use two aliases groups:

standard aliases that group collections under a virtual name,
routed aliases that route your requests.

The routed aliases can be further divided into two categories – time routed aliases and category routed aliases. We will be looking into the second category in this blog post.

Why Do I Need Category Routed Aliases?

The problem of the standard aliases is the indexing limitation. If you will create an alias that covers two collections, we will not be able to really control where the data will be put – i.e., in Solr 8.2 the first collection from the alias will be used. Let me demonstrate you how that works.

First, let’s create two collections using API v2. We will start with the first collection:

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/v2/c' -d '{ 
 "create" : {
  "name" : "test_1",
  "numShards" : 1,
  "replicationFactor" : 1
 }
}'

And then the second collection:

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/v2/c' -d '{ 
 "create" : {
  "name" : "test_2",
  "numShards" : 1,
  "replicationFactor" : 1
 }
}'

So we have two collections one called test_1 and the second one called test_2. We should create an alias grouping those two collections called test. We can do that using the following command:

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/v2/c' -d '{ 
 "create-alias" : {
  "name" : "test",
  "collections" : [ "test_1", "test_2" ]
 }
}'

Now let’s index the document by using the following command:

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test/update?commit=true' -d '[
 {
  "id" : 1,
  "name" : "Test indexing"
 }
]'

That command will be successful in Solr 8.2 and the document will be put into the test_1 collection, you can check it yourself

And of course this is not the only problem. If you would like to physically separate the data of multiple tenants you need to prepare your application that handles sending data to Solr to do that. You could still use the alias for reading part though.

To overcome that problem we can use category routed aliases and let SolrCloud do the work.

Category Routed Aliases

The idea behind the category routed aliases is to manage the alias and the collections that are grouped using it by using a value inside a certain, defined field. Given that we basically get partitioning based on a certain field – for example on the company name. That way, each company that we store index data for will be able to have their own collection and we will not have to worry about that manually during indexing or querying time.

Simplified view over category routed aliases

Creating Category Routed Aliases

The process of creation of the category routed aliases is slightly different from what we did with the standard aliases. We do not start with the creation of the collections – those will be created automatically. Instead we are starting with alias creation.

Let’s create our first category based alias by using the following command:

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/v2/c' -d '{ 
 "create-alias" : {
  "name" : "test_cra_company",
  "router" : {
   "name" : "category",
   "field" : "company_name",
   "maxCardinality" : 2
  },
  "create-collection" : {
   "numShards" : "1",
   "replicationFactor" : "1",
   "config" : "_default"
  }
 }
}'

Let’s stop for a minute here. We’ve used the create-alias command to create a new alias called test_cra_company. We provided a few properties here. First of all you ca see the router object that includes three properties:

name – the name of the router that can be used. For now Solr supports time and category. The time creates time-routed alias, while the category created the category based alias.
field – the name of the field that will be used for routing.
maxCardinality – the maximum number of unique values that the field can take to avoid creating too many collections.

Because the routed aliases will create collections in the background we need to provide the properties for the collections. We do that using the create-collection object in our request. We provided the number of shards, the replication factor and the name of the configuration that will be used.

In the examples, for simplicity we are using the default, data-driven schema. This shouldn’t be used for production when using time or category routed aliases.

How Does the Updates in the Routed Aliases Work Internally?

When Solr processes an update request an UpdateRequestProcessor is initialized. In the SolrCloud, like in our case, the DistributedUpdateProcessor is initialized, at least that is the case when we use standard aliases. When the time or category routed aliases are used the RoutedAliasUpdateProcessor is used before the actual DistributedUpdateProcessor. It is done automatically and doesn’t have to be done manually. Then the RoutedAliasUpdateProcessor is responsible for routing the data to the appropriate collection and as we saw if we don’t have any kind of routed aliases Solr will just use the first collection on the alias list and use it to perform the update operation.

What you should also know is that there will be a special place-holder collection created called test_cra_company__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA__TEMP that will be eventually deleted and the collections that are created in the background will be named test_cra_company__CRA__, so for example test_cra_company__CRA__company1 if the company_name field will have the value of the company1. This provides naming limitations, but we will talk about it later in the blog post.

Indexing Data With Routed Aliases

Let’s now see the alias in action. To do that we will index two documents using the following commands:

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test_cra_company/update?commit=true' -d '[
 {
  "id" : 1,
  "name" : "Test indexing",
  "company_name" : "company1"
 }
]'

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test_cra_company/update?commit=true' -d '[
 {
  "id" : 2,
  "name" : "Test indexing",
  "company_name" : "company2"
 }
]'

Remember the maxCardinality property that we’ve set to the value of 2? If we will try to use a third company name, for example using the following command:

curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test_cra_company/update?commit=true' -d '[
 {
  "id" : 3,
  "name" : "Test indexing",
  "company_name" : "company3"
 }
]'

Solr will get back to us with the following error:

{
  "responseHeader":{
    "status":400,
    "QTime":1},
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"Max cardinality 2 reached for Category Routed Alias: test_cra_company",
    "code":400}}

Queries

We can also try to search on our data, for example using a match all query:

http://localhost:8983/solr/test_cra_company/select?q=*:*

Solr will return both of the indexed documents:

{
  "responseHeader": {
    "zkConnected": true,
    "status": 0,
    "QTime": 12,
    "params": {
      "q": "*:*"
    }
  },
  "response": {
    "numFound": 2,
    "start": 0,
    "maxScore": 1,
    "docs": [
      {
        "id": "1",
        "name": [
          "Test indexing"
        ],
        "company_name": [
          "company1"
        ],
        "_version_": 1647547697554522000
      },
      {
        "id": "2",
        "name": [
          "Test indexing"
        ],
        "company_name": [
          "company2"
        ],
        "_version_": 1647547721335177200
      }
    ]
  }
}

We can also filter our query as we would usually do:

http://localhost:8983/solr/test_cra_company/select?q=*:*&fq=company_name:company2

And the response will contain only a single document, which is expected:

{
  "responseHeader": {
    "zkConnected": true,
    "status": 0,
    "QTime": 25,
    "params": {
      "q": "*:*",
      "fq": "company_name:company2"
    }
  },
  "response": {
    "numFound": 1,
    "start": 0,
    "maxScore": 1,
    "docs": [
      {
        "id": "2",
        "name": [
          "Test indexing"
        ],
        "company_name": [
          "company2"
        ],
        "_version_": 1647547721335177200
      }
    ]
  }
}

As we can see, everything works as intended.

The collections that were created during our simple test looks as follows:

Limitations

There are a few limitations when it comes to the routed aliases, especially the category routed aliases.

The first limitation is naming. The values of our field that we use for routing needs to be ASCII based. Otherwise Solr will not be able to use the name for the collection name and we will run into issues. You should remember about that.

The second thing is deletion of the alias or collections that are handled. There is no automated way of doing that, so you can’t easily remove a category. So the procedure of removing a category would be:

Ensuring that there will not be more document with the category that we want to remove.
Modifying the alias definition in Zookeeper by removing the collection that is responsible for handling the data of the category that we want to remove.
Delete the collection using Solr API – you need to remove it from the alias first, otherwise Solr will fail to delete it.

Currently (in Solr 8.2 as of writing this post) the update is distributed to an appropriate collection based on the value of the category, while the query is run against all the collections defined by the alias. Improving the query time execution is mentioned as one of the possible improvements for the routed aliases feature in Solr.

Finally remember that the collection creation takes time, usually up to 3 seconds, depending on how loaded your SolrCloud cluster is. Take that in mind when designing and implementing a system that will use Solr with routed aliases. Your system needs to be able to handle a longer delay when a first value will appear in the document leading to collection creation.

Summary

As you can see the category routed aliases provide a very nice and convenient way of automatically creating collections based on the value of a field. So if we need that and we want Solr to take care of that for us – this is one of the ways to go, especially in the newer Solr versions. Is the feature perfect – no. Is there a room for improvement – yes and there are already possible improvements that can be done and are mentioned in the official documentation of Solr. Hopefully we will see them in the next Solr versions.

Solr 4.1: Stored fields compression

Rafał Kuć — Mon, 19 Nov 2012 22:53:02 +0000

Despite the fact that Lucene and Solr 4.0 is still very fresh we decided that its time to take a look at the changes in the approaching 4.1 version. One of those changes will be stored fields compression to decrease the size of the index size, when we use such fields. Let’s take a look and see how that works.

Some theory

In case our index consists of many stored fields they can be consuming most space when compared to other information in the index. How to know how much space the stored fields take ? Its easy – just go to the directory that holds your index and check how much space the files with .fdt extension takes. Despite the fact, that stored fields don’t influence the search performance directly, your I/O subsystem and its cache can be forced to work much harder because of the larger amount of data on the disk. Because of that your queries can be executed longer and you may need more time to index your data.

With the incoming release of Lucene 4.1 stored fields will be compressed with the use of LZ4 algorithm (http://code.google.com/p/lz4/), which should decrease the size of the index when we use high number of stored fields, but also shouldn’t be CPU demanding when it comes to compression and decompression.

Test data

For the discussed functionality tests we’ve used Polish Wikipedia articles data, from 2012.11.10 (http://dumps.wikimedia.org/plwiki/20121110/plwiki-20121110-pages-articles.xml.bz2). Unpacked XML file was about 4.7GB on disk.

Index structure

We’ve used the following index structure to index the above data:

DIH configuration

We’ve used the following DIH configuration in order to index Wikipedia data:

Indexing time

In both cases indexing time was very similar, for the same amount of documents (there was 1.301.394 documents after indexing). In case of Solr 4.0 indexing took 14 minutes and 33 seconds. In case of Solr 4.1 indexing took 14 minutes and 43 seconds. As you can see Solr 4.1 was slightly slower, but because I made the tests on my laptop, we can assume that the indexing performance is very similar.

Index size

The size of the index is what interest us the most in this case. In case of Solr 4.0 the index created with the Wikipedia data was about 5.1GB – 5.464.809.863 bytes. In case of Solr 4.1 the index weighted approximately 3.24GB – 3.480.457.399 bytes. So when comparing index created by Solr 4.0 to the one created by Solr 4.1 we got about 35% smaller index.

Wrapping up

You can clearly see, that the gain from compressing stored fields is quite big. Despite the fact that we need additional CPU cycles for compression handling we benefit from less I/O subsystem pressure and we can be sure that the gain will be greater than the loss of a few CPU cycles. After seeing this I’m not wondering with the stored fields compressing is turned on by default in Lucene 4.1 and thus in Solr 4.1 too. However if you would like to turn off that behavior you’ll need to implement your own codec – one that doesn’t use compression, at least for now. However you don’t need to fork Lucene code to do that and this again shows how powerful the flexible indexing is.

Solr 3.6: CurrencyField

Rafał Kuć — Mon, 19 Mar 2012 22:43:17 +0000

The incoming Solr 3.6 will bring us an interesting feature in the form of currency handling. Some may ask “What for ? We can just use float and we can use it for currency handling”. So let’s take a look at solr.CurrencyField which will be presented in Solr 3.6.

Configuration

Lets start with the field type configuration, which is quite typical when it comes to Solr. To the schema.xml file we need to add another field type declaration, like the following:

In the above configuration we see two additional attributes (compared to the usual type definition) that define currencyField behavior. First we see the defaultCurrency attribute, which define the default currency for a given field. It defines in which form data will be written into the index (change of this attribute value requires reindexing). The second attribute, the currencyConfig defines an exchange file. Its worth to remember that using the second parameter only makes sense with the default exchange provider (FileExchangeRateProvided) distributed with Solr. But let’s take a look at the currencyExchange.xml file:

Exchange configuration file for FileExchangeRateProvider

So now, lets look at the contents of the currencyExchange.xml file which provides the exchange rates for the default exchange provider:

As you can see the structure is quite simple, we define the base currency (from), output currency (to) and the exchange rate (rate). So it’s quite simple

Data indexing

In order to index the data with the defined currencyField we should specify the value and the currency prefixed with a comma character. For example:

21.99,EUR

Querying

Querying is more or less like index. You have to pass two informations to Solr – the value and currency. Let’s look at the two example filters using currency, the first one with a simple term query and the second one a range query:

fq=price:29.99,PLN

fq=price:[10.00 TO 29.99,EUR]

As you can see, after setting the value (or range) we have to specify the comma character and the currency we are interested in. Whats more, we are not bound to the default currency – we can query other currencies as long as we have the exchange rate defined. This means that Solr will make the exchange calculation for us. Please remember, that for now, the values returned in the results will only contain the default currency and there is no way to change that behavior.

Your own currency provider

In addition to the default exchange provider, Solr will enable us to plug in our own implementation. In order to do that we need to develop a class that implements org.apache.solr.schema.ExchangeRateProvider interface and let Solr know by passing the class name as the value of providerClass attribute defined for our currency type. Assuming that we have the class pl.solr.schema.DynamicRateExchangeProvider that implements the mentioned interface and we would like to use that class, our field type definition would look something like that:

I like this feature as we are not bound to XML exchange file, but we can develop our own implementation which for example use webservice to dynamically read data from some external source.

What’s left to implement ?

When the post was published you could not use range faceting on fields based on CurrencyField type.

To sum up

In my opinion the CurrencyField is a functionality worth waiting for. Instead of calculating exchange in our application we can let Solr do that for us. In addition to that we will be able to plug in our own implementations of exchange providers, which will enable us to write connectors to different systems and just watch how Solr does all the work for us

Solr 4.0: new fl parameter functionalities – first look

Rafał Kuć — Tue, 22 Nov 2011 20:54:35 +0000

In connection with the work of slowly upcoming release of Apache Solr version 4.0 I thought that it is time to bring some light on the functionalities that you will get into your own hands with the release of Apache Solr 4.0. The first change we will look at is a simple, but albeit useful functionality called pseudo fields, together with additional features related to the fl parameter.

Lets begin

The Apache Solr 4.0 has changed slightly how fl parameter can be used. In 4.0 this parameters can be added to the query multiple times and Solr will take all the values into consideration. Sometimes it will be useful, at least in my case.

Custom field names

In Solr 4.0 you will be able to rename fields that are returned in the results. Imagine that, depending on the query we would like rename fields like price_en, price_pl or price_fr to price. In Solr 4.0, we can do this by placing the following query:

fl=price:price_pl

This will cause the field price_pl to be returned as price.

All fields with a common name start

If we want Solr to return the all fields whose name starts with the word price (useful for dynamic fields) all we will need to do is add the following parameter to the query:

fl=price*

Returning function values

The last functionality, which we will look at today, is the ability add the result of the function, as a field in the documents returned by Solr. Thus, in Solr 4.0 will have the option to add such values as sum of prices, or calculated distance between two points. Quite useful. To use this functionality You will have to add the appropriate function call to the fl parameter for example:

fl=*,stock:sum(stockMain,stockShop)

The above will result in Solr returning all the fields for the document (value *) and a field named stock, which will be the sum of two fields: stockShop and stockMain.

A few words at the end

In addition to the new features mentioned above, there is one more thing that is connected to the parameter fl – the DocTransformer. However I decided to leave it for a separate blog post.

Quick look – FieldCollapsing

Rafał Kuć — Mon, 20 Sep 2010 12:12:39 +0000

FieldCollapsing, or in other words grouping of search results has just been commited to the svn repository. I decided to take a look at this functionality and see how it works.

I want to begin with brief information – FieldCollapsing is only available in version 4.0 of Solr, which is a development version of Solr project, and it’s rather unlikely to be transfered to version 3.X.

FieldCollapsing – what is it ?

Imagine that our index contains information about companies from different cities. We want to show our users one (or, for example two or three) companies in each city, of course, the companies that meet the search criteria. How to do that – just use the FieldCollapsing mechanism. It allows the returned results to be grouped based on field contents. The search results can be grouped into a single document, or a fixed quantity of documents.

Parameters

Similarly, as with most features available in Solr, the behavior of FieldCollapsing mechanism can be configured through a number of parameters, here they are:

group – setting this parameter to true enables FieldCollapsing mechanism. The default value is false.
group.field – this parameter determines on the contents of what field grouping is going to take place.
group.func – definition of function, based on the outcome of which grouping will be made.
group.limit – the number of documents returned in each group. The default is 1.
group.sort – parameter specifying how to sort the documents in groups. The default value is the value score desc.

It is worth noting that the rows parameter passed to the query will determine the number of groups to be returned in search results not the amount of individual documents. Sort parameter behaviour is also changed. This parameter will tell Solr how to sort groups not individual documents. Groups wil be sorted based on the content of fields of the first documents in every group.

Search Results

Search results are different from those to which we are accustomed. They are grouped according to the parameters that we have passed. The main element of the search results are no longer documents – when we use FieldCollapsing the main search result element is a group of documents. Within the groups the documents are shown (their number is defined by group.limit parameter). For example, making the following query:

http://localhost:8983/solr/select/?q=*:*&group=true&group.field=instock&indent=true

to Solr which index was created by indexing all documents in XML format from a catalog exampledocs will result in getting the following response:




  0
  0
  
    inStock
    true
    true
    *:*
  


  
    19
    
     
        T
        
          
            electronicshard drive
            7200RPM, 8MB cache, IDE Ultra ATA-133NoiseGuard, SilentSeek technology, Fluid Dynamic Bearing (FDB) motor
            SP2514N
            true
            Samsung Electronics Co. Ltd.
            2006-02-13T15:26:37Z
            Samsung SpinPoint P120 SP2514N - hard drive - 250 GB - ATA-133
            6
            92.0
            45.17614,-93.87341
            45.17614
            -93.87341
            45.17614,-93.87341
          
        
      
      
        F
        
          
            electronicsconnector
            car power adapter, white
            F8V7067-APL-KIT
            false
            Belkin
            2005-08-01T16:30:25Z
            Belkin Mobile Power Cord for iPod w/ Dock
            1
            19.95
            45.17614,-93.87341
            45.17614
            -93.87341
            45.17614,-93.87341
            4.0

At the end

An interesting feature that will certainly find use in some systems. However, please note that this functionality will be further developed. So far there is no support for distributed search and for grouping on multivalued fields. At this time there’s no point of a performance testing, first because of the changes that will come to the mechanism, and secondly because of the fact that this is Lucene and Solr 4.0 which are both in development. However, I will be definitely watching how this functionality evolves

What is schema.xml?

Rafał Kuć — Mon, 16 Aug 2010 12:05:34 +0000

One of the configuration files that describe each implementation Solr is schema.xml file. It describes one of the most important things of the implementation – the structure of the data index. The information contained in this file allow you to control how Solr behaves when indexing the data, or when making queries. Schema.xml is not only the very structure of the index, is also detailed information about data types that have a large influence on the behavior Solr, and usually are treated with neglect. This entry will try to bring some insight about schema.xml.

Schema.xml file consists of several parts:

version,
type definitions,
field definitions,
copyField section,
additional definitions.

Version

The first thing we come across in the schema.xml file is the version. This is the information for Solr how to treat some of the attributes in schema.xml file. The definition is as follows:

Please note that this is not the definition of the version from the perspective of your project. At this point Solr supports four versions of a schema.xml file:

1.0 – multiValued attribute does not exist, all fields are multivalued by default.
1.1 – introduced multiValued attribute, the default attribute value is false.
1.2 – introduced omitTermFreqAndPositions attribute, the default value is true for all fields, besides text fields.
1.3 – removed the possibility of an optional compression of fields.

Type definitions

Type definitions can be logically divided into two separate sections – the simple types and complex types. Simple types as opposed to the complex types do not have a defined filters and tokenizer.

Simple types

First thing we see in the schema.xml file after version are types definition. Each type is described as a number of attributes defining the behavior of that type. First, some attributes that describe each type and are mandatory:

name – name of the type (required attribute).
class – class that is responsible for the implementation. Please note that classes are delivered from standard Solr packaged will have names with ‘solr’ prefix.

Besides the two mentioned above, types can have the following optional attributes:

sortMissingLast – attribute specifying how values in a field based on this type should be treated in case of sorting. When set to true documents without value in a field of this type will always be at the end of the results list regardless of sort order. The default attribute value is false. Attribute can be used only for types that are considered by Lucene as a string.
sortMissingFirst – attribute specifying how values in a field based on this type should be treated in case of sorting. When set to true documents without value in a field of this type will always be at the first positions of the results list regardles of sort order. The default attribute value is false. Attribute can be used only for types that are considered by Lucene as a string.
omitNorms – attribute specifying whether field normalization should take place.
omitTermFreqAndPositions – attribute specifying whether term frequency and term positions should be calculated.
indexed – attribute specifying whether the field based on this type will keep their original values.
positionIncrementGap – attribute specifying how many position Lucene should skip.

It is worth remembering that in the default settings sortMissingLast and sortMissingFirst attributes Lucene will apply behavior of placing a document with blank field values at the beginning of the ascending sort, and at the end of the list of results for descending sorting.

One more options for simple types, but only those based on Trie*Field classes:

precisionStep – attribute specifying the number of bits of precision. The greater the number of bits, the faster the queries based on numerical ranges. This however, also increases the size of the index, as more values are indexed. Set attribute value to 0 to disable the functionality of indexing at various precisions.

An example of a simple type defined:

Complex types

In addition to simple types, schema.xml file may include types consisting of a tokenizer and filters. Tokenizer is responsible for dividing the contents of the field in the tokens, while the filters are responsible for further token analysis. For example, the type that is responsible for dealing with the texts in Polish, would consist of a tokenizer in charge of the division of words based on whitespace, commas and periods. Filters for that type could be responsible for bringing generated tokens to lowercase, further division of tokens (for example on the basis of dashes), and then bringing tokens to the basic form.

Complex types, like simple types, have their name (name attribute) and the class which is responsible for implementation (class attribute). They can also be characterized by other attributes as described in the case of simple types (on the same basis). In addition, however, complex types can have a definition of tokenizer and filters to be used at the stage of indexing, and at the stage of query. As most of you know, for a given phase (indexing, or query) there can can be many filters defined but only one tokenizer. For example, just looks like a text type definition look like in the example provided with Solr:

It is worth noting that there is an additional attribute for the text field type:

autoGeneratePhraseQueries

This attribute is responsible for telling filters how to behave when dividing tokens. Some filters (such as WordDelimiterFilter) can divide tokens into a set of tokens. Setting the attribute to true (default value) will automatically generate phrase queries. This means that WordDelimiterFilter will divide the word “wi-fi” into two tokens “wi” and “fi”. With autoGeneratePhraseQueries set to true query sent to Lucene will look like "field:wi fi", while with set to false Lucene query will look like field:wi OR field:fi. However, please note, that this attribute only behaves well with tokenizers based on white spaces.

Returning to the type definition. As you can see, I gave an example which has two main sections:

and

The first section is responsible for the definition of the type, which will be used for indexing documents, the second section is responsible for the definition of type used for queries to fields based on this type. Note that if you want to use the same definitions for indexing and query phase, you can opt out of the two sections. Then our definition will look like this:

As I mentioned in the definition of each complex type there is a tokenizer and a series of filters (though not necessarily). I will not describe each filter and tokenizer available in Solr. This information is available at the following address: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

At the end I wanted to add an important thing. Starting from 1.4 Solr tokenizer does not need to be the first mechanism that deals with the analysis of the field. Solr 1.4 introduced new filters – CharFilters that operate on the field before tokenizer and transmit the result to the tokenizer. It is worth to know because it might come in useful.

Multi-dimensional types

At the end I left myself a little addition – a novelty in Solr 1.4 – multi-dimensional fields – fields consisting of a number of other fields. Generally speaking, the assumption of this type of field was simple – to store in Solr pairs of values, triples or more related data, such as georaphical point coordinates. In practice this is realized by means of dynamic fields, but let me not get into the implementation details. The sample type definition that will consist two fields:

In addition to standard attributes: name and class there are two others:

dimension – the number of dimensions (used by the class attribute solr.PointType).
subFieldSuffix – suffix, which will be added to the dynamic fields created by that type. It is important to remember that the field based on the presented type will create three fields in the index – the actual field (for example named mylocation and two additional dynamic fields).

Field Definitions

Definitions of the fields is another section in the schema.xml file, the section, which in theory should be of interest to us the most during the design of Solr index. As a rule, we find here two kinds of field definitions:

Static Fields
Dynamic Fields

These fields are treated differently by the Solr. The first type of fields, are fields that are available under one name. Dynamic fields are fields that are available under many names – actually their name are a simple regular expression (name starting or ending with a ‘*’ sign). Please note that Solr first selects the static field, then the dynamic field. In addition, if the field name matches more than one definition, Solr will select a field with a longer name pattern.

Returning to the definition of the fields (both static and dynamic), they consist of the following attributes:

name – the name of the field (required attribute).
type – type of field, which is one of the pre-defined types (required attribute).
indexed – if a field is to be indexed (set to true, if you want to search or sort on this field).
stored – whether you want to store the original values (set to true, if we want to retrieve the original value of the field).
omitNorms – whether you want norms to be ignored (set to true for the fields for which You will apply the full-text search).
termVectors – set to true in the case when we want to keep so called term vectors. The default parameter value is false. Some features require setting this parameter to true (eg MoreLikeThis or FastVectorHighlighting).
termPositions – set to true, if You want to keep term positions with the term vector. Setting to true will cause the index to expand its size.
termOffsets – set to true, if You want to keep term offsets together with term vector. Setting to true will cause the index to expand its size.
default – the default value to be given to the field when the document was not given any value in this field.

The following examples of definitions of fields:

And finally, additional information to remember. In addition to the attributes listed above in the fields definition, we can overwrite the attributes that have been defined for type (eg whether a field is to be multiValued – the above example for a field called timestamp). Sometimes, this functionality can be useful if you need a specific field whose type is slightly different from other types (as in the example – only multiValued attribute). Of course, keep in mind the limitations imposed on the individual attributes associated with types.

CopyField section

In short, this section is responsible for copying the contents of fields to other fields. We define the field which value should be copied, and the destination field. Please note that copying takes place before the field value is analyzed. Example copyField definition:

For the sake of accuracy, occurring attributes mean:

source – the source field,
dest – the destination field.

Additional definitions

1. Unique key definition

The definition of a unique key that makes possible to unambiguously identify the document. Defining a unique key is not necessary, but is recommended. Sample definition:

id

2. Default search field definition

The Section is responsible for defining a default search field, which Solr use in case You have not given any field. Sample definition:

content

3. Default logical operator definition

This section is responsible for the definition of default logical operator that will be used. Sample definition looks as follows:

Possible values are: OR and AND.

4. Defining similarity

Finally we define the similarity that we will use. It is rather a topic for another post, but you must know that if necessary You can change the default similarity (currently in Solr trunk there are already two classes of similarity). The sample definition is as follows:

A few words at the end

Information presented above should give some insight on what schema.xml file is and what correspond to the different sections in this file. Soon I will try to write what You should avoid when designing the index.