analysis – Solr.pl

Solr 4.0 and Polish language analysis

Rafał Kuć — Mon, 02 Apr 2012 21:43:51 +0000

Because Polish language analysis functionality is present in Lucene (and Solr) for some time I decided to take a look and compare it on the basis of upcoming Lucene and Solr 4.0.

Options

At the time of writing, the following options were present when it comes to analyzing Polish:

Use Stempel library (available since Solr 3.1)
Use Hunspell and Polish dictionaries (available since Solr 3.5)
Use Morfologik library (will be available in Solr 4.0, SOLR-3272).

Configuration

Lets look how to configure all the above options in Solr (please remember that all the following configuration examples are based on Solr 4.0).

Stempel

In order to add Polish stemming using Stempel library, we just need to add the following filter to our type definition:

In addition to that, you need to add lucene-analyzers-stempel-4.0.jar library and apache-solr-analysis-extras-4.0.jar library to SOLR_HOME/lib. It’s also a good idea to use solr.LowerCaseFilterFactory before Stempel filter.

Hunspell

Similar to the configuration above, to use Hunspell you need to add a new filter to your type definition. For example in the following way:

Parameters dictionary and affix are responsible for dictionary definition that we want to use. The ignoreCase parameter set to true tells Hunspell to ignore character case. You can find Hunspell dictionaries at the following URL: http://wiki.services.openoffice.org/wiki/Dictionaries.

Morfologik

Similar to the two above examples all you need to change in your schema.xml is adding a new filter, this time the following way:

The dictionary parameter tell Solr which dictionary you would like to use. You can choose the one from the following three:

MORFOLOGIK
MORFEUSZ
COMBINED

In addition to that, you need to add the following libraries to the SOLR_HOME/lib: lucene-analyzers-morfologik-4.0.jar, apache-solr-analysis-extras-4.0.jar, morfologik-fsa-1.5.2.jar, morfologik-polish-1.5.2.jar and morfologik-stemming-1.5.2.jar.

Results Comparison

Of course I wasn’t able to judge the results of analysis from the above three filters on the whole Polish language corpus and that’s why I decided to choose four work, to see the each of the filters behave. Those words are: “urodzić urodzony urodzona urodzeni” (this words are variations of the born word in Polish). The results are as follows:

Stempel

The terms I got from Stempel were the following ones:

[urodzić] [urodzo] [urodzona] [urodzeni]

Not all of them are words, but you have to remember that Stempel is a stemmer and because of that it produce stems which can be different from the actual words or their root forms. It is important to have the words we are interested in to be processed to the same tokens, which will allow to find those words by Lucene/Solr. Remembering that, I have to say, that the results of analysis using Stempel are not as good as I would like them to be. For example by searching for urodzić word you won’t be able to find documents with words like urodzona or urodzić.

Hunspell

The result of Hunspell analysis were as follows:

[urodzić, urodzić] [urodzony, urodzić] [urodzić] [urodzić, urodzony, urodzenie]

Comparing the results I got when using Hunspell to those Stempel produced we can see the difference. Our sample query for the urodzić word, would find documents with words like urodzony, urodzona oraz urodzeni, which is quite nice. You can also notice, that with three words we got more than one term on the same positions. The results I got when using Hunspell are OK and I think they should satisfy most of the users (they do satisfy me), but lets have a look on the newly introduced filter in Lucene and Solr – Morrfologik.

Morfologik

The results of Morfologik analysis were as follows:

[urodzić] [urodzony, urodzić] [urodzić] [urodzić, urodzony]

Again, if you compare those the the ones got when using Hunspell you can hardly see the difference (of course in this particular case). The only difference between Hunspell and Morfologik is the last term for which we got different results. In my opinion the results achieved with Morfologik, are satisfying.

Performance

The performance test was done in a simple manner – for each filter I’ve indexed 5 million documents, where all the text fields were based on Polish language analysis with appropriate filter (in addition to that some standard filters like stopwords, synonyms and so on). Every time the indexation was done on a clean Solr 4.0 instance. Because of using Data Import Handler I’ve sent commit every 100k documents. The index contained several fields, but the actual index structure was not crucial for the test as I indexed the same set of documents every time. Following are the test results:

[table “21” not found /]

Warning: At the time of writing, according to SOLR-3245 JIRA issue there is a problem with Hunspell performance with Polish dictionaries and Solr 4.0. I’m almost certain that this situation will be resolved by the time Solr 4.0 will be released. But right now performance of Hunspell with Polish dictionaries and Solr 4.0 may not be sufficient.

Short Summary

Despite not having performance results for Hunspell (because I don’t count the ones I have right now as correct ones) we can see that Hunspell and Morfologik are a good candidates for Polish language analysis. Looking at Morfologik we have similar performance to Stempel, but Morfologik results are better in my opinion and that will make your user more happy.

Solr filters: PatternReplaceCharFilter

Rafał Kuć — Mon, 09 May 2011 18:45:16 +0000

Continuing the overview of the filters included in Solr today we look at the PatternReplaceCharFilter.

As you might guess the task of the filter is to change the matching input stream parts that match the given regular expression.

You have the following parameters:

pattern (required) – the value to be changed (regular expressions)
replacement (default: “”) – the value that will be used as a replament for the fragment that matched the regular expression
blockDelimiters
maxBlockChars (default: 10000, must be greater than 0) – buffer used for comparison

Use examples

The use of a filter is simple – we add its definition to the type definition in schema.xml file, for example:

Poniżej przykładowe definicji dla różnych przypadków.

Below are examples of definitions for different cases.

Cut pieces of text

You just need to specify, in the pattern attribute, what we want to cut. Example:

which will suppress the content of the data elements: “#TAG”

Text fragments replacement

A similar case to the one above, but we want to convert text to another.

Changing patterns

The two above cases were trivial. What is the strength of this filter is handling regular expressions. (You use regular expressions, right?) The following example is simple – it hides all the numbers by turning them into stars. It also handles the numbers separated by hyphens, treating them as a single number.

Text Manipulation

The replacement doesn’t have to be plain text. This filter supports references which allow you to refer to parts of the matched pattern. For details, refer to the documentation of regular expressions. In the following example, all multiplied characters are replaced with a single sign.

Advanced Parameters

So far I have not mentioned the following parameters: blockDelimiters and maxBlockChars. If you look at the source code you would see that those parameters are related to the way the filter is implemented. CharFilter operates on a single character, and pattern matching requires an internal buffer to read more characters. MaxBlockChars allows you to specify the size of the buffer. You do not have to worry about it, if the pattern you defined, does not match piece of text larger than 10k characters). BlockDelimiters can further optimize filling of the buffer. It can be used if the information in the analyzed field is somehow divided into sections (eg, it is a CSV, sentences, etc.). It is a text that informs the scanner, that a new section starts, therefore, parts matched in the previous section are no longer useful.

Limits

An important limitation of the filter is that it directly manipulates the input data and does not keep information related to the original text. This means that if the filter removes a portion of the string, or add a new fragment, tokenizer will not notice that and the location of tokens in the original box will not be saved properly. You should be aware of that when using queries that operate on the relative positions of tokens or if you use highlighting.

What is schema.xml?

Rafał Kuć — Mon, 16 Aug 2010 12:05:34 +0000

One of the configuration files that describe each implementation Solr is schema.xml file. It describes one of the most important things of the implementation – the structure of the data index. The information contained in this file allow you to control how Solr behaves when indexing the data, or when making queries. Schema.xml is not only the very structure of the index, is also detailed information about data types that have a large influence on the behavior Solr, and usually are treated with neglect. This entry will try to bring some insight about schema.xml.

Schema.xml file consists of several parts:

version,
type definitions,
field definitions,
copyField section,
additional definitions.

Version

The first thing we come across in the schema.xml file is the version. This is the information for Solr how to treat some of the attributes in schema.xml file. The definition is as follows:

Please note that this is not the definition of the version from the perspective of your project. At this point Solr supports four versions of a schema.xml file:

1.0 – multiValued attribute does not exist, all fields are multivalued by default.
1.1 – introduced multiValued attribute, the default attribute value is false.
1.2 – introduced omitTermFreqAndPositions attribute, the default value is true for all fields, besides text fields.
1.3 – removed the possibility of an optional compression of fields.

Type definitions

Type definitions can be logically divided into two separate sections – the simple types and complex types. Simple types as opposed to the complex types do not have a defined filters and tokenizer.

Simple types

First thing we see in the schema.xml file after version are types definition. Each type is described as a number of attributes defining the behavior of that type. First, some attributes that describe each type and are mandatory:

name – name of the type (required attribute).
class – class that is responsible for the implementation. Please note that classes are delivered from standard Solr packaged will have names with ‘solr’ prefix.

Besides the two mentioned above, types can have the following optional attributes:

sortMissingLast – attribute specifying how values in a field based on this type should be treated in case of sorting. When set to true documents without value in a field of this type will always be at the end of the results list regardless of sort order. The default attribute value is false. Attribute can be used only for types that are considered by Lucene as a string.
sortMissingFirst – attribute specifying how values in a field based on this type should be treated in case of sorting. When set to true documents without value in a field of this type will always be at the first positions of the results list regardles of sort order. The default attribute value is false. Attribute can be used only for types that are considered by Lucene as a string.
omitNorms – attribute specifying whether field normalization should take place.
omitTermFreqAndPositions – attribute specifying whether term frequency and term positions should be calculated.
indexed – attribute specifying whether the field based on this type will keep their original values.
positionIncrementGap – attribute specifying how many position Lucene should skip.

It is worth remembering that in the default settings sortMissingLast and sortMissingFirst attributes Lucene will apply behavior of placing a document with blank field values at the beginning of the ascending sort, and at the end of the list of results for descending sorting.

One more options for simple types, but only those based on Trie*Field classes:

precisionStep – attribute specifying the number of bits of precision. The greater the number of bits, the faster the queries based on numerical ranges. This however, also increases the size of the index, as more values are indexed. Set attribute value to 0 to disable the functionality of indexing at various precisions.

An example of a simple type defined:

Complex types

In addition to simple types, schema.xml file may include types consisting of a tokenizer and filters. Tokenizer is responsible for dividing the contents of the field in the tokens, while the filters are responsible for further token analysis. For example, the type that is responsible for dealing with the texts in Polish, would consist of a tokenizer in charge of the division of words based on whitespace, commas and periods. Filters for that type could be responsible for bringing generated tokens to lowercase, further division of tokens (for example on the basis of dashes), and then bringing tokens to the basic form.

Complex types, like simple types, have their name (name attribute) and the class which is responsible for implementation (class attribute). They can also be characterized by other attributes as described in the case of simple types (on the same basis). In addition, however, complex types can have a definition of tokenizer and filters to be used at the stage of indexing, and at the stage of query. As most of you know, for a given phase (indexing, or query) there can can be many filters defined but only one tokenizer. For example, just looks like a text type definition look like in the example provided with Solr:

It is worth noting that there is an additional attribute for the text field type:

autoGeneratePhraseQueries

This attribute is responsible for telling filters how to behave when dividing tokens. Some filters (such as WordDelimiterFilter) can divide tokens into a set of tokens. Setting the attribute to true (default value) will automatically generate phrase queries. This means that WordDelimiterFilter will divide the word “wi-fi” into two tokens “wi” and “fi”. With autoGeneratePhraseQueries set to true query sent to Lucene will look like "field:wi fi", while with set to false Lucene query will look like field:wi OR field:fi. However, please note, that this attribute only behaves well with tokenizers based on white spaces.

Returning to the type definition. As you can see, I gave an example which has two main sections:

and

The first section is responsible for the definition of the type, which will be used for indexing documents, the second section is responsible for the definition of type used for queries to fields based on this type. Note that if you want to use the same definitions for indexing and query phase, you can opt out of the two sections. Then our definition will look like this:

As I mentioned in the definition of each complex type there is a tokenizer and a series of filters (though not necessarily). I will not describe each filter and tokenizer available in Solr. This information is available at the following address: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters.

At the end I wanted to add an important thing. Starting from 1.4 Solr tokenizer does not need to be the first mechanism that deals with the analysis of the field. Solr 1.4 introduced new filters – CharFilters that operate on the field before tokenizer and transmit the result to the tokenizer. It is worth to know because it might come in useful.

Multi-dimensional types

At the end I left myself a little addition – a novelty in Solr 1.4 – multi-dimensional fields – fields consisting of a number of other fields. Generally speaking, the assumption of this type of field was simple – to store in Solr pairs of values, triples or more related data, such as georaphical point coordinates. In practice this is realized by means of dynamic fields, but let me not get into the implementation details. The sample type definition that will consist two fields:

In addition to standard attributes: name and class there are two others:

dimension – the number of dimensions (used by the class attribute solr.PointType).
subFieldSuffix – suffix, which will be added to the dynamic fields created by that type. It is important to remember that the field based on the presented type will create three fields in the index – the actual field (for example named mylocation and two additional dynamic fields).

Field Definitions

Definitions of the fields is another section in the schema.xml file, the section, which in theory should be of interest to us the most during the design of Solr index. As a rule, we find here two kinds of field definitions:

Static Fields
Dynamic Fields

These fields are treated differently by the Solr. The first type of fields, are fields that are available under one name. Dynamic fields are fields that are available under many names – actually their name are a simple regular expression (name starting or ending with a ‘*’ sign). Please note that Solr first selects the static field, then the dynamic field. In addition, if the field name matches more than one definition, Solr will select a field with a longer name pattern.

Returning to the definition of the fields (both static and dynamic), they consist of the following attributes:

name – the name of the field (required attribute).
type – type of field, which is one of the pre-defined types (required attribute).
indexed – if a field is to be indexed (set to true, if you want to search or sort on this field).
stored – whether you want to store the original values (set to true, if we want to retrieve the original value of the field).
omitNorms – whether you want norms to be ignored (set to true for the fields for which You will apply the full-text search).
termVectors – set to true in the case when we want to keep so called term vectors. The default parameter value is false. Some features require setting this parameter to true (eg MoreLikeThis or FastVectorHighlighting).
termPositions – set to true, if You want to keep term positions with the term vector. Setting to true will cause the index to expand its size.
termOffsets – set to true, if You want to keep term offsets together with term vector. Setting to true will cause the index to expand its size.
default – the default value to be given to the field when the document was not given any value in this field.

The following examples of definitions of fields:

And finally, additional information to remember. In addition to the attributes listed above in the fields definition, we can overwrite the attributes that have been defined for type (eg whether a field is to be multiValued – the above example for a field called timestamp). Sometimes, this functionality can be useful if you need a specific field whose type is slightly different from other types (as in the example – only multiValued attribute). Of course, keep in mind the limitations imposed on the individual attributes associated with types.

CopyField section

In short, this section is responsible for copying the contents of fields to other fields. We define the field which value should be copied, and the destination field. Please note that copying takes place before the field value is analyzed. Example copyField definition:

For the sake of accuracy, occurring attributes mean:

source – the source field,
dest – the destination field.

Additional definitions

1. Unique key definition

The definition of a unique key that makes possible to unambiguously identify the document. Defining a unique key is not necessary, but is recommended. Sample definition:

id

2. Default search field definition

The Section is responsible for defining a default search field, which Solr use in case You have not given any field. Sample definition:

content

3. Default logical operator definition

This section is responsible for the definition of default logical operator that will be used. Sample definition looks as follows:

Possible values are: OR and AND.

4. Defining similarity

Finally we define the similarity that we will use. It is rather a topic for another post, but you must know that if necessary You can change the default similarity (currently in Solr trunk there are already two classes of similarity). The sample definition is as follows:

A few words at the end

Information presented above should give some insight on what schema.xml file is and what correspond to the different sections in this file. Soon I will try to write what You should avoid when designing the index.