As you might guess the task of the filter is to change the matching input stream parts that match the given regular expression.

You have the following parameters:

• pattern (required) – the value to be changed (regular expressions)
• replacement (default: “”) – the value that will be used as a replament for the fragment that matched the regular expression
• blockDelimiters
• maxBlockChars (default: 10000, must be greater than 0) – buffer used for comparison

Use examples

The use of a filter is simple – we add its definition to the type definition in schema.xml file, for example:

<fieldType name="textCharNorm" class="solr.TextField">
  <analyzer>
    <charFilter class="solr.PatternReplaceCharFilterFactory" …/>
    <charFilter class="solr.MappingCharFilterFactory" mapping="mapping-ISOLatin1Accent.txt"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
  </analyzer>
</fieldType>

Poniżej przykładowe definicji dla różnych przypadków.

Below are examples of definitions for different cases.

Cut pieces of text

You just need to specify, in the pattern attribute, what we want to cut. Example:

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="#TAG" />

which will suppress the content of the data elements: “#TAG”

Text fragments replacement

A similar case to the one above, but we want to convert text to another.

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="#TAG" replacement="[CENZORED]"/>

Changing patterns

The two above cases were trivial. What is the strength of this filter is handling regular expressions. (You use regular expressions, right?) The following example is simple – it hides all the numbers by turning them into stars. It also handles the numbers separated by hyphens, treating them as a single number.

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="(\\d+-*\\d+)+" replacement="*"/>

Text Manipulation

The replacement doesn’t have to be plain text. This filter supports references which allow you to refer to parts of the matched pattern. For details, refer to the documentation of regular expressions. In the following example, all multiplied characters are replaced with a single sign.

<charFilter class="solr.PatternReplaceCharFilterFactory" pattern="(.)\\1" replacement="$1"/>

Advanced Parameters

So far I have not mentioned the following parameters: blockDelimiters and maxBlockChars. If you look at the source code you would see that those parameters are related to the way the filter is implemented. CharFilter operates on a single character, and pattern matching requires an internal buffer to read more characters. MaxBlockChars allows you to specify the size of the buffer. You do not have to worry about it, if the pattern you defined, does not match piece of text larger than 10k characters). BlockDelimiters can further optimize filling of the buffer. It can be used if the information in the analyzed field is somehow divided into sections (eg, it is a CSV, sentences, etc.). It is a text that informs the scanner, that a new section starts, therefore, parts matched in the previous section are no longer useful.

Limits

An important limitation of the filter is that it directly manipulates the input data and does not keep information related to the original text. This means that if the filter removes a portion of the string, or add a new fragment, tokenizer will not notice that and the location of tokens in the original box will not be saved properly. You should be aware of that when using queries that operate on the relative positions of tokens or if you use highlighting.

At the beginning I wanted to point out that the following examples are not all mistakes that can be made, this are only examples of what should be considered when using Solr.

1. I`m sure I`ll need it

As in case of schema.xml file, I suggest minimalism in solrconfig.xml too. If we know that we will only use response in JSON format there is no need to configure additional response formats. I often come across situations when a person setup Solr with all the possible handlers, response writers and a number of additional features despite the fact that they do not even know what some of them do. Although the use of memory for the standard configuration elements is not large, remember that maintenance of a minimalist solrconfig.xml file is definitely easier than that which is blown to the borders of impossibility.

2. Why should I cache ?

Extreme case, but true. I was once asked if the cache is necessary if the application using Solr will use the cache on it`s own side. My answer was yes, of course. People who do not know Solr, imagine it’s cache, as the next, sometimes unnecessary, level of retention of search results. However, please note that in addition to cache mechanism based on HTTP Solr has its own cache – to be more accurate – Solr have more than one type of cache. When we adjust Solr cache to our needs we monitor test servers – how hits distribute – is the cache too big or maybe too small. Please note that Solr cache configuration is not a one-time process, and from time to time we must take a look at the statistics and possibly update our configuration.

3. Because you need to know how to warm up

Solr starts a few minutes and replication lasts forever, even though the index is relatively small. So the question arises – why? Look at solrconfig.xml file and we have a winner – a huge number of warming queries, those that run at startup and those that run during the warm-up of new searcher after data replication. We must remember not to overdo the number of queries, because we achieve counterproductive effect – despite potentially good warm-up Solr will run poorly or not at all.

4. I`ll save it in the configuration

Sometimes I meet the approach, where the person using the Solr would like to save all the query parameters, even those that are changing the configuration files. This approach leads to many handlers definitions, which barely differ from each other – the difference is in set of parameters, and the application must “remember” which handler to use with the appropriate query. Of course, if you want to add some static or default configuration parameters, such an approach is absolutely correct. In my opinion, it is a wring decision to create dozens of handlers differing only in certain parameters or values of these parameters. Lets let the application, using Solr, have little bit of freedom make the application responsible for querying Solr.

5. Why do I need a newer version

As in the case file that describes the structure of the index, in case of solrconfig.xml file it is worth to take the time to look at what has changed since the last deployed version of Solr. As you know Solr is developed pretty fast, and thus the configuration tends to change. From my experience I know that for various reasons (such as tight deadlines, lack of knowledge Solr)  configuration files, during deployment updates, are usually left alone. I`ll repeat once again – try to update the configuration files – it take a little time, and You can only gain profit by doing updates.

6. The default configuration is optimal for me

This time, I left the most common mistake as the last one. This is a very frequently repeated error, which draws not just my attention. I emphasize this again – it is worth taking a moment (sometimes it take a bit longer) and adjust the configuration files for our needs. In a large number of cases, the configuration that You we will prepare will be much more optimal for Your implementation, than the configuration that come as default with Solr.

Finally

As in the case of entry for errors in the schema.xml file (http://solr.pl/2010/08/30/5-sins-of-schema-xml-modifications/?lang=en), I recommend the entry titled “The Seven Deadly Sins of Solr” which can be read at: http://www.lucidimagination.com/blog/2010/01/21/the-seven-deadly-sins-of-solr/. A reading worth the time.