At the beginning

You should remember that the described functionality was introduced in Solr 3.5.

Assumptions

We will be using two fields to identify the document language: title and body. We want to store the information of the detected language in the lang field.

Index structure

The structure of our index is of course simplified and contain only fields needed for the test. So the field definition part of the schema.xml file looks like this:

<field name="id" type="string" indexed="true" stored="true" required="true" />
<field name="title" type="text_ws" indexed="true" stored="true" />
<field name="body" type="text_ws" indexed="true" stored="true" />
<field name="lang" type="string" indexed="true" stored="true" />

All the fields as marked as stored=”true” for simplicity.

Update request processor configuration

In order to be able to use the language identification feature we need to configure Solr update request processor. We will be using the one that is using Apache Tika (there is a second implementation based on http://code.google.com/p/language-detection/). In order to configure the process we add the following to the solrconfig.xml file:

<updateRequestProcessorChain name="langid">
  <processor name="langid" class="org.apache.solr.update.processor.TikaLanguageIdentifierUpdateProcessorFactory">
    <lst name="defaults">
      <str name="langid.fl">title,body</str>
      <str name="langid.langField">lang</str>
    </lst>
  </processor>
  <processor class="solr.LogUpdateProcessorFactory" />
  <processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>

Other parameters of the TikaLanguageIdentifierUpdateProcessorFactory are described on Apache Solr wiki pages available at the following URL address: http://wiki.apache.org/solr/LanguageDetection.

Additional libraries

In order for the update request processor to be working we need some additional libraries. From the dist directory from Apache Solr distribution we copy the apache-solr-langid-3.5.0.jar to tikaDir (for example), which we make on the same level as the webapps directory. Then we add the following line to the solrconfig.xml file:

<lib dir="../tikaLib/" regex="apache-solr-langid-\d.*\.jar" />

The next library we will need is the Tika jar with all the goodiess (tika-app-1.0.jar) which we can download at the following URL address: http://tika.apache.org/. We place it in the same tikaDir directory and then we add the following entry to the solrconfig.xml file:

<lib dir="../tikaLib/" regex="tika-app-1.0.jar" />

Test documents

For the testing purposes I decided to prepare three documents. The first was in English, the second one in Polish and the third one in German. Their content was downloaded from Wikipedia. They look as follows:

tika_en.xml

<add>
<doc>
  <field name="id">1</field>
  <field name="title">Water</field>
  <field name="body">Water is a chemical substance with the chemical formula H2O. A water molecule contains one oxygen and two hydrogen atoms connected by covalent bonds. Water is a liquid at ambient conditions, but it often co-exists on Earth with its solid state, ice, and gaseous state (water vapor or steam). Water also exists in a liquid crystal state near hydrophilic surfaces.[1][2] Under nomenclature used to name chemical compounds, Dihydrogen monoxide is the scientific name for water, though it is almost never used.</field>
</doc>
</add>

tika_pl.xml

<add>
<doc>
  <field name="id">2</field>
  <field name="title">Woda</field>
  <field name="body">Woda (tlenek wodoru; nazwa systematyczna IUPAC: oksydan) – związek chemiczny o wzorze H2O, występujący w warunkach standardowych w stanie ciekłym. W stanie gazowym wodę określa się mianem pary wodnej, a w stałym stanie skupienia – lodem. Słowo woda jako nazwa związku chemicznego może się odnosić do każdego stanu skupienia.</field>
</doc>
</add>

tika_de.xml

<add>
<doc>
  <field name="id">3</field>
  <field name="title">Wasser</field>
  <field name="body">Wasser (H2O) ist eine chemische Verbindung aus den Elementen Sauerstoff (O) und Wasserstoff (H). Wasser ist die einzige chemische Verbindung auf der Erde, die in der Natur in allen drei Aggregatzuständen vorkommt. Die Bezeichnung Wasser wird dabei besonders für den flüssigen Aggregatzustand verwendet. Im festen (gefrorenen) Zustand spricht man von Eis, im gasförmigen Zustand von Wasserdampf.</field>
</doc>
</add>

More testing

To index the data I used the following shell commands:

curl 'http://localhost:8983/solr/update?update.chain=langid' --data-binary @tika_pl.xml -H 'Content-type:application/xml'
curl 'http://localhost:8983/solr/update?update.chain=langid' --data-binary @tika_en.xml -H 'Content-type:application/xml'
curl 'http://localhost:8983/solr/update?update.chain=langid' --data-binary @tika_de.xml -H 'Content-type:application/xml'
curl 'http://localhost:8983/solr/update?update.chain=langid' --data-binary '<commit/>' -H 'Content-type:application/xml'

It is worth to notice the additional update.chain=langid parameter added to the request. This parameter is used to tell Solr which update processor to use when indexing the data. In the example we told Solr that it should use our defined update processor.

Indexed data

So let’s have a look at the indexed data. We will do that by running the following query: q=*:*&indent=true.

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
  <int name="status">0</int>
  <int name="QTime">0</int>
  <lst name="params">
    <str name="indent">true</str>
    <str name="q">*:*</str>
  </lst>
</lst>
<result name="response" numFound="3" start="0">
  <doc>
    <str name="body">Woda (tlenek wodoru; nazwa systematyczna IUPAC: oksydan) – związek chemiczny o wzorze H2O, występujący w warunkach standardowych w stanie ciekłym. W stanie gazowym wodę określa się mianem pary wodnej, a w stałym stanie skupienia – lodem. Słowo woda jako nazwa związku chemicznego może się odnosić do każdego stanu skupienia.</str>
    <str name="id">2</str>
    <str name="lang">pl</str>
    <str name="title">Woda</str>
  </doc>
  <doc>
    <str name="body">Water is a chemical substance with the chemical formula H2O. A water molecule contains one oxygen and two hydrogen atoms connected by covalent bonds. Water is a liquid at ambient conditions, but it often co-exists on Earth with its solid state, ice, and gaseous state (water vapor or steam). Water also exists in a liquid crystal state near hydrophilic surfaces.[1][2] Under nomenclature used to name chemical compounds, Dihydrogen monoxide is the scientific name for water, though it is almost never used.</str>
    <str name="id">1</str>
    <str name="lang">en</str>
    <str name="title">Water</str>
  </doc>
  <doc>
    <str name="body">Wasser (H2O) ist eine chemische Verbindung aus den Elementen Sauerstoff (O) und Wasserstoff (H). Wasser ist die einzige chemische Verbindung auf der Erde, die in der Natur in allen drei Aggregatzuständen vorkommt. Die Bezeichnung Wasser wird dabei besonders für den flüssigen Aggregatzustand verwendet. Im festen (gefrorenen) Zustand spricht man von Eis, im gasförmigen Zustand von Wasserdampf.</str>
    <str name="id">3</str>
    <str name="lang">de</str>
    <str name="title">Wasser</str>
  </doc>
</result>
</response>

As you can see, Solr with the use of Tika, was able to identify the languages of the indexed documents. Of course, let’s not be too optimistic, because mistakes happen, especially when dealing with multi-language documents, but that’s understandable.

To sum up

You should remember, that the language identification feature is not perfect and can make mistakes. Also remember, that the longer the documents, the better the functionality will work. Of course the problem is that we can’t use the language identification during query time, but it’s not only problem with Solr and Tika. You can deal with that by identifying your user, it’s web browser or place he is located in.

Introduction

First, a few words about the opportunities that we have when we choose Apache Tika. Apache Tika is a framework designed to extract information from the so-called “rich documents”-documents such as PDF files, the files in Microsoft Office format, rtf, but not only. Using Apache Tika we can also extract information from compressed documents, HTML files, images (eg jpg, png, gif), audio files (eg mp3, midi, wave), and compiled Java bytecode files. In addition, Apache Tika can detect the type of file being processed, which further simplifies the work with such documents. It is worth mentioning that the described framework is based on libraries such as PDFBox, Apache POI, or Neko HTML which indirectly guarantee very good results of extracted data.

Sample index structure

I’ll skip how to manually start the extraction of the contents of the documents in the Apache Tika and I will focus on the integration of this framework with Solr and how trivial it is. Assume that we are interested in the ID, title and contents of the documents we have to index. Thus we create a simple schema.xml file describing the index structure, which could look like this:

<field name="id" type="string" indexed="true" stored="true" required="true" />
<field name="tytul" type="text" indexed="true" stored="true"/>
<field name="zawartosc" type="text" indexed="true" stored="false" multiValued="true"/>

Configuration

To the solrconfig.xml file we add the following entry which defines a handler that will handle the indexing of documents using Apache Tika:

<requestHandler name="/update/extract" class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
   <lst name="defaults">
      <str name="fmap.Last-Modified">last_modified</str>
      <str name="uprefix">ignored_</str>
    </lst>
</requestHandler>

All update requests sent to the /update/extract address will be handled by Apache Tika. Of course, remember to send the commit command to the update handler after sending the documents to the handler using Apache Tika. Otherwise, your documents won’t be visible. In the standard Solr deployment you shoudl send the commit command to the handler located under /update.

In the configuration we told the extraction handler to assign the Last-Modified attribute to the last_modified field and to ignore the fields that do are not specified.

Additional notes

If you are going to index large binary files, remember to change the size limits. To do that, change the following values in the solrconfig.xml file:<requestDispatcher handleSelect=”true”> <requestParsers enableRemoteStreaming=”false” multipartUploadLimitInKB=”10240″ />

fmap.Last-Modified

The end

All parameters defining the ExtractingRequestHandler can be found at: http://wiki.apache.org/solr/ExtractingRequestHandler.

There are a few ways to import data:

• Update Handler
• Cvs Request Handler
• Data Import Handler
• Extracting Request Handler (Solr Cell)
• Client libraries (for example Solrj)
• Apache Connector Framework (formerly Lucene Connector Framework)
• Apache Nutch

In addition to the mentioned above you casn stream your data to search server. As you can see, there is some confusion here and its to provide the best method to use in a particular case at first glance.

Update Handler

Perhaps the most popular method because of simplicity. It requires the preparation of the corresponding XML file and then You must send it via HTTP to a Solr server. It enables document and individual fields boosting.

CSV Request Handler

When we have data in CSV format (Coma Separated Values) or in TSV format (Tab Separated Values) this option may be most convenient. Unfortunately, in contrast to the Update Handler is not possible to boost documents or fields.

Data Import Handler

This method is less common, requires additional and sometimes quite complicated configuration, but allows direct linking to the data source. Using DIH we do not need any additional scripts for data exporting from a source to the format required by Solr. What we get out of the box is: integration with databases (based on JDBC), integration with sources available in XML (for example RSS), e-mail integration (via IMAP protocol) and integration with documents which can be parsed by Apache Tika (like OpenOffice documents, Microsoft Word, RTF, HTML, and many, many more). In addition it is possible to develop your own sources and transformations.

Extracting Request Handler (Solre Cell)

Specialized handler for indexing the content of documents stored in files of different formats. List of supported formats is quite extensive and the indexing is performed by Apache Tika. The drawback of this method is the need of building additional solutions that provide Solr the information about the document and its identifier and that there is no support for providing additional meta data, external to the document.

Client Libraries

Solr provides client libraries for many programming languages. Their capabilities differ, but if the data are generated onboard by the application and the time after in which the data must be available for searching is very low, this way of indexing is often the only available option.

Apache Connector Framework

ACF is a relatively new project, which revealed a wider audience in early 2010. The project was initially an internal project run by the company MetaCarta, and was donated to the open source community and is currently being developed within Apache incubator. The idea is to build a system that allows making connection to the data source with a help of a series of plug-ins. At the moment there is no published version, but the system itself is already worth of interest in the case of the need to integrate with such systems as: FileNet P8 (IBM), Documentum (EMC), LiveLink (OpenText), Patriarch (Memex), Meridio (Autonomy) Windows shares (Microsoft) and SharePoint (Microsoft).

Apache Nutch

Nutch is in fact, a separate project run by the Apache (previously under the Apache Lucene, now a top level project). For the person using Solr Nutch is interesting as it allows to crawl through Web pages and index them by Solr.

Word about streaming

Streaming means the ability to notice Solr, where to download the data to be indexed. This avoids unnecessary data transmission over the network, if the data is on the same server as indexer, or double data transmission (from the source to the importer and from the importer to Solr).

And a word about security

Solr, bye design, is intended to be used in a architecture assuming safe environment. It is very important to note, who and how is able to query solr. While the returned data can be easily reduced, by forcing the use of filters in the definition of the handler, then in the case of indexing is not so easy. In particular, the most dangerous seems to be Solr Cell – it will not only allow to read any file to which Solr have access(eg. files with passwords), but will also will provide a convenient method of searching in those files

Other options

I tried to mention all the methods that does not require any additional work to make indexing work. The problem may be the definition of this additional work, because sometimes it is easier to write additional plug-in than break through numerous configuration options and create a giant XML file. Therefore, the choice of methods was guided by my own sense, which resulted in skipping of some methods (like fetching data from WWW pages with the use of Apache Droids or Heritrix, or solutionsa based on Open Pipeline or Open Pipe).

Certainly in this short article I managed to miss some interesting methods. If so, please comment, I`ll be glad update this entry