Assumptions

Assumptions were quite simple – the user should be able to find photos by their file name, author and other data available in EXIF, like aperture, shutter speed, focal length or ISO value. Another thing was that Solr should take care of grabbing the meta data from JPEG files, so this was definitely something we wanted use Solr cell for. As You can see, those assumptions were simple.

Index structure

Index structure was very simple and contained only most needed fields. The fields section of the schema.xml file looked as follows:

<field name="id" type="string" indexed="true" stored="true" required="true" />
<field name="name" type="text" indexed="true" stored="true" />
<field name="author" type="text" indexed="true" stored="true" />
<field name="iso" type="text" indexed="true" stored="true" multiValued="true" />
<field name="iso_string" type="text" indexed="true" stored="true" multiValued="true" />
<field name="aperture" type="double" indexed="true" stored="true" />
<field name="exposure" type="string" indexed="true" stored="true" />
<field name="exposure_time" type="double" indexed="true" stored="true" />
<field name="focal" type="string" indexed="true" stored="true" />
<field name="focal_35" type="string" indexed="true" stored="true" />
<dynamicField name="ignored_*" type="string" indexed="false" stored="false" multiValued="true" />

The dynamic field was added to ignore the data we weren’t interested in. Also the copyField was introduced to copy the iso field value to iso_string field to enable faceting.

Solr configuration

The following handler definition was added to solrconfig.xml file:

<requestHandler name="/update/extract" class="solr.extraction.ExtractingRequestHandler">
 <lst name="defaults">
  <str name="uprefix">ignored_</str>
  <str name="lowernames">true</str>
  <str name="captureAttr">true</str>
  <str name="fmap.stream_name">name</str>
  <str name="fmap.artist">author</str>
  <str name="fmap.exif_isospeedratings">iso</str>
  <str name="fmap.exif_fnumber">aperture</str>
  <str name="fmap.exposure_time">exposure</str>
  <str name="fmap.exif_exposuretime">exposure_time</str>
  <str name="fmap.focal_length">focal</str>
  <str name="fmap.focal_length_35">focal_35</str>
 </lst>
</requestHandler>

A few words about configuration. The uprefix parameter tells Solr which prefix it should use for the fields that were not mentioned explicitly in the handler configuration. In the above case, the fields which were not mentioned will be prefixed with the ignored_ word. That means that they will be matched by the dynamic field and thus they won’t be indexed (stored=”false” and indexed=”false”). The lowernames parameter with the value of true will cause all the field names to be lowercased. The captureAttr parameter tell Solr, to catch file attributes. The next parameters in the above configuration is mapping definition between fields returned by Tika and fields in the index. For example, fmap.exif_fnumber with the value of aperture says Solr to place the value of Tika exif_fnumber in the aperture index field.

Additional, needed libraries

In order for the above configuration to work we need some additional libraries (similar to the ones described in language identification). From the dist directory that is available in Solr distribution we copy the apache-solr-cell-3.5.0.jar file to tikaDir directory that should be created at the same level as the webapps directory in Solr deployment (of course this is an example). Next we add the following like to the solrconfig.xml file:

<lib dir="../tikaLib/" />

The above tell Solr to include all the libraries from the given directory. Next we need to copy all the jar files from the contrib/extraction/ Solr distribution directory to the created tikaDir directory. Additional solrconfig.xml changes are not needed.

Data indexation

The assumptions were, that there will be about 10.000 new photos a week that will need to be indexed. Those photos will be stored in a shared file system location. A simple bash script was responsible for choosing the files that were needed to be indexed and during its work it run the following command for each file:

curl 'http://solrmaster:8983/solr/photos/update/extract?literal.id=9926&commit=true" -F "myfile=@Wisla_2011_10_10.JPG"

The above command sends a file names Wisla_2011_10_10.JPG to /extract handler and says to run commit command after its processing. In addition to that, the unique id of the file is set (the literal.id parameter).

Queries

I addition to some standard filtering by author or other attributes of the photo it was also desired for the search to work. Yeah, just work We decided, that if we were the users of the application, we would like the fields like author or file name to be important. So, we decided to start with the following query:

q=jan+kowalski+wisla&qf=name^100+author^1000+iso+aperture+exposure_time+focal&defType=dismax

As you can see, the query is simple. Two fields in the index are more valuable then others – name of the photo and its author. The value of those fields were set up by adding query time boosts. The rest of the fields are without boost, so the default boost of 1 applies.

To sum up

The described deployment is really simple. The applications works as so the search The next steps that will have to be done is the JVM and Solr tunning. One of the most important things would be looking at the users behavior and tune up searches to make search experience as good as possible. But let’s leave it for other solr.pl post.

There are a few ways to import data:

• Update Handler
• Cvs Request Handler
• Data Import Handler
• Extracting Request Handler (Solr Cell)
• Client libraries (for example Solrj)
• Apache Connector Framework (formerly Lucene Connector Framework)
• Apache Nutch

In addition to the mentioned above you casn stream your data to search server. As you can see, there is some confusion here and its to provide the best method to use in a particular case at first glance.

Update Handler

Perhaps the most popular method because of simplicity. It requires the preparation of the corresponding XML file and then You must send it via HTTP to a Solr server. It enables document and individual fields boosting.

CSV Request Handler

When we have data in CSV format (Coma Separated Values) or in TSV format (Tab Separated Values) this option may be most convenient. Unfortunately, in contrast to the Update Handler is not possible to boost documents or fields.

Data Import Handler

This method is less common, requires additional and sometimes quite complicated configuration, but allows direct linking to the data source. Using DIH we do not need any additional scripts for data exporting from a source to the format required by Solr. What we get out of the box is: integration with databases (based on JDBC), integration with sources available in XML (for example RSS), e-mail integration (via IMAP protocol) and integration with documents which can be parsed by Apache Tika (like OpenOffice documents, Microsoft Word, RTF, HTML, and many, many more). In addition it is possible to develop your own sources and transformations.

Extracting Request Handler (Solre Cell)

Specialized handler for indexing the content of documents stored in files of different formats. List of supported formats is quite extensive and the indexing is performed by Apache Tika. The drawback of this method is the need of building additional solutions that provide Solr the information about the document and its identifier and that there is no support for providing additional meta data, external to the document.

Client Libraries

Solr provides client libraries for many programming languages. Their capabilities differ, but if the data are generated onboard by the application and the time after in which the data must be available for searching is very low, this way of indexing is often the only available option.

Apache Connector Framework

ACF is a relatively new project, which revealed a wider audience in early 2010. The project was initially an internal project run by the company MetaCarta, and was donated to the open source community and is currently being developed within Apache incubator. The idea is to build a system that allows making connection to the data source with a help of a series of plug-ins. At the moment there is no published version, but the system itself is already worth of interest in the case of the need to integrate with such systems as: FileNet P8 (IBM), Documentum (EMC), LiveLink (OpenText), Patriarch (Memex), Meridio (Autonomy) Windows shares (Microsoft) and SharePoint (Microsoft).

Apache Nutch

Nutch is in fact, a separate project run by the Apache (previously under the Apache Lucene, now a top level project). For the person using Solr Nutch is interesting as it allows to crawl through Web pages and index them by Solr.

Word about streaming

Streaming means the ability to notice Solr, where to download the data to be indexed. This avoids unnecessary data transmission over the network, if the data is on the same server as indexer, or double data transmission (from the source to the importer and from the importer to Solr).

And a word about security

Solr, bye design, is intended to be used in a architecture assuming safe environment. It is very important to note, who and how is able to query solr. While the returned data can be easily reduced, by forcing the use of filters in the definition of the handler, then in the case of indexing is not so easy. In particular, the most dangerous seems to be Solr Cell – it will not only allow to read any file to which Solr have access(eg. files with passwords), but will also will provide a convenient method of searching in those files

Other options

I tried to mention all the methods that does not require any additional work to make indexing work. The problem may be the definition of this additional work, because sometimes it is easier to write additional plug-in than break through numerous configuration options and create a giant XML file. Therefore, the choice of methods was guided by my own sense, which resulted in skipping of some methods (like fetching data from WWW pages with the use of Apache Droids or Heritrix, or solutionsa based on Open Pipeline or Open Pipe).

Certainly in this short article I managed to miss some interesting methods. If so, please comment, I`ll be glad update this entry