Data Import Handler – import from Solr XML files

So far, in previous articles, we looked at the import data from SQL databases. Today it’s time to import from XML files.

Example

Lets look at the following example:

<dataConfig>
  <dataSource type="FileDataSource" />
  <document>
    <entity
      name="document"
      processor="FileListEntityProcessor"
      baseDir="/home/import/data/2011-06-27"
      fileName=".*.xml$"
      recursive="false"
      rootEntity="false"
      dataSource="null">
      <entity
        processor="XPathEntityProcessor"
        url="${document.fileAbsolutePath}"
        useSolrAddSchema="true"
        stream="true">
      </entity>
    </entity>
  </document>
</dataConfig>

Explanation of the example

In comparison with the examples from the earlier articles a new type appeared – the FileDataSource. Example of a complete call:

<dataSource
  type="FileDataSource"
  basePath="/home/import/input"
  encoding="utf-8"/>

The additional, not mandatory attributes are obvious:

  • basePath – the directory which will be used to calculate the relative path of the “entity” tag
  • encoding – the file encoding (default: the OS default encoding)

After the source definition, we have document definition with two nested entities.

The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor.This entity is self-supporting and doesn’t need any data source (thus dataSource=”null”). The used attributes:

  • fileName (mandatory) – regular expression that says which files to choose
  • recursive – should subdirectories be checked  (default: no)
  • rootEntity – says about if the data from the entity should be treated as documents source. Because we don’t want to index files list, which this entity provides, we need to set this attribute to false. After setting this attribute to false the next entity will be treated as main entity and its data will be indexed.
  • baseDir (mandatory) – the directory where the files should be located
  • dataSource – in this case we set this parameter to “null” because the entity doesn’t use data source  (we can ommit this parameter in Solr > 1.3)
  • excludes – regular expression which says which files to exclude from indexing
  • newerThan – says, that only files newer that the parameter value will be taken into consideration. This parameter can take a value in the format: YYYY-MM-dd HH:mm:ss or a single quoted string, like: ‘NOW – 7DAYS’ or an variable that contains the data, for example: ${variable}
  • olderThan – the same as above, but says about older files
  • biggerThan – only files bigger than the value of the parameter will be taken into consideration
  • smallerThan –only files smaller than the value of this parameter will be taken into consideration

If you already have a list of files we can go further, the inner entity – its task is to download specific data contained in the files. The data is taken from the file specified by an external entity using a data source. The processor type: XpathEntityProcessor used for XML files have the following attributes:

  • url – the input data
  • useSolrAddSchemainformation, that the input data is in Solr XML format
  • stream – should we use stream for document processing. In case of large XML files, it’s good to use stream=”true” which will  use far less memory and won’t try to load the whole XML file into the memory.

Additional parameters are not useful in our case and we describe them on another occasion:)

But why all this?

The example allows to read all XML files from the selected directory. We used exactly the same file format as the “classical” method of sending documents to Solr using HTTP POST. So why use this method?

Push and Pull

The first argument can be the control of the connections between the Solr server and the system that is responsible for generating the files that are going to be indexed. When we do not have full control over the data source, it’s better to retrieve the data from the source than to provide additional services that could potentially become a target of attack.

Prototyping and change testing

How does it work in practice? In my case I decided to add the possibility of a more advanced search and an ability to do faceting on the category tree. The document contained the “category” field, where it stored the path such as “Cars / Four sits / Audi”. To create a new queries, we should have an additional fields in the index that hold the category name, the level of the category and also how many levels we have.

To add the required fields we used the ability to define scripts. Quoted previously imported file now looks like this:

<dataConfig>
  <script><![CDATA[
    function CategoryPieces(row) {
      var pieces = row.get('category').split('/');
      var arr = new Array();
      for (var i=0; i < pieces.length; i++) {
        row.put('category_level_' + i, pieces[i].trim());
        arr[i] = pieces[i].trim();
      }
      row.put('category_level_max', (pieces.length - 1).toFixed());
      row.put('category', arr.join('/'));
      return row;
  }
  ]]></script>
  <dataSource type="FileDataSource" />
  <document>
    <entity
      name="document"
      processor="FileListEntityProcessor"
      baseDir="/home/import/data/2011-06-27"
      fileName=".*.xml$"
      recursive="false"
      rootEntity="false"
      dataSource="null">
      <entity
        processor="XPathEntityProcessor"
        transformer=”script:CategoryPieces”
        url="${document.fileAbsolutePath}"
        useSolrAddSchema="true"
        stream="true">
      </entity>
    </entity>
  </document>
</dataConfig>

Note at the end

When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values ​​to a field that is not multivalued (in the schema.xml file) in DIH succeeds – the excess value will be ignored. In the case of “classical” indexing method Solr will return an error.

This post is also available in: Polish

This entry was posted on Tuesday, August 16th, 2011 at 08:19 and is filed under About Solr. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.