{"id":256,"date":"2011-04-04T20:37:27","date_gmt":"2011-04-04T18:37:27","guid":{"rendered":"http:\/\/sematext.solr.pl\/?p=256"},"modified":"2020-11-11T20:38:12","modified_gmt":"2020-11-11T19:38:12","slug":"indexing-files-like-doc-pdf-solr-and-tika-integration","status":"publish","type":"post","link":"https:\/\/solr.pl\/en\/2011\/04\/04\/indexing-files-like-doc-pdf-solr-and-tika-integration\/","title":{"rendered":"Indexing files like doc, pdf – Solr and Tika integration"},"content":{"rendered":"

In the previous article<\/a> we have given basic information about how to enable the indexing of binary files, ie MS Word files, PDF files or LibreOffice files. Today we will do the same thing, using the Data Import Handler. Since a few days ago a new version of the Solr server (3.1) have been released, the following guidelines are based on this version. For the purpose of the article I used the “example” application – all of the changes relate to this application.<\/p>\n\n\n\n\n\n

Assumptions<\/h2>\n

We assume that the data is available in the XML format and contain basic information about the document along with the file name where the document contents is located. The files are located in a defined directory. Example file look like this:\n<\/p>\n

<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<albums>\n    <album>\n        <author>John F.<\/author>\n        <title>Life in picture<\/title>\n        <description>1.jpg<\/description>\n    <\/album>\n    <album>\n        <author>Peter Z.<\/author>\n        <title>Simple presentation<\/title>\n        <description>2.pptx<\/description>\n    <\/album>\n<\/abums><\/pre>\n
As you can see the data is characterized by the fact that individual elements do not have a unique identifier. But we can handle it \ud83d\ude42
\nFirst we modify the schema by adding a definition of a field that holds the contents of the file:\n<\/p>\n
<field name=\"content\" type=\"text\" indexed=\"true\" stored=\"true\" multiValued=\"true\"\/><\/pre>\n
Next we modify the solrconfig.xml and add DIH configuration:\n<\/p>\n
   \n<requestHandler name=\"\/dataimport\">\n   <lst name=\"defaults\">\n      <str name=\"config\">data-config.xml<\/str>\n   <\/lst>\n<\/requestHandler><\/pre>\n
Since  we will use the entity processor located in the extras  (TikaEntityProcessor<\/em>), we need to modify the line loading the DIH  library:\n<\/p>\n
 <lib dir=\"..\/..\/dist\/\" regex=\"apache-solr-dataimporthandler-.*\\.jar\" \/><\/pre>\n
The next step is to create a data-config.xml file. In our case it should look like this:\n<\/p>\n
<dataConfig>\n    <script><![CDATA[\n        id = 1;\n        function GenerateId(row) {\n            row.put('id', (id ++).toFixed());\n            return row;\n        }       \n       ]]><\/script>\n   <dataSource type=\"BinURLDataSource\" name=\"data\"\/>\n    <dataSource type=\"URLDataSource\" baseUrl=\"http:\/\/localhost\/tmp\/bin\/\" name=\"main\"\/>\n    <document>\n        <entity name=\"rec\" processor=\"XPathEntityProcessor\" url=\"data.xml\" forEach=\"\/albums\/album\" dataSource=\"main\" transformer=\"script:GenerateId\">\n            <field column=\"title\" xpath=\"\/\/title\" \/>\n            <field column=\"description\" xpath=\"\/\/description\" \/>\n            <entity processor=\"TikaEntityProcessor\" url=\"http:\/\/localhost\/tmp\/bin\/\n
Generating record identifier - scripts<\/h2>\n
The first interesting feature is the use of standard ScriptTransformer in order to generate documents identifiers. With javascript \"GenerateId\"<\/em> method and a references to it (transformer=\"script:GenerateId\")<\/em>, each record will be numbered. Frankly  it is not too good method of dealing with the lack of identifiers,  because it does not allow for incremental indexing (we are not able to  distinguish between the various versions of the record) - it is used  here only to show how easy it is to modify the records. If you do not like Javascript - you can use any scripting language supported by java6.<\/p>\n
The use of multiple data sources<\/h2>\n
Another interesting element is the use of several data sources. Since our metadata is available in the XML file, you need to download this file. We  are using the standard approach: we define the UrlDataSource <\/em>and then  we use the XpathEntityProcessor <\/em>to analyze the incoming data. Since  we have to download binary attachments to each record, we define an  additional data source: BinURLDataSource <\/em>and additional entity, using  the TikaEntityProcessor<\/em>. Now  we only need to notify the entity where to download the file (the url <\/em>attribute with the reference to the entity - a parent) and the  notification from which data source should be used (dataSource <\/span><\/em>attribute). The  whole is complemented by a list of fields to be indexed (an additional  attribute meta <\/em>means that the data are retrieved from the file's  metadata).<\/p>\n
Available fields<\/h2>\n
Apache Tika allows you to download a number of additional data from the document. In the example above we used only the title, author, and content of the document. Complete  information about the available fields are included in the interfaces  that are implemented by the class Metadata  (http:\/\/tika.apache.org\/0.9\/api\/org\/apache\/tika\/metadata\/Metadata.html<\/a>)  exactly in the defined constants. In particular interesting are DublinCore <\/em>and MSOffice<\/em>.<\/p>\n
The end<\/h2>\n
A short time after starting solr and running the import process (calling the address: http:\/\/localhost:8983\/solr\/dataimport?command=full-import<\/em>) documents are indexed and that should be visible after sending the following query to Solr: http:\/\/localhost:8983\/solr\/select?q=*:*<\/em><\/p>\n{rec.description}\" dataSource=\"data\">\n                <field column=\"text\" name=\"content\" \/>\n                <field column=\"Author\" name=\"author\" meta=\"true\" \/>\n                <field column=\"title\" name=\"title\" meta=\"true\" \/>\n            <\/entity>\n        <\/entity>\n    <\/document>\n<\/dataConfig><\/pre>\n
Generating record identifier – scripts<\/h2>\n
The first interesting feature is the use of standard ScriptTransformer in order to generate documents identifiers. With javascript “GenerateId”<\/em> method and a references to it (transformer=”script:GenerateId”)<\/em>, each record will be numbered. Frankly  it is not too good method of dealing with the lack of identifiers,  because it does not allow for incremental indexing (we are not able to  distinguish between the various versions of the record) – it is used  here only to show how easy it is to modify the records. If you do not like Javascript – you can use any scripting language supported by java6.<\/p>\n
The use of multiple data sources<\/h2>\n
Another interesting element is the use of several data sources. Since our metadata is available in the XML file, you need to download this file. We  are using the standard approach: we define the UrlDataSource <\/em>and then  we use the XpathEntityProcessor <\/em>to analyze the incoming data. Since  we have to download binary attachments to each record, we define an  additional data source: BinURLDataSource <\/em>and additional entity, using  the TikaEntityProcessor<\/em>. Now  we only need to notify the entity where to download the file (the url <\/em>attribute with the reference to the entity – a parent) and the  notification from which data source should be used (dataSource <\/span><\/em>attribute). The  whole is complemented by a list of fields to be indexed (an additional  attribute meta <\/em>means that the data are retrieved from the file’s  metadata).<\/p>\n
Available fields<\/h2>\n
Apache Tika allows you to download a number of additional data from the document. In the example above we used only the title, author, and content of the document. Complete  information about the available fields are included in the interfaces  that are implemented by the class Metadata  (http:\/\/tika.apache.org\/0.9\/api\/org\/apache\/tika\/metadata\/Metadata.html<\/a>)  exactly in the defined constants. In particular interesting are DublinCore <\/em>and MSOffice<\/em>.<\/p>\n
The end<\/h2>\n
A short time after starting solr and running the import process (calling the address: http:\/\/localhost:8983\/solr\/dataimport?command=full-import<\/em>) documents are indexed and that should be visible after sending the following query to Solr: http:\/\/localhost:8983\/solr\/select?q=*:*<\/em><\/p>","protected":false},"excerpt":{"rendered":"
In the previous article we have given basic information about how to enable the indexing of binary files, ie MS Word files, PDF files or LibreOffice files. Today we will do the same thing, using the Data Import Handler. Since<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[27],"tags":[256,254],"class_list":["post-256","post","type-post","status-publish","format-standard","hentry","category-solr-en","tag-data-import-handler-2","tag-dih-2"],"_links":{"self":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/256","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/comments?post=256"}],"version-history":[{"count":1,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/256\/revisions"}],"predecessor-version":[{"id":257,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/256\/revisions\/257"}],"wp:attachment":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/media?parent=256"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/categories?post=256"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/tags?post=256"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}