So far, in previous articles, we looked at the import data from SQL databases. Today it’s time to import from XML files.<\/p>\n\n\n\n\n\n
Lets look at the following example:\n<\/p>\n
<dataConfig>\n <dataSource type=\"FileDataSource\" \/>\n <document>\n <entity\n name=\"document\"\n processor=\"FileListEntityProcessor\"\n baseDir=\"\/home\/import\/data\/2011-06-27\"\n fileName=\".*\\.xml\nExplanation of the example<\/h2>\n
In comparison with the examples from the earlier articles a new type appeared - the FileDataSource. Example of a complete call:\n<\/p>\n
<dataSource\n type=\"FileDataSource\"\n basePath=\"\/home\/import\/input\"\n encoding=\"utf-8\"\/><\/pre>\nThe additional, not mandatory attributes are obvious:<\/p>\n
After the source definition, we have document definition with two nested entities.<\/p>\n
The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor<\/strong>.This entity is self-supporting and doesn't need any data source (thus dataSource=\"null\"<\/em>). The used attributes:<\/p>\n If you already have a list of files we can go further, the inner entity - its task is to download specific data contained in the files. The data is taken from the file specified by an external entity using a data source. The processor type: XpathEntityProcessor<\/strong> used for XML files have the following attributes:<\/p>\n Additional parameters are not useful in our case and we describe them on another occasion:)<\/p>\n The example allows to read all XML files from the selected directory. We used exactly the same file format as the \"classical\" method of sending documents to Solr using HTTP POST. So why use this method?<\/p>\n The first argument can be the control of the connections between the Solr server and the system that is responsible for generating the files that are going to be indexed. When we do not have full control over the data source, it's better to retrieve the data from the source than to provide additional services that could potentially become a target of attack.<\/p>\n How does it work in practice? In my case I decided to add the possibility of a more advanced search and an ability to do faceting on the category tree. The document contained the \"category\" field, where it stored the path such as \"Cars \/ Four sits \/ Audi\". To create a new queries, we should have an additional fields in the index that hold the category name, the level of the category and also how many levels we have.<\/p>\n To add the required fields we used the ability to define scripts. Quoted previously imported file now looks like this:\n<\/p>\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds - the excess value will be ignored. In the case of \"classical\" indexing method Solr will return an error.<\/p>\n\"\n recursive=\"false\"\n rootEntity=\"false\"\n dataSource=\"null\">\n <entity\n processor=\"XPathEntityProcessor\"\n url=\"\n In comparison with the examples from the earlier articles a new type appeared - the FileDataSource. Example of a complete call:\n<\/p>\n The additional, not mandatory attributes are obvious:<\/p>\n After the source definition, we have document definition with two nested entities.<\/p>\n The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor<\/strong>.This entity is self-supporting and doesn't need any data source (thus dataSource=\"null\"<\/em>). The used attributes:<\/p>\n If you already have a list of files we can go further, the inner entity - its task is to download specific data contained in the files. The data is taken from the file specified by an external entity using a data source. The processor type: XpathEntityProcessor<\/strong> used for XML files have the following attributes:<\/p>\n Additional parameters are not useful in our case and we describe them on another occasion:)<\/p>\n The example allows to read all XML files from the selected directory. We used exactly the same file format as the \"classical\" method of sending documents to Solr using HTTP POST. So why use this method?<\/p>\n The first argument can be the control of the connections between the Solr server and the system that is responsible for generating the files that are going to be indexed. When we do not have full control over the data source, it's better to retrieve the data from the source than to provide additional services that could potentially become a target of attack.<\/p>\n How does it work in practice? In my case I decided to add the possibility of a more advanced search and an ability to do faceting on the category tree. The document contained the \"category\" field, where it stored the path such as \"Cars \/ Four sits \/ Audi\". To create a new queries, we should have an additional fields in the index that hold the category name, the level of the category and also how many levels we have.<\/p>\n To add the required fields we used the ability to define scripts. Quoted previously imported file now looks like this:\n<\/p>\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds - the excess value will be ignored. In the case of \"classical\" indexing method Solr will return an error.<\/p>\n{document.fileAbsolutePath}\"\n useSolrAddSchema=\"true\"\n stream=\"true\">\n <\/entity>\n <\/entity>\n <\/document>\n<\/dataConfig><\/pre>\n In comparison with the examples from the earlier articles a new type appeared - the FileDataSource. Example of a complete call:\n<\/p>\n The additional, not mandatory attributes are obvious:<\/p>\n After the source definition, we have document definition with two nested entities.<\/p>\n The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor<\/strong>.This entity is self-supporting and doesn't need any data source (thus dataSource=\"null\"<\/em>). The used attributes:<\/p>\n If you already have a list of files we can go further, the inner entity - its task is to download specific data contained in the files. The data is taken from the file specified by an external entity using a data source. The processor type: XpathEntityProcessor<\/strong> used for XML files have the following attributes:<\/p>\n Additional parameters are not useful in our case and we describe them on another occasion:)<\/p>\n The example allows to read all XML files from the selected directory. We used exactly the same file format as the \"classical\" method of sending documents to Solr using HTTP POST. So why use this method?<\/p>\n The first argument can be the control of the connections between the Solr server and the system that is responsible for generating the files that are going to be indexed. When we do not have full control over the data source, it's better to retrieve the data from the source than to provide additional services that could potentially become a target of attack.<\/p>\n How does it work in practice? In my case I decided to add the possibility of a more advanced search and an ability to do faceting on the category tree. The document contained the \"category\" field, where it stored the path such as \"Cars \/ Four sits \/ Audi\". To create a new queries, we should have an additional fields in the index that hold the category name, the level of the category and also how many levels we have.<\/p>\n To add the required fields we used the ability to define scripts. Quoted previously imported file now looks like this:\n<\/p>\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds - the excess value will be ignored. In the case of \"classical\" indexing method Solr will return an error.<\/p>\n\"\n recursive=\"false\"\n rootEntity=\"false\"\n dataSource=\"null\">\n <entity\n processor=\"XPathEntityProcessor\"\n transformer=\u201dscript:CategoryPieces\u201d\n url=\"\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds - the excess value will be ignored. In the case of \"classical\" indexing method Solr will return an error.<\/p>\n\"\n recursive=\"false\"\n rootEntity=\"false\"\n dataSource=\"null\">\n <entity\n processor=\"XPathEntityProcessor\"\n url=\"\n In comparison with the examples from the earlier articles a new type appeared - the FileDataSource. Example of a complete call:\n<\/p>\n The additional, not mandatory attributes are obvious:<\/p>\n After the source definition, we have document definition with two nested entities.<\/p>\n The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor<\/strong>.This entity is self-supporting and doesn't need any data source (thus dataSource=\"null\"<\/em>). The used attributes:<\/p>\n If you already have a list of files we can go further, the inner entity - its task is to download specific data contained in the files. The data is taken from the file specified by an external entity using a data source. The processor type: XpathEntityProcessor<\/strong> used for XML files have the following attributes:<\/p>\n Additional parameters are not useful in our case and we describe them on another occasion:)<\/p>\n The example allows to read all XML files from the selected directory. We used exactly the same file format as the \"classical\" method of sending documents to Solr using HTTP POST. So why use this method?<\/p>\n The first argument can be the control of the connections between the Solr server and the system that is responsible for generating the files that are going to be indexed. When we do not have full control over the data source, it's better to retrieve the data from the source than to provide additional services that could potentially become a target of attack.<\/p>\n How does it work in practice? In my case I decided to add the possibility of a more advanced search and an ability to do faceting on the category tree. The document contained the \"category\" field, where it stored the path such as \"Cars \/ Four sits \/ Audi\". To create a new queries, we should have an additional fields in the index that hold the category name, the level of the category and also how many levels we have.<\/p>\n To add the required fields we used the ability to define scripts. Quoted previously imported file now looks like this:\n<\/p>\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds - the excess value will be ignored. In the case of \"classical\" indexing method Solr will return an error.<\/p>\n{document.fileAbsolutePath}\"\n useSolrAddSchema=\"true\"\n stream=\"true\">\n <\/entity>\n <\/entity>\n <\/document>\n<\/dataConfig><\/pre>\n In comparison with the examples from the earlier articles a new type appeared – the FileDataSource. Example of a complete call:\n<\/p>\n The additional, not mandatory attributes are obvious:<\/p>\n After the source definition, we have document definition with two nested entities.<\/p>\n The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor<\/strong>.This entity is self-supporting and doesn’t need any data source (thus dataSource=”null”<\/em>). The used attributes:<\/p>\n If you already have a list of files we can go further, the inner entity – its task is to download specific data contained in the files. The data is taken from the file specified by an external entity using a data source. The processor type: XpathEntityProcessor<\/strong> used for XML files have the following attributes:<\/p>\n Additional parameters are not useful in our case and we describe them on another occasion:)<\/p>\n The example allows to read all XML files from the selected directory. We used exactly the same file format as the “classical” method of sending documents to Solr using HTTP POST. So why use this method?<\/p>\n The first argument can be the control of the connections between the Solr server and the system that is responsible for generating the files that are going to be indexed. When we do not have full control over the data source, it’s better to retrieve the data from the source than to provide additional services that could potentially become a target of attack.<\/p>\n How does it work in practice? In my case I decided to add the possibility of a more advanced search and an ability to do faceting on the category tree. The document contained the “category” field, where it stored the path such as “Cars \/ Four sits \/ Audi”. To create a new queries, we should have an additional fields in the index that hold the category name, the level of the category and also how many levels we have.<\/p>\n To add the required fields we used the ability to define scripts. Quoted previously imported file now looks like this:\n<\/p>\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds – the excess value will be ignored. In the case of “classical” indexing method Solr will return an error.<\/p>\n{document.fileAbsolutePath}”\n useSolrAddSchema=”true”\n stream=”true”>\n <\/entity>\n <\/entity>\n <\/document>\n<\/dataConfig>\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds – the excess value will be ignored. In the case of “classical” indexing method Solr will return an error.<\/p>\n”\n recursive=”false”\n rootEntity=”false”\n dataSource=”null”>\n <entity\n processor=”XPathEntityProcessor”\n url=”\n In comparison with the examples from the earlier articles a new type appeared – the FileDataSource. Example of a complete call:\n<\/p>\n The additional, not mandatory attributes are obvious:<\/p>\n After the source definition, we have document definition with two nested entities.<\/p>\n The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor<\/strong>.This entity is self-supporting and doesn’t need any data source (thus dataSource=”null”<\/em>). The used attributes:<\/p>\n If you already have a list of files we can go further, the inner entity – its task is to download specific data contained in the files. The data is taken from the file specified by an external entity using a data source. The processor type: XpathEntityProcessor<\/strong> used for XML files have the following attributes:<\/p>\n Additional parameters are not useful in our case and we describe them on another occasion:)<\/p>\n The example allows to read all XML files from the selected directory. We used exactly the same file format as the “classical” method of sending documents to Solr using HTTP POST. So why use this method?<\/p>\n The first argument can be the control of the connections between the Solr server and the system that is responsible for generating the files that are going to be indexed. When we do not have full control over the data source, it’s better to retrieve the data from the source than to provide additional services that could potentially become a target of attack.<\/p>\n How does it work in practice? In my case I decided to add the possibility of a more advanced search and an ability to do faceting on the category tree. The document contained the “category” field, where it stored the path such as “Cars \/ Four sits \/ Audi”. To create a new queries, we should have an additional fields in the index that hold the category name, the level of the category and also how many levels we have.<\/p>\n To add the required fields we used the ability to define scripts. Quoted previously imported file now looks like this:\n<\/p>\n When using DIH we need to be aware that it works a little differently. In particular, the attempt to load multiple values \u200b\u200bto a field that is not multivalued (in the schema.xml file) in DIH succeeds – the excess value will be ignored. In the case of “classical” indexing method Solr will return an error.<\/p>\n{document.fileAbsolutePath}”\n useSolrAddSchema=”true”\n stream=”true”>\n <\/entity>\n <\/entity>\n <\/document>\n<\/dataConfig>\n In comparison with the examples from the earlier articles a new type appeared – the FileDataSource. Example of a complete call:\n<\/p>\n The additional, not mandatory attributes are obvious:<\/p>\n After the source definition, we have document definition with two nested entities.<\/p>\n The purpose of the main entity is to generate the files list. To do that, we use the FileListEntityProcessor<\/strong>.This entity is self-supporting and doesn’t need any data source (thus dataSource=”null”<\/em>). The used attributes:<\/p>\n\n
\n
But why all this?<\/h2>\n
Push and Pull<\/h2>\n
Prototyping and change testing<\/h2>\n
<dataConfig>\n <script><![CDATA[\n function CategoryPieces(row) {\n var pieces = row.get('category').split('\/');\n var arr = new Array();\n for (var i=0; i < pieces.length; i++) {\n row.put('category_level_' + i, pieces[i].trim());\n arr[i] = pieces[i].trim();\n }\n row.put('category_level_max', (pieces.length - 1).toFixed());\n row.put('category', arr.join('\/'));\n return row;\n }\n ]]><\/script>\n <dataSource type=\"FileDataSource\" \/>\n <document>\n <entity\n name=\"document\"\n processor=\"FileListEntityProcessor\"\n baseDir=\"\/home\/import\/data\/2011-06-27\"\n fileName=\".*\\.xml\nNote at the end<\/h2>\n
Explanation of the example<\/h2>\n
<\/pre>\n
\n
\n
\n
But why all this?<\/h2>\n
Push and Pull<\/h2>\n
Prototyping and change testing<\/h2>\n
<\/pre>\n
Note at the end<\/h2>\n
Explanation of the example<\/h2>\n
<\/pre>\n
\n
\n
\n
But why all this?<\/h2>\n
Push and Pull<\/h2>\n
Prototyping and change testing<\/h2>\n
<\/pre>\n
Note at the end<\/h2>\n
Note at the end<\/h2>\n
Explanation of the example<\/h2>\n
<\/pre>\n
\n
\n
\n
But why all this?<\/h2>\n
Push and Pull<\/h2>\n
Prototyping and change testing<\/h2>\n
<\/pre>\n
Note at the end<\/h2>\n
Explanation of the example<\/h2>\n
<\/pre>\n
\n
\n
\n
But why all this?<\/h2>\n
Push and Pull<\/h2>\n
Prototyping and change testing<\/h2>\n
<\/pre>\n
Note at the end<\/h2>\n
Note at the end<\/h2>\n
Explanation of the example<\/h2>\n
<\/pre>\n
\n
\n
\n
But why all this?<\/h2>\n
Push and Pull<\/h2>\n
Prototyping and change testing<\/h2>\n
<\/pre>\n
Note at the end<\/h2>\n
Explanation of the example<\/h2>\n
<\/pre>\n
\n