{"id":260,"date":"2011-04-18T20:42:09","date_gmt":"2011-04-18T18:42:09","guid":{"rendered":"http:\/\/sematext.solr.pl\/?p=260"},"modified":"2020-11-11T20:43:05","modified_gmt":"2020-11-11T19:43:05","slug":"solr-3-1-json-update-handler","status":"publish","type":"post","link":"https:\/\/solr.pl\/en\/2011\/04\/18\/solr-3-1-json-update-handler\/","title":{"rendered":"Solr 3.1: JSON Update Handler"},"content":{"rendered":"

After the release of Solr 3.1<\/a> I decided to look into the extended list of formats through which we can update the indexes. Until now we had a choice of three kinds of formats with which we were able to provide data – XML, CSV, and so. called JavaBin. The release of Solr 3.1 introduces the fourth format – JSON.<\/p>\n\n\n\n\n\n

Let’s start<\/h3>\n

<\/em>The new handler (JsonUpdateRequestHandler<\/em>) allows us to transfer data in the JSON format which in theory should translate into a smaller amount of data sent over the network and the speedup of indexing, as the JSON parser is theoretically faster than XML parsers. But let’s leave the performance for now.<\/p>\n

Configuration<\/h3>\n

Let’s start by defining a handler. To do that add the following definition to the solrconfig.xml <\/em>file (if you use the default solrconfig.xml file provided with Solr 3.1 than this handler is already defined):\n<\/p>\n

<requestHandler name=\"\/update\/json\" class=\"solr.JsonUpdateRequestHandler\" startup=\"lazy\" \/><\/pre>\n
The entry above defines a new handler that will be initialized when used for the first time (startup=”lazy”<\/em>).<\/p>\n
Indexing<\/h3>\n
The next step is to prepare the data – of course in JSON format. Here’s an example showing two documents in one file called data.json<\/em>:\n<\/p>\n
{\n\n\"add\": {\n  \"doc\": {\n    \"id\" : \"123456788\",\n    \"region\" : [\"abc\",\"def\"],\n    \"name\" : \"ABCDEF\",\n  }\n}\n\n,\n\"add\": {\n  \"doc\": {\n    \"id\" : \"123456789\",\n    \"region\" : [\"abc\",\"def\"],\n    \"name\" : \"XYZMN\",\n  }\n}\n\n}<\/pre>\n
Such prepared file can be sent to the \/update\/json<\/em> address and thus be indexed. Remember  to send a commit command to the appropriate address (standard \/update<\/em>)  in order to tell Solr to open a new index searcher.<\/p>\n
Performance<\/h3>\n
At the end I left myself what I’m really most interested in – the performance of the new handler. According  to the information stored in JIRA system we can be expect that  JsonUpdateRequestHandler <\/em>will be faster than its counterpart processor  of XML format. To examine this, I prepared the files of 10.000, 100.000  and 1 million documents. Every document contained an identifier (string field), two  regions (String field, multivalued) and the name (text field). One  file was saved in the JSON format, the second one was saved in XML  format, the third one was saved in CSV format. All files were then  indexed separately. Here is an outcome of this simple test:<\/p>\n[table \u201c10\u201d not found \/]
\n\n
The conclusions suggest themselves. First, XML data is relatively larger than the one written in JSON format (the difference is about 35%). However,  a file stored in JSON format, is larger (which might be expected) than  the one written in the CSV. If you send data not on the local network,  the size is relevant – the difference in file size is significant enough  that it is worth thinking about changing the XML to any of the formats  that require less space.<\/p>\n
Indexation time<\/h3>\n
Another thing is the indexing time. Leaning  on the results of this simple test we can think that  JsonUpdateRequestHandler <\/em>is slightly (about 7 – 9%) faster than the  XmlUpdateRequestHandler<\/em>. As  you can see, the difference is similar for JsonUpdateRequestHandler <\/em>and  CSVRequestHandler<\/em>, where the handler operates on files in CSV format is  faster than its counterpart that operates in JSON format by about 7 to  9%. Let’s  hope that when the noggit <\/a>library comes out of Apache Labs, its  performance will be even greater, and thus we will see even faster  JsonUpdateRequestHandler<\/em>.<\/p>","protected":false},"excerpt":{"rendered":"
After the release of Solr 3.1 I decided to look into the extended list of formats through which we can update the indexes. Until now we had a choice of three kinds of formats with which we were able to<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[27],"tags":[337,315,338,164,340,341],"class_list":["post-260","post","type-post","status-publish","format-standard","hentry","category-solr-en","tag-file-format-2","tag-handler","tag-json-2","tag-solr-2","tag-update-2","tag-update-handler-2"],"_links":{"self":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/260","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/comments?post=260"}],"version-history":[{"count":1,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/260\/revisions"}],"predecessor-version":[{"id":261,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/260\/revisions\/261"}],"wp:attachment":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/media?parent=260"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/categories?post=260"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/tags?post=260"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}