{"id":990,"date":"2019-04-15T16:07:46","date_gmt":"2019-04-15T14:07:46","guid":{"rendered":"http:\/\/sematext.solr.pl\/?p=990"},"modified":"2020-11-14T16:08:17","modified_gmt":"2020-11-14T15:08:17","slug":"solr-8-bytebuffersdirectory-quick-look","status":"publish","type":"post","link":"https:\/\/solr.pl\/en\/2019\/04\/15\/solr-8-bytebuffersdirectory-quick-look\/","title":{"rendered":"Solr 8 – ByteBuffersDirectory – quick look"},"content":{"rendered":"\n

One of the new features introduced in the recently released Solr 8.0 is new implementation of the Directory<\/em> interface – one that will replace not scalable RAMDirectory<\/em>. The new implementation called ByteBuffersDirectory<\/em> is dedicated to small, short lived data that is held only in memory. Let’s have a quick look into potential use cases, advantages and drawbacks of this new implementation.<\/p>\n\n\n\n\n\n\n\n

Configuration<\/h2>\n\n\n\n

Simplest things first – let’s start with configuration. This time the situation is very simple. The only thing that we need to take care of is proper DirectoryFactory<\/em> implementation in the solrconfig.xml<\/em> file, file example<\/p>\n\n\n\n

<directoryFactory name=\"DirectoryFactory\" class=\"solr.ByteBuffersDirectoryFactory\" \/><\/pre>\n\n\n\n
One thing we need to remember though is the lockType<\/em> implementation. At the time of writing, the only possible value for lockType<\/em> when ByteBuffersDirectoryFactory<\/em> is used is simple<\/em>. No other lock type is supported.<\/p>\n\n\n\n
Possible Usage<\/h2>\n\n\n\n
The new implementation of the Directory<\/em> interface will replace RAMDirectory<\/em> – implementation that is present in Lucene and Solr for a long period of time now. However, keep in mind that RAMDirectory<\/em> is not suggested to be used because of its issues. <\/p>\n\n\n\n
Theortically we should use the in-memory Directory<\/em> implementation for small, short lived Lucene indices. This allows us to avoid disk usage for writes and reads for potentially higher indexing throughput and lower query latency.<\/p>\n\n\n\n
For example, you can use the newly introduced Directory<\/em> implementation for keeping highly updated data like number of products or for that needed only once – i.e. reports coming from multiple data sources. <\/p>\n\n\n\n
You need to remember though that similar to RAMDirectory<\/em>, the ByteBuffersDirectory<\/em> doesn’t save data to disk. This means that you need to re-index your data after each Solr restart or failure. <\/p>\n\n\n\n
Advantages<\/h2>\n\n\n\n
The main advantage, that we already mentioned, is that the data indexed to the core\/collection that uses ByteBuffersDirectory<\/em> will not use disk resources. That means that the data write and read will be blazingly fast, which can result in higher indexing throughput and lower query latency. However keep in mind that similar behavior can be achieved when using mmap<\/em> calls to the I\/O subsystem of the operating system, so for example when using the MMapDirectory<\/em>. This Directory<\/em> implementation shares the I\/O cache of the operating system and will re-use data that is put in memory by the operating system. The second advantage, especially when compared to the RAMDirectory<\/em> is that the ByteBuffersDirectory<\/em> fully supports multi-threading, so we won’t have issues when using the multi-threaded data indexing and access.<\/p>\n\n\n\n
Drawbacks<\/h2>\n\n\n\n
Of course not everything is bright and shining, there are also drawbacks when using the ByteBuffersDirectory<\/em> – we should mention three things here. First of all, when using this directory, the data will be volatile – every restart will require data re-indexing. It is one of the principles of that Directory<\/em> implementation, but it is worth remembering about that. The second thing is heap memory. If the data will be stored on heap it will mean more heap memory usage and more work for the garbage collector, which can lead to less performance. Finally – we are partially limited by the amount of physical memory and the memory assigned to the JVM heap. This means that the core\/collections that are using ByteBuffersDirectory<\/em> shouldn’t be too large.<\/p>\n","protected":false},"excerpt":{"rendered":"
One of the new features introduced in the recently released Solr 8.0 is new implementation of the Directory interface – one that will replace not scalable RAMDirectory. The new implementation called ByteBuffersDirectory is dedicated to small, short lived data that<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[27],"tags":[617,619,164],"class_list":["post-990","post","type-post","status-publish","format-standard","hentry","category-solr-en","tag-8-0-2","tag-directory-2","tag-solr-2"],"_links":{"self":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/990","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/comments?post=990"}],"version-history":[{"count":1,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/990\/revisions"}],"predecessor-version":[{"id":991,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/990\/revisions\/991"}],"wp:attachment":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/media?parent=990"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/categories?post=990"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/tags?post=990"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}