{"id":1020,"date":"2020-04-13T16:18:32","date_gmt":"2020-04-13T14:18:32","guid":{"rendered":"http:\/\/sematext.solr.pl\/?p=1020"},"modified":"2020-11-14T16:18:55","modified_gmt":"2020-11-14T15:18:55","slug":"solr-8-4-0-plugin-management","status":"publish","type":"post","link":"https:\/\/solr.pl\/en\/2020\/04\/13\/solr-8-4-0-plugin-management\/","title":{"rendered":"Solr 8.4.0 – Plugin Management"},"content":{"rendered":"\n

With the release of Solr 8.4.0 we’ve got a new functionality that we can use that can help us with extending Solr functionality – plugin management. Starting with the mentioned version we can install and remove plugins that are downloaded from external plugin repositories. Let’s see how it works. <\/p>\n\n\n\n\n\n\n\n

Plugin Management<\/h2>\n\n\n\n

With the Solr 8.4.0 we didn’t only get the script itself but also the whole package of changes such as package management, class loader isolation, artifact read and write API and so on. <\/p>\n\n\n\n

Let’s start from the beginning though. By default Solr comes with the package loading turned off and to be able to use that feature we need to run Solr with the enable.packges<\/strong> property set to true<\/strong>:<\/p>\n\n\n\n

$ bin\/solr start -c -f -Denable.packages=true<\/pre>\n\n\n\n
Now we can start playing around with the packages.<\/p>\n\n\n\n
Package Management Basics<\/h2>\n\n\n\n
Let’s try using the bin\/solr<\/strong> script and type in the following command:<\/p>\n\n\n\n
$ bin\/solr package <\/pre>\n\n\n\n
In result we will get the following response:<\/p>\n\n\n\n
Found 1 Solr nodes:\n \nSolr process 20949 running on port 8983\nPackage Manager\n\n.\/solr package add-repo  \nAdd a repository to Solr.\n\n.\/solr package install [:]\nInstall a package into Solr. This copies over the artifacts from the repository into Solr's internal package store and sets up classloader for this package to be used.\n\n.\/solr package deploy [:] [-y] [--update] -collections <package-name>[:] [-y] [--update] -collections  [-p param1=value1 -p param2=value2 \u2026\nBootstraps a previously installed package into the specified collections. It the package accepts parameters for its setup commands, they can be specified (as per package documentation).\n\n.\/solr package list-installed\nPrint a list of packages installed in Solr.\n\n.\/solr package list-available\nPrint a list of packages available in the repositories.\n\n.\/solr package list-deployed -c \nPrint a list of packages deployed on a given collection.\n\n.\/solr package list-deployed \nPrint a list of collections on which a given package has been deployed.\n\n.\/solr package undeploy  -collections \nUndeploys a package from specified collection(s)\n\nNote: (a) Please add '-solrUrl http:\/\/host:port' parameter if needed (usually on Windows).\n      (b) Please make sure that all Solr nodes are started with '-Denable.packages=true' parameter.<\/pre>\n\n\n\n
As you can see the number of options seems to fulfill the needs and the mechanism itself verifies if Solr node is running and it is able to check the installed packages. Let’s look into that.<\/p>\n\n\n\n
Adding New Repository<\/h2>\n\n\n\n
To add a new repository we need to provide its name and the URL under which it is available. Of course Solr expects at least a single file to be present in the defined locations – the repository.json<\/strong> file. The mentioned file can look as follows (taken from the Solr source code):<\/p>\n\n\n\n
[\n  {\n    \"name\": \"question-answer\",\n    \"description\": \"A natural language question answering plugin\",\n    \"versions\": [\n      {\n        \"version\": \"1.0.0\",\n        \"date\": \"2019-01-01\",\n        \"artifacts\": [\n          {\n            \"url\": \"qarh-1.0.jar\",\n            \"sig\": \"C9UWKkucmY3UNzqn0VLneVMe9kCbJjw7Urc76vGenoRwp32xvNn5ZIGZ7G34xZP7cVjqn\/ltDlLWBZ\/C3eAtuw==\"\n          }\n        ],\n        \"manifest\": {\n          \"version-constraint\": \"8 - 9\",\n          \"plugins\": [\n            {\n              \"name\": \"request-handler\",\n              \"setup-command\": {\n                \"path\": \"\/api\/collections\/${collection}\/config\",\n                \"payload\": {\"add-requesthandler\": {\"name\": \"${RH-HANDLER-PATH}\", \"class\": \"question-answer:fullstory.QARequestHandler\"}},\n                \"method\": \"POST\"\n              },\n              \"uninstall-command\": {\n                \"path\": \"\/api\/collections\/${collection}\/config\",\n                \"payload\": {\"delete-requesthandler\": \"${RH-HANDLER-PATH}\"},\n                \"method\": \"POST\"\n              },\n              \"verify-command\": {\n                \"path\": \"\/api\/collections\/${collection}\/config\/requestHandler?componentName=${RH-HANDLER-PATH}&meta=true\",\n                \"method\": \"GET\",\n                \"condition\": \"$['config'].['requestHandler'].['${RH-HANDLER-PATH}'].['_packageinfo_'].['version']\",\n                \"expected\": \"${package-version}\"\n              }\n            }\n          ],\n          \"parameter-defaults\": {\n            \"RH-HANDLER-PATH\": \"\/mypath\"\n          }\n        }\n      }\n    ]\n  }\n]<\/code><\/pre>\n\n\n\n
OK, let’s try adding the repository then. For that purpose I created a new repository at the http:\/\/repo.solr.pl<\/strong> using the files that I found in the Solr source code<\/a> and I used the following command to add that repository to my local SolrCloud cluster:<\/p>\n\n\n\n
$ bin\/solr package add-repo solrpl http:\/\/repo.solr.pl<\/code><\/pre>\n\n\n\n
In my case the operation was successful and I got the following response on my console:<\/p>\n\n\n\n
Added repository: solrpl<\/code><\/pre>\n\n\n\n
Warning<\/strong>: in the above example I used a repository that I could access without using SSL. Please don’t do that in your environment. It is not a good practice – it is susceptible to the man in the middle<\/em> attacks during which the downloaded files can be replaced.  <\/p>\n\n\n\n
Installing and Removing Packages<\/h2>\n\n\n\n
After adding the repository we can list the packages that we can install and use. To list such packages we should run the following command:<\/p>\n\n\n\n
$ bin\/solr package list-available<\/code><\/pre>\n\n\n\n
The response to the above command should be similar to the following one:<\/p>\n\n\n\n
Available packages:\n-----\nquestion-answer \t\tA natural language question answering plugin\n\tVersion: 1.0.0<\/code><\/pre>\n\n\n\n
One last step before installing the package itself is adding a public key to Solr – one that will be able to verify the packages downloaded from the added repository. Where to look for such a key? It will be provided to you and I assume that some repositories can also have such. Once we will have the key we can add it to Solr by using the following command:<\/p>\n\n\n\n
$ bin\/solr package add-key publickey.der<\/code><\/pre>\n\n\n\n
After all this steps we can finally start installing the packages. It is as simple as running the following command:<\/p>\n\n\n\n
$ bin\/solr package install question-answer:1.0.0<\/code><\/pre>\n\n\n\n
The response that I got from Solr was as follows:<\/p>\n\n\n\n
Posting manifest...\nPosting artifacts...\nExecuting Package API to register this package...\nResponse: {\"responseHeader\":{\n    \"status\":0,\n    \"QTime\":66}}\nquestion-answer installed.<\/code><\/pre>\n\n\n\n
This means that our package is now ready to be used. We just need to say which  collection or collections should be allowed to use that package. We do that by sending the deploy<\/em> command to Solr, for example by using the following command (I created the test<\/em> collection before running this command):<\/p>\n\n\n\n
$ bin\/solr package deploy question-answer:1.0.0 -collections test<\/code><\/pre>\n\n\n\n
The response was as follows:<\/p>\n\n\n\n
Executing {\"add-requesthandler\":{\"name\":\"\/mypath\",\"class\":\"question-answer:fullstory.QARequestHandler\"}} for path:\/api\/collections\/test\/config\nExecute this command (y\/n):\ny\nExecuting http:\/\/localhost:8983\/api\/collections\/test\/config\/requestHandler?componentName=\/mypath&meta=true for collection:test\n{\n  \"responseHeader\":{\n    \"status\":0,\n    \"QTime\":0},\n  \"config\":{\"requestHandler\":{\"\/mypath\":{\n        \"name\":\"\/mypath\",\n        \"class\":\"question-answer:fullstory.QARequestHandler\",\n        \"_packageinfo_\":{\n          \"package\":\"question-answer\",\n          \"version\":\"1.0.0\",\n          \"files\":[\"\/package\/question-answer\/1.0.0\/question-answer-request-handler-1.0.jar\"],\n          \"manifest\":\"\/package\/question-answer\/1.0.0\/manifest.json\",\n          \"manifestSHA512\":\"a91ab5a2c5abd53f0f72c256592c2be8b667cecb8226ac054aeed4d28aac9d743311442f2d58539bb83663a19bd1efb310aaadfd77bea458f3d475161721a114\"}}}}}\n\nActual: 1.0.0, expected: 1.0.0\nDeployed on [test] and verified package: question-answer, version: 1.0.0\nDeployment successful<\/code><\/pre>\n\n\n\n
If everything went well and we said that we want to add the new package to our collection we are ready \ud83d\ude42 We can start using it – for example by using Solr API to create new handlers if the plugin adds them. <\/p>\n\n\n\n
When we no longer need a package we just need to undeploy<\/em> it. For example, if we would like to remove the previously deployed package we just need to run the following command:<\/p>\n\n\n\n
$ bin\/solr package undeploy question-answer -collections test<\/code><\/pre>\n\n\n\n
In this case the response would be as follows:<\/p>\n\n\n\n
Executing {\"delete-requesthandler\":\"\/mypath\"} for path:\/api\/collections\/test\/config<\/code><\/pre>\n\n\n\n
How It Really Works<\/h2>\n\n\n\n
The heart of the implementation is related to class path and isolation of the plugins and the core Solr classes. The plugin mechanism assumes that any change in the files that are in the Solr classpath<\/em> requires restart. The rest files can be loaded dynamically and are bound to the configuration stored in Zookeeper. <\/p>\n\n\n\n
The basis of the mechanism is a so called Package Store<\/em>. It is a distributed file system which keeps its data on each Solr node in the $SOLR_HOME\/filestore<\/em> directory and each of the files is described by metadata written in a JSON file. Of course, each file stores the checksum in its metadata for verification purposes.  <\/p>\n\n\n\n
On top of all of that we have an API allowing us not only to manage the whole packages, but also single files. <\/p>\n\n\n\n
The API<\/h2>\n\n\n\n
Of course our bin\/solr<\/strong> tool and installing the packages using it to manage packages is not everything that Solr give us. In addition to that we got the API that allows us to:<\/p>\n\n\n\n