With the Solr 8.7 release, we were given a very useful feature called circuit breakers. A circuit breaker design pattern allows stopping execution when certain criteria are met. For example, when the memory usage is higher than defined a query execution may be stopped, or when the CPU usage on a given node is too high. Let’s look at what Solr 8.7 brings us.<\/p>\n\n\n\n\n\n\n\n
New Functionality<\/h2>\n\n\n\n
The new functionality that was introduced to Solr aims to prevent the execution of a request that causes a node to go beyond defined thresholds. For example, when the memory utilization of the JVM reaches 75% you may want to stop all requests that are hitting Solr for the ones that are processed to finish.<\/p>\n\n\n\n
That’s why with the Solr 8.7 a circuit breaker code was added with two circuit breaker implementations:<\/p>\n\n\n\n
JVM memory-based circuit breaker<\/li>
CPU utilization based circuit breaker<\/li><\/ul>\n\n\n\n
So when to use the circuit breaker functionality? When you want to trade-in request throughput for stability. If you want more stability from your Solr nodes you include circuit breakers, if you want to go full throttle – you don’t.<\/p>\n\n\n\n
Circuit Breaker Configuration<\/h2>\n\n\n\n
The circuit breaker configuration should be included in the solrconfig.xml<\/strong> and should be included inside the circuitBreaker<\/strong> tag:<\/p>\n\n\n\n
The enabled<\/strong> attribute turns the circuit breaker functionality on and off globally. When set to true<\/strong> they are enabled, when set to false<\/strong> they are disabled. <\/p>\n\n\n\n
At the moment of writing this blog post we could use two circuit breakers:<\/p>\n\n\n\n
CPU based<\/li>
JVM memory based<\/li><\/ul>\n\n\n\n
The CPU based one tracks the CPU utilization and checks the average CPU usage over the last minute. If that crossed the defined threshold the circuit breaker is tripped and the execution of the request will be prevented. To enable it we would include the following properties inside the circuitBreaker<\/strong> tag:<\/p>\n\n\n\n
The first property enables the CPU based circuit breaker and the second one specifies the threshold.<\/p>\n\n\n\n
The JVM memory-based circuit breaker tracks the memory usage of the JVM and rejects the execution of the request if the usage is above the percentage of the maximum heap size – the one defined with the Xmx<\/strong>. To enable it we would include the following properties inside the circuitBreaker<\/strong> tag:<\/p>\n\n\n\n
The above configuration means that the JVM memory circuit breaker is enabled and it will be triggered when the JVM heap usage is above 80%. So if our Solr heap would be set to a maximum size of 10G the circuit breaker would prevent the execution of requests if the usage is above 8G.<\/p>\n\n\n\n
One thing to keep in mind is that the memThreshold<\/strong> can take the values between 50<\/strong> and 95<\/strong>. <\/p>\n\n\n\n
Our final configuration would look as follows:<\/p>\n\n\n\n
The response code 503<\/strong> and the message about the tripped circuit breaker. In the above case, it was a memory circuit breaker. We should back-off and wait for the next request. An exponential back-off is a good practice. Though the circuit breakers do not add any significant overhead too many checks can cause performance overhead. <\/p>\n\n\n\n
The Downsides<\/h2>\n\n\n\n
Having circuit breakers in Solr doesn’t mean that you can’t go into out of memory situations. And this is the first downside of the functionality. To simulate that I experimented.<\/p>\n\n\n\n
![\"\"](\"https:\/\/solr.pl\/wp-content\/uploads\/2020\/11\/solr_circuit_breaker-1024x324.png\")
<\/figure>\n\n\n\n