\u201cCar sale application\u201d users started to use wildard queries more and more often. This fact forced us to think about wildcard queries optimization. solr.ReversedWildcardFilter comes to rescue us.<\/p>\n\n\n\n\n\n
The solr.ReversedWildcardFilter filter provides us with new tokens, which in fact are reverses tokens, that are indexed to provide faster leading wildcard queries. The filter supports the following init arguments:<\/p>\n
New filter is added to the \u201ctext\u201d field type:\n<\/p>\n
<fieldType name=\"text\" class=\"solr.TextField\"\n\tpositionIncrementGap=\"100\">\n\t<analyzer type=\"index\">\n\t\t<tokenizer class=\"solr.WhitespaceTokenizerFactory\" \/>\n\t\t<filter class=\"solr.PatternReplaceFilterFactory\" pattern=\"'\"\n\t\t\treplacement=\"\" replace=\"all\" \/>\n\t\t<filter class=\"solr.WordDelimiterFilterFactory\"\n\t\t\tgenerateWordParts=\"1\" generateNumberParts=\"1\" catenateWords=\"1\"\n\t\t\tstemEnglishPossessive=\"0\" \/>\n\t\t<filter class=\"solr.LowerCaseFilterFactory\" \/>\n\t\t<filter class=\"solr.ReversedWildcardFilterFactory\" \/><\/strong>\n\t<\/analyzer>\n\t<analyzer type=\"query\">\n\t\t<tokenizer class=\"solr.WhitespaceTokenizerFactory\" \/>\n\t\t<filter class=\"solr.PatternReplaceFilterFactory\" pattern=\"'\"\n\t\t\treplacement=\"\" replace=\"all\" \/>\n\t\t<filter class=\"solr.WordDelimiterFilterFactory\"\n\t\t\tgenerateWordParts=\"1\" generateNumberParts=\"1\" catenateWords=\"1\"\n\t\t\tstemEnglishPossessive=\"0\" \/>\n\t\t<filter class=\"solr.LowerCaseFilterFactory\" \/>\n\t<\/analyzer>\n<\/fieldType><\/pre>\nsolr.ReversedWildcardFilterFactory filter is added only to the index analyzer. We do not define any arguments in the filter definition, because we would like to use the default configuration, which is:<\/p>\n
\n
- withOriginal<\/em> – \u201etrue\u201d, we would like to produce original tokens<\/li>\n
- maxPosAsterisk<\/em> – 2<\/li>\n
- maxPosQuestion<\/em> – 1<\/li>\n
- maxPosQuestion<\/em> – 0.0f (disabled)<\/li>\n
- maxPosQuestion<\/em> – 2<\/li>\n<\/ul>\n
Sample data<\/h3>\n
Let’s index some sample data:\n<\/p>\n
<add>\n <doc>\n <field name=\"id\">1<\/field>\n <field name=\"make\">Lancia<\/field>\n <field name=\"model\">Delta<\/field>\n ...\n <\/doc>\n <doc>\n <field name=\"id\">2<\/field>\n <field name=\"make\">Land Rover<\/field>\n <field name=\"model\">Defender<\/field>\n ...\n <\/doc>\n <doc>\n <field name=\"id\">3<\/field>\n <field name=\"make\">Acura<\/field>\n <field name=\"model\">MDX<\/field>\n ...\n <\/doc>\n <doc>\n <field name=\"id\">4<\/field>\n <field name=\"make\">Acura<\/field>\n <field name=\"model\">RDX<\/field>\n ...\n <\/doc>\n <doc>\n <field name=\"id\">5<\/field>\n <field name=\"make\">Acura<\/field>\n <field name=\"model\">RSX<\/field>\n ...\n <\/doc>\n<\/add><\/pre>\nLet’s create queries<\/h3>\n
Let me remind you that the default search field is the \u201ccontent\u201d field, that among others contains \u201cmake\u201d and \u201cmodel\u201d field. To analyse query results and solr.ReversedWildcardFilter filter behaviour, we will set the \u201estored\u201d argument of the \u201econtent\u201d field to \u201ctrue\u201d. We will also add the debugQuery query argument, which will allow us to find out, which tokens are used in the query processing (original or reversed).<\/p>\n
\n
- ?q=lan*&fl=id,content&debugQuery=on\n
<result name=\"response\" numFound=\"2\" start=\"0\">\n <doc>\n <arr name=\"content\">\n <str>Lancia<\/str>\n <str>Delta<\/str>\n <str>2002<\/str>\n <\/arr>\n <str name=\"id\">1<\/str>\n <\/doc>\n <doc>\n <arr name=\"content\">\n <str>Land Rover<\/str>\n <str>Defender<\/str>\n <str>2002<\/str>\n <\/arr>\n <str name=\"id\">2<\/str>\n <\/doc>\n<\/result>\n<lst name=\"debug\">\n <str name=\"rawquerystring\">lan*<\/str>\n <str name=\"querystring\">lan*<\/str>\n <str name=\"parsedquery\">content:lan*<\/str>\n <str name=\"parsedquery_toString\">content:lan*<\/str>\n ...\n<\/lst><\/pre>\nWe have used asterisk wildcard (‘*’) at the end of the query (position = 4), so the original tokens were used:\n<\/p>\n
<str name=\"parsedquery\">content:lan*<\/str><\/pre>\n<\/li>\n- ?q=*dx&fl=id,content&debugQuery=on\n
<result name=\"response\" numFound=\"2\" start=\"0\">\n <doc>\n <arr name=\"content\">\n <str>Acura<\/str>\n <str>MDX<\/str>\n <str>2002<\/str>\n <\/arr>\n <str name=\"id\">3<\/str>\n <\/doc>\n <doc>\n <arr name=\"content\">\n <str>Acura<\/str>\n <str>RDX<\/str>\n <str>2003<\/str>\n <\/arr>\n <str name=\"id\">4<\/str>\n <\/doc>\n<\/result>\n<lst name=\"debug\">\n <str name=\"rawquerystring\">*dx<\/str>\n <str name=\"querystring\">*dx<\/str>\n <str name=\"parsedquery\">content:#1;xd*<\/str>\n <str name=\"parsedquery_toString\">content:#1;xd*<\/str>\n ...\n<\/lst><\/pre>\nWe have used asterisk wildcard (‘*’) at the beginning of the query (position = 1) and additionally we have two trailing characters after the last wildcard. That’s why the revesed tokens were used:\n<\/p>\n
<str name=\"parsedquery\">content:#1;xd*<\/str><\/pre>\nAs we can see, the reversed tokens have a special prefix in order to avoid collisions and false matches.<\/p>\n<\/li>\n
- ?q=r?x&fl=id,content&debugQuery=on\n
<result name=\"response\" numFound=\"2\" start=\"0\">\n <doc>\n <arr name=\"content\">\n <str>Acura<\/str>\n <str>RDX<\/str>\n <str>2003<\/str>\n <\/arr>\n <str name=\"id\">4<\/str>\n <\/doc>\n <doc>\n <arr name=\"content\">\n <str>Acura<\/str>\n <str>RSX<\/str>\n <str>2006<\/str>\n <\/arr>\n <str name=\"id\">5<\/str>\n <\/doc>\n<\/result>\n<lst name=\"debug\">\n <str name=\"rawquerystring\">r?x<\/str>\n <str name=\"querystring\">r?x<\/str>\n <str name=\"parsedquery\">content:r?x<\/str>\n <str name=\"parsedquery_toString\">content:r?x<\/str>\n ...\n<\/lst><\/pre>\nWe have used question mark wildcard (‘?’) on position number 2 and additionally we have only one trailing character after the wildcard. The original tokens were used:\n<\/p>\n
<str name=\"parsedquery\">content:r?x<<\/str><\/pre>\n<\/li>\n<\/ol>\nThe end<\/h3>\n
Thanks to the solr.ReversedWildcardFilter filter, we have successfully optimized wildcard queries. \u201cCar sale application\u201d users can now effectively use them \ud83d\ude42<\/p>","protected":false},"excerpt":{"rendered":"
\u201cCar sale application\u201d users started to use wildard queries more and more often. This fact forced us to think about wildcard queries optimization. solr.ReversedWildcardFilter comes to rescue us.<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[27],"tags":[],"class_list":["post-372","post","type-post","status-publish","format-standard","hentry","category-solr-en"],"_links":{"self":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/372","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/comments?post=372"}],"version-history":[{"count":1,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/372\/revisions"}],"predecessor-version":[{"id":373,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/372\/revisions\/373"}],"wp:attachment":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/media?parent=372"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/categories?post=372"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/tags?post=372"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}