{"id":268,"date":"2011-05-23T20:46:24","date_gmt":"2011-05-23T18:46:24","guid":{"rendered":"http:\/\/sematext.solr.pl\/?p=268"},"modified":"2020-11-11T20:46:53","modified_gmt":"2020-11-11T19:46:53","slug":"car-sale-application-spellcheckcomponent-did-you-really-mean-that-part-5","status":"publish","type":"post","link":"https:\/\/solr.pl\/en\/2011\/05\/23\/car-sale-application-spellcheckcomponent-did-you-really-mean-that-part-5\/","title":{"rendered":"\u201cCar sale application\u201d \u2013 SpellCheckComponent \u2013 did you really mean that ? (part 5)"},"content":{"rendered":"

The time has come to add another important functionality to our car sale application. It will be the spell checking mechanism with the ability to construct a new query from the suggestions. It has become the main functionality of every search engine so we will also make use of it.<\/p>\n\n\n\n\n\n

Requirements specification<\/h2>\n

Our car database is so large that it contains many different names of makes and models. Some of that names could be really hard to spell\/write:<\/p>\n

    \n
  1. \n
      \n
    • make<\/em>: Bugatti<\/li>\n
    • model<\/em>: Veyron<\/li>\n<\/ul>\n<\/li>\n
    • \n
        \n
      • make<\/em>: Daewoo<\/li>\n
      • model<\/em>: Lacetti<\/li>\n<\/ul>\n<\/li>\n
      • \n
          \n
        • make<\/em>: Cadillac<\/li>\n
        • model<\/em>: Brougham<\/li>\n<\/ul>\n<\/li>\n
        • \n
            \n
          • make<\/em>: Ford<\/li>\n
          • model<\/em>: Capri<\/li>\n<\/ul>\n<\/li>\n
          • \n
              \n
            • make<\/em>: Maserati<\/li>\n
            • model<\/em>: Coupe<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

              The query examples, where misspelled words caused the query to provide no search results:<\/p>\n

                \n
              1. ?q=bugati+weyron<\/li>\n
              2. ?q=daewo+laceti<\/li>\n
              3. ?q=cadilac+brogham<\/li>\n
              4. ?q=ford+kapri<\/li>\n
              5. ?q=maseratti+coupe<\/li>\n<\/ol>\n

                We would like to add the functionality, that in case of entering incorrect names will be able to suggest the phrase which probably was the intention of an application user. Then we will be able to make use of it to find the documents related to the proper phrase.<\/p>\n

                solrconfig.xml changes<\/h2>\n

                The most important element, which should be added to the solrconfig.xml configuration file is the solr.SpellCheckComponent<\/em>. Let’s try to add the simple standard configuration of this component and find out how it works:\n<\/p>\n

                <searchComponent name=\"spellcheck\" class=\"solr.SpellCheckComponent\">\n    <lst name=\"spellchecker\">\n      <str name=\"classname\">solr.IndexBasedSpellChecker<\/str>\n      <str name=\"spellcheckIndexDir\">.\/spellchecker<\/str>\n      <str name=\"field\">content<\/str>\n      <str name=\"buildOnCommit\">true<\/str>\n    <\/lst>\n<\/searchComponent><\/pre>\n
                Let’s explain the attributes used in this component:<\/p>\n
                  \n
                1. \n
                    \n
                  • classname<\/em> \u2013 the name of the class which is the implementation of our spellcheck mechanism. We are using the solr.IndexBasedSpellChecker class,  which use a spelling dictionary that is based on the Solr\/Lucene index.<\/li>\n<\/ul>\n
                      \n
                    • spellcheckIndexDir<\/em> \u2013 the directory name which holds the spellcheck index.<\/li>\n<\/ul>\n
                        \n
                      • field<\/em> \u2013 the name of the field defined in the schema.xml file, used as the  source field to generate the spellcheck index. In our case it will be the \u201ccontent\u201d field (why? – it will be explained later).<\/li>\n<\/ul>\n
                          \n
                        • buildOnCommit<\/em> \u2013 if the value of this attribute is set to true<\/em>, then the spellcheck index will be automatically build after the main solr index commit.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
                          Now when we have our component defined, let’s add it to some handler to be able to make use of it. The best option is to add it to our standard, default handler, which would provide query results with the suggestions when hitting only one request. Before the changes, our default handler looked like this:\n<\/p>\n
                          <requestHandler name=\"standard\" class=\"solr.SearchHandler\" default=\"true\">\n     <lst name=\"defaults\">\n       <str name=\"echoParams\">explicit<\/str>\n     <\/lst>\n<\/requestHandler><\/pre>\n
                          Po zmianie, wygl\u0105da tak:<\/p>\n
                          <requestHandler name=\"standard\" class=\"solr.SearchHandler\" default=\"true\">\n     <lst name=\"defaults\">\n       <str name=\"echoParams\">explicit<\/str>\n       <str name=\"spellcheck\">true<\/str>\n       <str name=\"spellcheck.collate\">true<\/str>\n     <\/lst>\n     <arr name=\"last-components\">\n       <str>spellcheck<\/str>\n     <\/arr>\n<\/requestHandler><\/pre>\n
                          As we can see, we have added the spellcheck component and yet two another default values:<\/p>\n
                            \n
                          1. \n
                              \n
                            • spellcheck<\/em> \u2013 when set to true<\/em> causes every request should also generate a spellcheck suggestion.<\/li>\n<\/ul>\n
                                \n
                              • spellcheck.collate<\/em> – when set to true<\/em> causes the mechanism to choose the best suggestion for every word entered and to construct a new query containing proper words. If the spellchecker recognises a word to be correct, it leaves it unchanged.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
                                schema.xml changes<\/h2>\n
                                The possible changes in the schema.xml configuration file would be to add the field which would be used by the solr.SpellCheckComponent<\/em> component as the source of tokens used for spell checking. The field should contain the data which we would like to be used when creating the spellcheck index. The type of that field should ensure the proper data tokenization. It should be also out of any stemming\/lametization filters that could affect the spellcheck results badly.<\/p>\n
                                Our schema already contains the field which fulfil all those requirements – \u201ccontent\u201d field. Just to remind, it is the default search field used by our search engine. The current field and type definitions look like this:\n<\/p>\n
                                <field name=\"content\" type=\"text\" indexed=\"true\" stored=\"false\" multiValued=\"true\"\/><\/pre>\n
                                <fieldType name=\"text\" positionIncrementGap=\"100\">\n <analyzer>\n  <tokenizer class=\"solr.WhitespaceTokenizerFactory\"\/>\n   <filter class=\"solr.PatternReplaceFilterFactory\" pattern=\"'\" replacement=\"\" replace=\"all\" \/>\n   <filter class=\"solr.WordDelimiterFilterFactory\"\n    generateWordParts=\"1\"\n    generateNumberParts=\"1\"\n    catenateWords=\"1\"\n    stemEnglishPossessive=\"0\"\n  \/>\n  <filter class=\"solr.LowerCaseFilterFactory\"\/>\n <\/analyzer>\n<\/fieldType><\/pre>\n
                                There are values of three fields copied to the “content” field: make, model and year:\n<\/p>\n
                                <copyField source=\"make\" dest=\"content\"\/>\n<copyField source=\"model\" dest=\"content\"\/>\n<copyField source=\"year\" dest=\"content\"\/><\/pre>\n
                                Let\u2019s create queries<\/h2>\n
                                Let’s take the no results queries from the requirements specification and add the spellcheck.q parameter which value will be the same as entered in the q parameter. Now, hitting only one query, we are able to get the search results wit the spellcheck suggestions:<\/p>\n
                                  \n
                                1. ?q=bugati+weyron&spellcheck.q=bugati+weyron\n
                                    \n
                                    <result name=\"response\" numFound=\"0\" start=\"0\" \/>\n<lst name=\"spellcheck\">\n  <lst name=\"suggestions\">\n    <lst name=\"bugati\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">0<\/int>\n      <int name=\"endOffset\">6<\/int>\n      <arr name=\"suggestion\">\n        <str>bugatti<\/str>\n      <\/arr>\n    <\/lst>\n    <lst name=\"weyron\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">7<\/int>\n      <int name=\"endOffset\">13<\/int>\n      <arr name=\"suggestion\">\n        <str>veyron<\/str>\n      <\/arr>\n    <\/lst>\n      <str name=\"collation\">bugatti veyron<\/str>\n    <\/lst>\n<\/lst><\/pre>\n
                                    The spellcheck mechanism has corrected the query tokens and the collation functionality has generated the proper phrase query, which now can be simply used in order to provide us the proper search results. Let’s check the rest of the queries:<\/p>\n<\/ul>\n<\/li>\n
                                  • ?q=daewo+laceti&spellcheck.q=?q=daewo+laceti\n
                                      \n
                                      <result name=\"response\" numFound=\"0\" start=\"0\" \/>\n<lst name=\"spellcheck\">\n  <lst name=\"suggestions\">\n    <lst name=\"daewo\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">0<\/int>\n      <int name=\"endOffset\">5<\/int>\n      <arr name=\"suggestion\">\n        <str>daewoo<\/str>\n      <\/arr>\n    <\/lst>\n    <lst name=\"laceti\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">6<\/int>\n      <int name=\"endOffset\">12<\/int>\n      <arr name=\"suggestion\">\n        <str>lacetti<\/str>\n      <\/arr>\n    <\/lst>\n      <str name=\"collation\">daewoo lacetti<\/str>\n    <\/lst>\n<\/lst><\/pre>\n<\/ul>\n<\/li>\n
                                    • ?q=cadilac+brogham&spellcheck.q=cadilac+brogham\n
                                        \n
                                        <result name=\"response\" numFound=\"0\" start=\"0\" \/>\n<lst name=\"spellcheck\">\n  <lst name=\"suggestions\">\n    <lst name=\"cadilac\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">0<\/int>\n      <int name=\"endOffset\">7<\/int>\n      <arr name=\"suggestion\">\n        <str>cadillac<\/str>\n      <\/arr>\n    <\/lst>\n    <lst name=\"brogham\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">8<\/int>\n      <int name=\"endOffset\">15<\/int>\n      <arr name=\"suggestion\">\n        <str>brougham<\/str>\n      <\/arr>\n    <\/lst>\n      <str name=\"collation\">cadillac brougham<\/str>\n    <\/lst>\n<\/lst><\/pre>\n<\/ul>\n<\/li>\n
                                      • ?q=ford+kapri& spellcheck.q=?q=ford+kapri\n
                                          \n
                                          <result name=\"response\" numFound=\"0\" start=\"0\" \/>\n<lst name=\"spellcheck\">\n  <lst name=\"suggestions\">\n    <lst name=\"kapri\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">5<\/int>\n      <int name=\"endOffset\">10<\/int>\n      <arr name=\"suggestion\">\n        <str>capri<\/str>\n      <\/arr>\n    <\/lst>\n      <str name=\"collation\">ford capri<\/str>\n    <\/lst>\n<\/lst><\/pre>\n<\/ul>\n<\/li>\n
                                        • ?q=maseratti+coupe&spellcheck.q=?q=maseratti+coupe\n
                                            \n
                                            <result name=\"response\" numFound=\"0\" start=\"0\" \/>\n<lst name=\"spellcheck\">\n  <lst name=\"suggestions\">\n    <lst name=\"maseratti\">\n      <int name=\"numFound\">1<\/int>\n      <int name=\"startOffset\">0<\/int>\n      <int name=\"endOffset\">9<\/int>\n      <arr name=\"suggestion\">\n        <str>maserati<\/str>\n      <\/arr>\n    <\/lst>\n      <str name=\"collation\">maserati coupe<\/str>\n    <\/lst>\n<\/lst><\/pre>\n<\/ul>\n<\/li>\n<\/ol>\n
                                            The spellcheck mechanism has worked great, correcting all of the misspellings and generating the proper phrase queries. In the last two cases (4,5) we can see that the component has not corrected the properly entered words (4 \u2013 ford, 5 \u2013 coupe) but used them to construct the proper queries (collation).<\/p>\n
                                            The end<\/h2>\n
                                            Our search engine has now yet another functionality. This time it was the spell checking mechanism. Now all we have to do is to wait for some comments \u2026 and maybe some improvements can be provided \ud83d\ude42<\/p>","protected":false},"excerpt":{"rendered":"
                                            The time has come to add another important functionality to our car sale application. It will be the spell checking mechanism with the ability to construct a new query from the suggestions. It has become the main functionality of every<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[27],"tags":[],"class_list":["post-268","post","type-post","status-publish","format-standard","hentry","category-solr-en"],"_links":{"self":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/268","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/comments?post=268"}],"version-history":[{"count":1,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/268\/revisions"}],"predecessor-version":[{"id":269,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/posts\/268\/revisions\/269"}],"wp:attachment":[{"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/media?parent=268"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/categories?post=268"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/solr.pl\/en\/wp-json\/wp\/v2\/tags?post=268"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}