Solr Text Tagger – Quick Look into Solr 7.4

Rafał Kuć Solr , , , , 0 Comment

We haven’t looked into any Solr functionalities on Solr.pl for a while so it is time to look into one of the new features – the text tagger. It works on the basis of Solr index and is able to check documents sent to a new handler and return occurrences of names with offsets and other metadata that we added to Solr index. However keep in mind that Solr Text Tagger doesn’t do any kind of natural language processing, so no what we will give it, that is what we can expect to get back.

Preparations

We will keep it as minimal as it can be. We start with running our Solr instance:

$ bin/solr start -c -f

And we create a test collection:

$ bin/solr create -c test

We will be using the standard data-driven schema. Though it is not suggested to be used in production, it is more than enough for our Text Tagger test.

Data For Tagging

Let’s focus on the data for a second. First of all, we need data that will be used for tagging. For the purpose of the blog post we will just go with simple name tagging, for example:

Thomas Jefferson
Alexander Hamilton
George Washington
John Adams
James Wilkinson

A few names – you may recognize them 😉 We will use those names for tagging purposes.

To index them to Solr we will need a new field type, two fields, and a copy field, just like this:

$ curl -XPOST -H 'Content-type:application/json'  'http://localhost:8983/solr/test/schema' -d '{
  "add-field-type":{
    "name":"tag",
    "class":"solr.TextField",
    "postingsFormat":"FST50",
    "omitNorms":true,
    "omitTermFreqAndPositions":true,
    "indexAnalyzer":{
      "tokenizer":{ "class":"solr.StandardTokenizerFactory" },
      "filters":[
        {"class":"solr.LowerCaseFilterFactory"},
        {"class":"solr.ConcatenateGraphFilterFactory", "preservePositionIncrements":false }
      ]},
    "queryAnalyzer":{
      "tokenizer":{
         "class":"solr.StandardTokenizerFactory" },
      "filters":[
        {"class":"solr.LowerCaseFilterFactory"}
      ]}
    },
  "add-field":{"name":"name", "type":"text_general"},
  "add-field":{"name":"name_tag", "type":"tag", "stored":false},
  "add-copy-field":{"source":"name", "dest":["name_tag"]}
}'

So let’s stop here and see why we added those. First of all, we added a new field type called tag. We needed that for the text tagger to work. There are two crucial things there – the postingsFormat which needs to be set to FST50 and the solr.ConcatenateGraphFilterFactory at index time analysis. Those two settings are required.

Next – we need to keep the names themselves. So we will keep the name in a text_general field that is stored and can be easily retrieved, we will have the name_tag field using our newly created tag field type and finally we have a copy field that we will use to copy the value from the name field to the name_tag field, so that we don’t have to send the data twice.

Now let’s index those names that we had by using the following request:

$ curl -XPOST -H 'Content-type:application/json'  'http://localhost:8983/solr/test/update' -d '[
 {"name":"Thomas Jefferson"},
 {"name":"Alexander Hamilton"},
 {"name":"George Washington"},
 {"name":"John Adams"},
 {"name":"James Wilkinson"}
]'

Tagging Text

Now that we have our data that will be used for tagging sent to Solr we can try tagging a real document. For that, we took a part of the document from Wikipedia describing the life of George Washington. To send it to Solr for tagging we first need to create the configuration for a new handler in Solr, for example like this:

$ curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test/config' -d '{
 "add-requesthandler" : {
  "name": "/tagger",
  "class": "solr.TaggerRequestHandler",
  "defaults": {
   "field":"name_tag"
  }
 }
}'

We added a new request handler called /tagger using the solr.TaggerRequestHandler and we set its defaults to use a field called name_tag – the one that we created the new field type for. Once that is done we can finally look at tagging the text. We do that by sending our document to newly added request handler:

$ curl -X POST 'http://localhost:8983/solr/test/tagger?fl=id,name&wt=json&indent=on' 
-H 'Content-Type:text/plain' -d 'George Washington (February 22, 
1732 December 14, 1799) was the first President of the United States 
(1789-1797), and was among the nation`s Founding Fathers. From 1775 
to 1783, he led the Patriot forces to victory over the British in 
the American Revolutionary War. He presided over the Constitutional 
Convention which formed the basis of the new federal government in 
Since the late 1780s, Washington has been known as the "Father 
of His Country" by compatriots. He is ranked by scholars among 
the top presidents in history. Washington was born to a moderately 
prosperous family of planters, who owned slaves in colonial 
Virginia. He had early opportunities in education, learned 
mathematics and quickly launched a successful career as a surveyor, 
which in turn enabled him to make considerable land investments. He 
then joined the Virginia militia and fought in the French and Indian
War. During the American Revolutionary War, Washington was 
appointed commander-in-chief of the Continental Army, taking 
initiative in raids like those at Trenton, commanding in 
conventional battles such as the Battle of Monmouth, and 
coordinating a combined French-Patriot allied campaign at the Siege 
of Yorktown ending the conflict. His devotion to American 
Republicanism impelled him to decline further power after victory, 
and he resigned as commander-in-chief in 1783. Washington was born 
to a moderately prosperous family of planters, who owned slaves in 
colonial Virginia. He had early opportunities in education, learned 
mathematics and quickly launched a successful career as a surveyor, 
which in turn enabled him to make considerable land investments. He 
then joined the Virginia militia and fought in the French and Indian 
War. During the American Revolutionary War, Washington was appointed 
commander-in-chief of the Continental Army, taking initiative in 
raids like those at Trenton, commanding in conventional battles such 
as the Battle of Monmouth, and coordinating a combined French-Patriot 
allied campaign at the Siege of Yorktown ending the conflict. His 
devotion to American Republicanism impelled him to decline further 
power after victory, and he resigned as commander-
in-chief in 1783. Washington was among the countryís premier statesmen
and was unanimously chosen as president by the Electoral 
College in the first two national elections. He promoted and oversaw 
the implementation of a strong, well-financed national government. 
He remained impartial in the fierce rivalry between two cabinet 
secretaries, Thomas Jefferson and Alexander Hamilton, though he 
adopted Hamilton economic plans. When the French Revolution plunged 
Europe into war, Washington assumed a policy of neutrality 
to protect American shipsóalthough the controversial Jay Treaty of 
1795 normalized trade relations with Great Britain. Washington set 
precedents still in use today, such as the Cabinet advisory system, 
the inaugural address, the title "Mr. President", and a two-term 
limit. In his Farewell Address he gave a primer on civic virtue, 
warning of partisanship, sectionalism, and involvement in foreign 
wars. Washington inherited slaves at age eleven, prospered from slavery 
most of his life, and as late as 1793 officially supported other 
slaveholders. Eventually he became troubled with slavery and in his 
final will in 1799 he freed all his slaves. He is renowned for his 
religious toleration; his personal religion and devotion to 
Freemasonry have been debated. Upon his death, Washington was famously 
eulogized as "first in war, first in peace, and first in 
the hearts of his countrymen". He has been widely memorialized by 
monuments, public works, places, stamps, and currency. The nations capital, Washington D.C., and the state of Washington bear his name; and since 1932 the quarter dollar has carried his effigy.'

The response returned by Solr looks as follows:

{
  "responseHeader":{
    "status":0,
    "QTime":2},
  "tagsCount":15,
  "tags":[[
      "startOffset",0,
      "endOffset",6,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",7,
      "endOffset",17,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",398,
      "endOffset",408,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",534,
      "endOffset",544,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",1693,
      "endOffset",1699,
      "ids",["7dad41c3-e607-4b6f-a86d-6d6490f48e4f"]],
    [
      "startOffset",1700,
      "endOffset",1709,
      "ids",["7dad41c3-e607-4b6f-a86d-6d6490f48e4f"]],
    [
      "startOffset",1714,
      "endOffset",1723,
      "ids",["5796481a-a698-4aa4-9666-9f5e4e565996"]],
    [
      "startOffset",1724,
      "endOffset",1732,
      "ids",["5796481a-a698-4aa4-9666-9f5e4e565996"]],
    [
      "startOffset",1752,
      "endOffset",1760,
      "ids",["5796481a-a698-4aa4-9666-9f5e4e565996"]],
    [
      "startOffset",1831,
      "endOffset",1841,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",1992,
      "endOffset",2002,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",2278,
      "endOffset",2288,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",2651,
      "endOffset",2661,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",2875,
      "endOffset",2885,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
    [
      "startOffset",2909,
      "endOffset",2919,
      "ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]]],
  "response":{"numFound":3,"start":0,"docs":[
      {
        "name":["Thomas Jefferson"],
        "id":"7dad41c3-e607-4b6f-a86d-6d6490f48e4f"},
      {
        "name":["Alexander Hamilton"],
        "id":"5796481a-a698-4aa4-9666-9f5e4e565996"},
      {
        "name":["George Washington"],
        "id":"d8d14516-7637-48c3-8042-ecf68b3822ef"}]
  }}

As you can see we have both the tags that were found along with the offsets, but also the name of the tag and its identifier. From the response that Solr gave us we see that three tags were found and where the tags were found – nice! 🙂

Tagger Parameters

Of course, you can control tagger. As you already seen in the above request that we’ve used fl parameter in the request and the required field property when adding new request handler. However, this is not everything. We can add filter queries, specify the maximum number of rows, choose an algorithm for overlapping tags and so on. You can find the full list of options in the Solr documentation at https://lucene.apache.org/solr/guide/7_4/the-tagger-handler.html.

Performance Considerations

As usual, there are a few things when it comes to best practices for handling the tagger collection and its layout as well as tagging of the documents. So the first problem is that in Solr 7.4.0 Text Tagger doesn’t support batching, so you are only able to send the documents one by one. This will probably change in the future, but for now, you can just combine multiple documents in a single request and add a dummy concatenations character between them. In addition to that, you should consider force merge. As you know, the fewer segments you have in your Lucene index – so your shard, the faster the queries will be. This is also true for tagging – try to keep the number of segments to the minimum to improve throughput and latency of your tagging queries.

Limitations

One last thing that I wanted to mention is that when we’ve been writing this text, so as of Solr 7.4.0 the text tagger was not supporting sharded index. Maybe this will change in the future, but for now, if you want to use text tagger functionality in Solr, you have to be aware of that limitation.

Leave a Reply

Your email address will not be published. Required fields are marked *

We use cookies to personalise content and ads, to provide social media features and to analyse our traffic. We also share information about your use of our site with our social media, advertising and analytics partners.
Cookies settings
Accept
Privacy & Cookie policy
Privacy & Cookies policy
Cookies list
Cookie name Active

Privacy Policy

What information do we collect?

We collect information from you when you register on our site or place an order. When ordering or registering on our site, as appropriate, you may be asked to enter your: name, e-mail address or mailing address.

What do we use your information for?

Any of the information we collect from you may be used in one of the following ways: To personalize your experience (your information helps us to better respond to your individual needs) To improve our website (we continually strive to improve our website offerings based on the information and feedback we receive from you) To improve customer service (your information helps us to more effectively respond to your customer service requests and support needs) To process transactions Your information, whether public or private, will not be sold, exchanged, transferred, or given to any other company for any reason whatsoever, without your consent, other than for the express purpose of delivering the purchased product or service requested. To administer a contest, promotion, survey or other site feature To send periodic emails The email address you provide for order processing, will only be used to send you information and updates pertaining to your order.

How do we protect your information?

We implement a variety of security measures to maintain the safety of your personal information when you place an order or enter, submit, or access your personal information. We offer the use of a secure server. All supplied sensitive/credit information is transmitted via Secure Socket Layer (SSL) technology and then encrypted into our Payment gateway providers database only to be accessible by those authorized with special access rights to such systems, and are required to?keep the information confidential. After a transaction, your private information (credit cards, social security numbers, financials, etc.) will not be kept on file for more than 60 days.

Do we use cookies?

Yes (Cookies are small files that a site or its service provider transfers to your computers hard drive through your Web browser (if you allow) that enables the sites or service providers systems to recognize your browser and capture and remember certain information We use cookies to help us remember and process the items in your shopping cart, understand and save your preferences for future visits, keep track of advertisements and compile aggregate data about site traffic and site interaction so that we can offer better site experiences and tools in the future. We may contract with third-party service providers to assist us in better understanding our site visitors. These service providers are not permitted to use the information collected on our behalf except to help us conduct and improve our business. If you prefer, you can choose to have your computer warn you each time a cookie is being sent, or you can choose to turn off all cookies via your browser settings. Like most websites, if you turn your cookies off, some of our services may not function properly. However, you can still place orders by contacting customer service. Google Analytics We use Google Analytics on our sites for anonymous reporting of site usage and for advertising on the site. If you would like to opt-out of Google Analytics monitoring your behaviour on our sites please use this link (https://tools.google.com/dlpage/gaoptout/)

Do we disclose any information to outside parties?

We do not sell, trade, or otherwise transfer to outside parties your personally identifiable information. This does not include trusted third parties who assist us in operating our website, conducting our business, or servicing you, so long as those parties agree to keep this information confidential. We may also release your information when we believe release is appropriate to comply with the law, enforce our site policies, or protect ours or others rights, property, or safety. However, non-personally identifiable visitor information may be provided to other parties for marketing, advertising, or other uses.

Registration

The minimum information we need to register you is your name, email address and a password. We will ask you more questions for different services, including sales promotions. Unless we say otherwise, you have to answer all the registration questions. We may also ask some other, voluntary questions during registration for certain services (for example, professional networks) so we can gain a clearer understanding of who you are. This also allows us to personalise services for you. To assist us in our marketing, in addition to the data that you provide to us if you register, we may also obtain data from trusted third parties to help us understand what you might be interested in. This ‘profiling’ information is produced from a variety of sources, including publicly available data (such as the electoral roll) or from sources such as surveys and polls where you have given your permission for your data to be shared. You can choose not to have such data shared with the Guardian from these sources by logging into your account and changing the settings in the privacy section. After you have registered, and with your permission, we may send you emails we think may interest you. Newsletters may be personalised based on what you have been reading on theguardian.com. At any time you can decide not to receive these emails and will be able to ‘unsubscribe’. Logging in using social networking credentials If you log-in to our sites using a Facebook log-in, you are granting permission to Facebook to share your user details with us. This will include your name, email address, date of birth and location which will then be used to form a Guardian identity. You can also use your picture from Facebook as part of your profile. This will also allow us and Facebook to share your, networks, user ID and any other information you choose to share according to your Facebook account settings. If you remove the Guardian app from your Facebook settings, we will no longer have access to this information. If you log-in to our sites using a Google log-in, you grant permission to Google to share your user details with us. This will include your name, email address, date of birth, sex and location which we will then use to form a Guardian identity. You may use your picture from Google as part of your profile. This also allows us to share your networks, user ID and any other information you choose to share according to your Google account settings. If you remove the Guardian from your Google settings, we will no longer have access to this information. If you log-in to our sites using a twitter log-in, we receive your avatar (the small picture that appears next to your tweets) and twitter username.

Children’s Online Privacy Protection Act Compliance

We are in compliance with the requirements of COPPA (Childrens Online Privacy Protection Act), we do not collect any information from anyone under 13 years of age. Our website, products and services are all directed to people who are at least 13 years old or older.

Updating your personal information

We offer a ‘My details’ page (also known as Dashboard), where you can update your personal information at any time, and change your marketing preferences. You can get to this page from most pages on the site – simply click on the ‘My details’ link at the top of the screen when you are signed in.

Online Privacy Policy Only

This online privacy policy applies only to information collected through our website and not to information collected offline.

Your Consent

By using our site, you consent to our privacy policy.

Changes to our Privacy Policy

If we decide to change our privacy policy, we will post those changes on this page.
Save settings
Cookies settings