We haven’t looked into any Solr functionalities on Solr.pl for a while so it is time to look into one of the new features – the text tagger. It works on the basis of Solr index and is able to check documents sent to a new handler and return occurrences of names with offsets and other metadata that we added to Solr index. However keep in mind that Solr Text Tagger doesn’t do any kind of natural language processing, so no what we will give it, that is what we can expect to get back.
Preparations
We will keep it as minimal as it can be. We start with running our Solr instance:
$ bin/solr start -c -f
And we create a test collection:
$ bin/solr create -c test
We will be using the standard data-driven schema. Though it is not suggested to be used in production, it is more than enough for our Text Tagger test.
Data For Tagging
Let’s focus on the data for a second. First of all, we need data that will be used for tagging. For the purpose of the blog post we will just go with simple name tagging, for example:
Thomas Jefferson Alexander Hamilton George Washington John Adams James Wilkinson
A few names – you may recognize them 😉 We will use those names for tagging purposes.
To index them to Solr we will need a new field type, two fields, and a copy field, just like this:
$ curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test/schema' -d '{
"add-field-type":{
"name":"tag",
"class":"solr.TextField",
"postingsFormat":"FST50",
"omitNorms":true,
"omitTermFreqAndPositions":true,
"indexAnalyzer":{
"tokenizer":{ "class":"solr.StandardTokenizerFactory" },
"filters":[
{"class":"solr.LowerCaseFilterFactory"},
{"class":"solr.ConcatenateGraphFilterFactory", "preservePositionIncrements":false }
]},
"queryAnalyzer":{
"tokenizer":{
"class":"solr.StandardTokenizerFactory" },
"filters":[
{"class":"solr.LowerCaseFilterFactory"}
]}
},
"add-field":{"name":"name", "type":"text_general"},
"add-field":{"name":"name_tag", "type":"tag", "stored":false},
"add-copy-field":{"source":"name", "dest":["name_tag"]}
}'
So let’s stop here and see why we added those. First of all, we added a new field type called tag. We needed that for the text tagger to work. There are two crucial things there – the postingsFormat which needs to be set to FST50 and the solr.ConcatenateGraphFilterFactory at index time analysis. Those two settings are required.
Next – we need to keep the names themselves. So we will keep the name in a text_general field that is stored and can be easily retrieved, we will have the name_tag field using our newly created tag field type and finally we have a copy field that we will use to copy the value from the name field to the name_tag field, so that we don’t have to send the data twice.
Now let’s index those names that we had by using the following request:
$ curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test/update' -d '[
{"name":"Thomas Jefferson"},
{"name":"Alexander Hamilton"},
{"name":"George Washington"},
{"name":"John Adams"},
{"name":"James Wilkinson"}
]'
Tagging Text
Now that we have our data that will be used for tagging sent to Solr we can try tagging a real document. For that, we took a part of the document from Wikipedia describing the life of George Washington. To send it to Solr for tagging we first need to create the configuration for a new handler in Solr, for example like this:
$ curl -XPOST -H 'Content-type:application/json' 'http://localhost:8983/solr/test/config' -d '{
"add-requesthandler" : {
"name": "/tagger",
"class": "solr.TaggerRequestHandler",
"defaults": {
"field":"name_tag"
}
}
}'
We added a new request handler called /tagger using the solr.TaggerRequestHandler and we set its defaults to use a field called name_tag – the one that we created the new field type for. Once that is done we can finally look at tagging the text. We do that by sending our document to newly added request handler:
$ curl -X POST 'http://localhost:8983/solr/test/tagger?fl=id,name&wt=json&indent=on'
-H 'Content-Type:text/plain' -d 'George Washington (February 22, 1732 December 14, 1799) was the first President of the United States (1789-1797), and was among the nation`s Founding Fathers. From 1775 to 1783, he led the Patriot forces to victory over the British in the American Revolutionary War. He presided over the Constitutional Convention which formed the basis of the new federal government in Since the late 1780s, Washington has been known as the "Father of His Country" by compatriots. He is ranked by scholars among
the top presidents in history. Washington was born to a moderately prosperous family of planters, who owned slaves in colonial Virginia. He had early opportunities in education, learned mathematics and quickly launched a successful career as a surveyor, which in turn enabled him to make considerable land investments. He then joined the Virginia militia and fought in the French and Indian War. During the American Revolutionary War, Washington was appointed commander-in-chief of the Continental Army, taking initiative in raids like those at Trenton, commanding in conventional battles such as the Battle of Monmouth, and coordinating a combined French-Patriot allied campaign at the Siege of Yorktown ending the conflict. His devotion to American Republicanism impelled him to decline further power after victory, and he resigned as commander-in-chief in 1783. Washington was born to a moderately prosperous family of planters, who owned slaves in colonial Virginia. He had early opportunities in education, learned mathematics and quickly launched a successful career as a surveyor, which in turn enabled him to make considerable land investments. He then joined the Virginia militia and fought in the French and Indian War. During the American Revolutionary War, Washington was appointed commander-in-chief of the Continental Army, taking initiative in raids like those at Trenton, commanding in conventional battles such as the Battle of Monmouth, and coordinating a combined French-Patriot allied campaign at the Siege of Yorktown ending the conflict. His devotion to American Republicanism impelled him to decline further power after victory, and he resigned as commander- in-chief in 1783. Washington was among the countryís premier statesmen and was unanimously chosen as president by the Electoral College in the first two national elections. He promoted and oversaw the implementation of a strong, well-financed national government. He remained impartial in the fierce rivalry between two cabinet secretaries, Thomas Jefferson and Alexander Hamilton, though he adopted Hamilton economic plans. When the French Revolution plunged Europe into war, Washington assumed a policy of neutrality to protect American shipsóalthough the controversial Jay Treaty of 1795 normalized trade relations with Great Britain. Washington set precedents still in use today, such as the Cabinet advisory system, the inaugural address, the title "Mr. President", and a two-term limit. In his Farewell Address he gave a primer on civic virtue, warning of partisanship, sectionalism, and involvement in foreign wars. Washington inherited slaves at age eleven, prospered from slavery most of his life, and as late as 1793 officially supported other slaveholders. Eventually he became troubled with slavery and in his final will in 1799 he freed all his slaves. He is renowned for his religious toleration; his personal religion and devotion to Freemasonry have been debated. Upon his death, Washington was famously eulogized as "first in war, first in peace, and first in the hearts of his countrymen". He has been widely memorialized by monuments, public works, places, stamps, and currency. The nations capital, Washington D.C., and the state of Washington bear his name; and since 1932 the quarter dollar has carried his effigy.'
The response returned by Solr looks as follows:
{
"responseHeader":{
"status":0,
"QTime":2},
"tagsCount":15,
"tags":[[
"startOffset",0,
"endOffset",6,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",7,
"endOffset",17,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",398,
"endOffset",408,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",534,
"endOffset",544,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",1693,
"endOffset",1699,
"ids",["7dad41c3-e607-4b6f-a86d-6d6490f48e4f"]],
[
"startOffset",1700,
"endOffset",1709,
"ids",["7dad41c3-e607-4b6f-a86d-6d6490f48e4f"]],
[
"startOffset",1714,
"endOffset",1723,
"ids",["5796481a-a698-4aa4-9666-9f5e4e565996"]],
[
"startOffset",1724,
"endOffset",1732,
"ids",["5796481a-a698-4aa4-9666-9f5e4e565996"]],
[
"startOffset",1752,
"endOffset",1760,
"ids",["5796481a-a698-4aa4-9666-9f5e4e565996"]],
[
"startOffset",1831,
"endOffset",1841,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",1992,
"endOffset",2002,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",2278,
"endOffset",2288,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",2651,
"endOffset",2661,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",2875,
"endOffset",2885,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]],
[
"startOffset",2909,
"endOffset",2919,
"ids",["d8d14516-7637-48c3-8042-ecf68b3822ef"]]],
"response":{"numFound":3,"start":0,"docs":[
{
"name":["Thomas Jefferson"],
"id":"7dad41c3-e607-4b6f-a86d-6d6490f48e4f"},
{
"name":["Alexander Hamilton"],
"id":"5796481a-a698-4aa4-9666-9f5e4e565996"},
{
"name":["George Washington"],
"id":"d8d14516-7637-48c3-8042-ecf68b3822ef"}]
}}
As you can see we have both the tags that were found along with the offsets, but also the name of the tag and its identifier. From the response that Solr gave us we see that three tags were found and where the tags were found – nice! 🙂
Tagger Parameters
Of course, you can control tagger. As you already seen in the above request that we’ve used fl parameter in the request and the required field property when adding new request handler. However, this is not everything. We can add filter queries, specify
Performance Considerations
As usual, there are a few things when it comes to best practices for handling the tagger collection and its layout as well as tagging of the documents. So the first problem is that in Solr 7.4.0 Text Tagger doesn’t support batching, so you are only able to send the documents one by one. This will probably change in the future, but for now, you can just combine multiple documents in a single request and add a dummy concatenations character between them. In addition to that, you should consider force merge. As you know, the fewer segments you have in your Lucene index – so your shard, the faster the queries will be. This is also true for tagging – try to keep the number of segments to the minimum to improve throughput and latency of your tagging queries.
Limitations
One last thing that I wanted to mention is that when we’ve been writing this text, so as of Solr 7.4.0 the text tagger was not supporting sharded index. Maybe this will change in the future, but for now, if you want to use text tagger functionality in Solr, you have to be aware of that limitation.