Possibilities

Let’s look at the new API by example.

Getting information in XML format

Many Solr users are used to getting their data in the XML format, at least when using Solr HTTP API. However, the schema API uses JSON as the default format. In order to get the data in the XML format in all the below examples, you’ll need to appeng the wt=xml parameter to the call, for example like that:

$curl 'http://localhost:8983/solr/collection1/schema/fieldtypes?wt=xml'

Defined fields information

Let’s start by looking at how to fetch information about the fields that are defined in Solr. In order to do that we have the following possibilities:

1. Get information about all the fields defined in the index
2. Get information for a one, explicitly defined field

In the first case we should use the following command:

$curl 'http://localhost:8983/solr/collection1/schema/fields'

In second case we should add the / character and the field name to the above command. For example in order to get the information about the author field we should use the following command:

$curl 'http://localhost:8983/solr/collection1/schema/fields/author'

Solr response for the first command will be similar to the following one:

{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "fields":[{
      "name":"_version_",
      "type":"long",
      "indexed":true,
      "stored":true},
    {
      "name":"author",
      "type":"text_general",
      "indexed":true,
      "stored":true},
    {
      "name":"cat",
      "type":"string",
      "multiValued":true,
      "indexed":true,
      "stored":true},
    {
      "name":"category",
      "type":"text_general",
      "indexed":true,
      "stored":true},
    {
      "name":"id",
      "type":"string",
      "multiValued":false,
      "indexed":true,
      "required":true,
      "stored":true,
      "uniqueKey":true},
    {
      "name":"url",
      "type":"text_general",
      "indexed":true,
      "stored":true},
    {
      "name":"weight",
      "type":"float",
      "indexed":true,
      "stored":true}]}

On the other hand the response for the second command would be as follows:

{
  "responseHeader":{
    "status":0,
    "QTime":0},
  "field":{
    "name":"author",
    "type":"text_general",
    "indexed":true,
    "stored":true}}

Getting information about defined dynamic fields

Similar to what information we can get about the fields defined in the schema.xml we can get the information about dynamic fields. Again we have to options:

1. Get information about all dynamic fields
2. Get information about specific dynamic field pattern

In order to get all the information about dynamic fields we should use the following command:

$curl 'http://localhost:8983/solr/collection1/schema/dynamicfields'

In order to get information about a specific pattern we append the / character followed by the pattern, for example like this:

$curl 'http://localhost:8983/solr/collection1/schema/dynamicfields/random_*'

Solr will return the following response for the first query:

{
  "responseHeader":{
    "status":0,
    "QTime":2},
  "dynamicfields":[{
      "name":"*_coordinate",
      "type":"tdouble",
      "indexed":true,
      "stored":false},
    {
      "name":"ignored_*",
      "type":"ignored",
      "multiValued":true},
    {
      "name":"random_*",
      "type":"random"},
    {
      "name":"*_p",
      "type":"location",
      "indexed":true,
      "stored":true},
    {
      "name":"*_c",
      "type":"currency",
      "indexed":true,
      "stored":true}]}

And the following response will be returned for the second command:

{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "dynamicfield":{
    "name":"random_*",
    "type":"random"}}

Getting field types

As you probably guess, in a way similar to the above describes examples, we can also get the information about the field types defined in our schema.xml files. We can fetch the following information:

1. All the field types defined in the schema.xml file
2. A single type

To get all the defined field types we should run the following command:

$curl 'http://localhost:8983/solr/collection1/schema/fieldtypes'

The get information about a single type we should again add the / character and append the field type name to it, for example like this:

$curl 'http://localhost:8983/solr/collection1/schema/fieldtypes/text_gl'

Solr will return the following information in response to the first command:

{
  "responseHeader":{
    "status":0,
    "QTime":3},
  "fieldTypes":[{
      "name":"alphaOnlySort",
      "class":"solr.TextField",
      "sortMissingLast":true,
      "omitNorms":true,
      "analyzer":{
        "class":"solr.TokenizerChain",
        "tokenizer":{
          "class":"solr.KeywordTokenizerFactory"},
        "filters":[{
            "class":"solr.LowerCaseFilterFactory"},
          {
            "class":"solr.TrimFilterFactory"},
          {
            "class":"solr.PatternReplaceFilterFactory",
            "replace":"all",
            "replacement":"",
            "pattern":"([^a-z])"}]},
      "fields":[],
      "dynamicFields":[]},
    {
      "name":"boolean",
      "class":"solr.BoolField",
      "sortMissingLast":true,
      "fields":["inStock"],
      "dynamicFields":["*_bs",
        "*_b"]},
    {
      "name":"text_gl",
      "class":"solr.TextField",
      "positionIncrementGap":"100",
      "analyzer":{
        "class":"solr.TokenizerChain",
        "tokenizer":{
          "class":"solr.StandardTokenizerFactory"},
        "filters":[{
            "class":"solr.LowerCaseFilterFactory"},
          {
            "class":"solr.StopFilterFactory",
            "words":"lang/stopwords_gl.txt",
            "ignoreCase":"true",
            "enablePositionIncrements":"true"},
          {
            "class":"solr.GalicianStemFilterFactory"}]},
      "fields":[],
      "dynamicFields":[]},
    {
      "name":"tlong",
      "class":"solr.TrieLongField",
      "precisionStep":"8",
      "positionIncrementGap":"0",
      "fields":[],
      "dynamicFields":["*_tl"]}]}

In response to the second command Solr will return the following:

{
  "responseHeader":{
    "status":0,
    "QTime":2},
  "fieldType":{
    "name":"text_gl",
    "class":"solr.TextField",
    "positionIncrementGap":"100",
    "analyzer":{
      "class":"solr.TokenizerChain",
      "tokenizer":{
        "class":"solr.StandardTokenizerFactory"},
      "filters":[{
          "class":"solr.LowerCaseFilterFactory"},
        {
          "class":"solr.StopFilterFactory",
          "words":"lang/stopwords_gl.txt",
          "ignoreCase":"true",
          "enablePositionIncrements":"true"},
        {
          "class":"solr.GalicianStemFilterFactory"}]},
    "fields":[],
    "dynamicFields":[]}}

As you can see, the amount information is nice as we are getting all the information about the field types and in addition to that the information which field are using give field (both dynamic and non-dynamic.

Retrieving information about copyFields

In addition to what we’ve discussed so far we are able to get information about copyFields section from the schema.xml. In order to do that one should run the following command:

$curl 'http://localhost:8983/solr/collection1/schema/copyfields'

And in response we will get the following data:

{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "copyfields":[{
      "source":"author",
      "dest":"text"},
    {
      "source":"cat",
      "dest":"text"},
    {
      "source":"content",
      "dest":"text"},
    {
      "source":"content_type",
      "dest":"text"},
    {
      "source":"description",
      "dest":"text"},
    {
      "source":"features",
      "dest":"text"},
    {
      "source":"author",
      "dest":"author_s",
      "destDynamicBase":"*_s"}]}

The future

In Solr 4.3 the described API was improved and is now being prepared to enable not only reading of the index structure, but also writing modifications to it with the use of HTTP requests. We can expect that feature in one of the upcoming versions of Apache Solr, so its worth waiting in my opinion, at least by those who needs it.

W Solr 4.3 opisywane API zostało usprawnione oraz jest przygotowywane do umożliwienia zmian w strukturze indeksu za pomocą protokołu HTTP. Możemy zatem spodziewać się, iż w jednej z kolejnych wersji serwera wyszukiwania Solr otrzymamy możliwość łatwej zmiany struktury indeksu, przynajmniej takich, które nie będą powodować konfliktów z już zaindeksowanymi danymi.

Each of us knows what is schema.xml file and what is (if not, I invite you to read the entry located at: http://solr.pl/2010/08/16/what-is-schema-xml/?lang=en). What are the most frequently commit errors creating or updating this file? I personally met with the following:

1. Trash in the configuration

I confess that the first principle is to keep the file schema.xml in the simplest possible form. Linked to this is a very important issue – this file should not be synonymous with chaos. In other word, do not stick with unnecessary comments, unwanted types, fields and so on. Order in the structure of the schema.xml file not only helps us to maintain this file and its modifications with ease, but also assures us that no information that is unnecessary will be stored in Solr index.

2. Cosmetic changes to the default configuration

How many of those who use Solr in their daily work took the default schema.xml file supplied in the example implementation Solr and only slightly modified the contents – for example, changing only the names of the fields ? I should raise my hand too, because I did it once. This is a pretty big mistake. Someone may ask why. Are you sure You need English stemming when implementing search for content written in Polish ? I think not. The same applies to field and type attributes like term vectors.

3. No updates

Sometimes I find the implementation of search based application, where update of Solr does not mean an update of schema.xml file. If it is a conscious decision, dictated by such costly or even impossible re-indexing of all data, I understand the situation. But there are cases where an upgrade would bring only benefits, and where costs of such upgrade would be minimal (eg less expensive re-index or slight changes in the application). Do not be afraid to update the schema.xml file – whether it is to update the fields, update types, whether the addition of newer stuff. A good example is the migration from Solr 1.3 to version 1.4 – newer version introduced significant changes associated with numeric types, where migration to the new types would result in great increase in query performance using those types (such as queries using value ranges).

4. “I`ll use it one day”

Adding new types, not removing unnecessary now, the same in the case of fields, or copyField definition. Most of us think – that old definition can be useful in the future, but remember that each type is some extra portion of memory needed by Solr, each field is a place in the index. My small advice – if you stop to use the type, field, or whatever else you have in your configuration file (not only in the schema.xml), simply remove it from this file. Applying this principle throughout the life cycle of the applications using Solr will ensure You that the index is in optimal condition, and after a few months since another feature implementation You will not need to be puzzled and as a result You will not need to dig into the application code to determine if the field is used in some forgotten code fragment.

5. Attributes, attributes and again attributes

Preservation of original values, adding term vectors and its properties are just examples of things we don`t need in every implementation. Sometimes we have more than required by the application index. A larger index, lower productivity, at least in some cases (eg, indexing). It is worth considering if you really need all this information, which we say to Solr to calculate and store. Removing some unnecessary, of course, from our point of view of information, may surprise us. Sometimes it is worth a try;)

Feel free to comment, because I will read eagerly, for what else we should pay attention to when modifying schema.xml file.

Finally, I think that it is worth to mention the article “The Seven Deadly Sins of Solr” LucidImagination published on the website at: http://www.lucidimagination.com/blog/2010/01/21/the-seven-deadly-sins-of-solr. It describes bad practices when working with Solr. In my opinion, interesting reading. I highly recommend it.