Add Solr 3 plugin or compatibility

[mkboudreau]

Feature Request

Proposal:

Add Solr 3 compatibility.

Current behavior:

The current Solr plugin introduced in telegraf 1.5, supports and was tested with Solr 4.3, 5.5.5, 6.6

Desired behavior:

Support older legacy Solr instances, such as 3.6.

Use case: [Why is this important (helps with prioritizing requests)]

Some organizations still have old, legacy installations of Solr that they'd like to monitor.

Additional info:

PR: #2019

Issue: #278

[mkboudreau]

FYI, I have a working plugin, based on the current solr plugin (#2019). I named it inputs/solr3 for internal/testing.

I'd like to know whether or not it's preferred to merge into the existing solr plugin and have multiple parsing paths based on a "version" config param or to keep it separate.

[mkboudreau]

One piece of incompatibility is in the stats parsing of the queryhandler:

 "stats": {
          "handlerStart": 1516083354311,
          "requests": 0,
          "errors": 0,
          "timeouts": 0,
          "totalTime": 0,
          "avgTimePerRequest": "NaN",
          "avgRequestsPerSecond": 0
        }

The "NaN" causes an error as well as the missing fields:

	"15min_rate_reqs_per_second"
	"5min_rate_reqs_per_second"
	"75th_pc_request_time"
	"95th_pc_request_time"
	"999th_pc_request_time"
	"99th_pc_request_time"

Additionally, there are some queryhandler values that have "stats" as an array instead of an object, which triggers a parsing error.

[danielnelson]

I would like to have any changes done to the existing plugin, so long as it is not too large of a change. I think the examples you gave here would be okay to fix, but make sure to create a unittest with a comment about it being for solr3 support.

If you can show me an example of the array vs object that would be helpful.

[mkboudreau]

The general structure of a response looks like this:

{
  "responseHeader": {
    "status": 0,
    "QTime": 1
  },
  "solr-mbeans": [
    "CORE",
    { ... }
    "QUERYHANDLER",
    {
      "handler1": { ... },
      "handler2": { 
      	"key": "value",
      	"stats": {}
      },
      ...
    },
    "QUERY",
    {
      "q1": { ... },
      ...
    },
    "UPDATEHANDLER",
    {
      "u1": { ... },
      ...
    },
    "UPDATE",
    {
      "u1": { ... },
      ...
    },
    "CACHE",
    {
      "c1": { ... },
      ...
    }
  ]
}

Under the QUERYHANDLER section, in Solr 6, we see the following:

    "/dataimport": {
        "class": "org.apache.solr.handler.dataimport.DataImportHandler",
        "version": "6.6.1",
        "description": "Manage data import from databases to Solr",
        "src": null,
        "stats": {
          "Status": "IDLE",
          "Total Documents Processed": "java.util.concurrent.atomic.AtomicLong:0",
          "Total Requests made to DataSource": "java.util.concurrent.atomic.AtomicLong:0",
          "Total Rows Fetched": "java.util.concurrent.atomic.AtomicLong:0",
          "Total Documents Deleted": "java.util.concurrent.atomic.AtomicLong:0",
          "Total Documents Skipped": "java.util.concurrent.atomic.AtomicLong:0",
          "handlerStart": 1516040508127,
          "requests": 0,
          "errors": 0,
          "serverErrors": 0,
          "clientErrors": 0,
          "timeouts": 0,
          "totalTime": 0,
          "avgRequestsPerSecond": 0,
          "5minRateRequestsPerSecond": 0,
          "15minRateRequestsPerSecond": 0,
          "avgTimePerRequest": 0,
          "medianRequestTime": 0,
          "75thPcRequestTime": 0,
          "95thPcRequestTime": 0,
          "99thPcRequestTime": 0,
          "999thPcRequestTime": 0
        }
      },

That same structure in Solr 3 looks as follows (see the "stats" attribute):

 "/dataimport": {
        "class": "org.apache.solr.handler.dataimport.DataImportHandler",
        "version": "1.0",
        "description": "Manage data import from databases to Solr",
        "srcId": "$Id: DataImportHandler.java 1171306 2011-09-15 22:43:33Z janhoy $",
        "src": "$URL: https://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_3_5/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/DataImportHandler.java $",
        "docs": null,
        "stats": [
          "Status",
          "IDLE",
          "Documents Processed",
          "java.util.concurrent.atomic.AtomicLong:788268",
          "Requests made to DataSource",
          "java.util.concurrent.atomic.AtomicLong:11",
          "Rows Fetched",
          "java.util.concurrent.atomic.AtomicLong:5841951",
          "Documents Deleted",
          "java.util.concurrent.atomic.AtomicLong:0",
          "Documents Skipped",
          "java.util.concurrent.atomic.AtomicLong:0",
          "Total Documents Processed",
          "java.util.concurrent.atomic.AtomicLong:788268",
          "Total Requests made to DataSource",
          "java.util.concurrent.atomic.AtomicLong:11",
          "Total Rows Fetched",
          "java.util.concurrent.atomic.AtomicLong:5841951",
          "Total Documents Deleted",
          "java.util.concurrent.atomic.AtomicLong:0",
          "Total Documents Skipped",
          "java.util.concurrent.atomic.AtomicLong:0",
          "handlerStart",
          1516083353928,
          "requests",
          28,
          "errors",
          0,
          "timeouts",
          0,
          "totalTime",
          4,
          "avgTimePerRequest",
          0.14285715,
          "avgRequestsPerSecond",
          0.00047454043
        ]
      },

Note:

This only occurs when the queryhandler's class equals "org.apache.solr.handler.dataimport.DataImportHandler". For all other queryhandlers, Solr 3 does return an object instead of an array. For some reason, it only uses an array for the data import handler. Very odd.

[danielnelson]

I think this should be okay to introduce to the existing plugin, we can probably just unmarshal into a interface{} and then switch on handling.

[mkboudreau]

Ok. I can definitely do that. I was just looking for some discussion on it. thank you

[mkboudreau]

FYI, I still intend on finishing up the Solr work for this issue as well as #3682 and #3665.

Waiting on a couple approvals (at my org) first.