[CI] PainlessDomainSplitIT fails after multibucket

[benwtrent]

To recreate

./gradlew :x-pack:qa:ml-single-node-tests:integTestRunner -Dtests.class=org.elasticsearch.xpack.ml.transforms.PainlessDomainSplitIT -Dtests.method="testHRDSplit"

Update On master after the restructure the reproduce command is now

./gradlew :x-pack:plugin:ml:qa:single-node-tests:integTestRunner -Dtests.class=org.elasticsearch.xpack.ml.transforms.PainlessDomainSplitIT -Dtests.method="testHRDSplit"

See failing build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/15432/consoleFull

[benwtrent]

In my investigation some strangeness arises when gathering records.

The test assumes that non-anomalous data is returned when gathering the record information. However, only the anomalous bar.com entries are ever returned.

See

elasticsearch/x-pack/qa/ml-single-node-tests/src/test/java/org/elasticsearch/xpack/ml/transforms/PainlessDomainSplitIT.java

Line 231 in 647705e

// domainSplit() tests had subdomain, testHighestRegisteredDomainCases() do not

A simple way to fix this would be to just make test.hostname slightly anomalous so that it is returned in the records call. Unsure if this defeats the purpose of the test however.

[dimitris-athanasiou]

This test is doing count by domain_split with a bucket_span of 1h. The data spans 100 buckets. It indexes 1 doc per bucket with a certain domain. Then on the 65th bucket, that domain is not in. Instead, there are 100 docs of a different domain. The test expects 2 anomalies: one for the new domain that has a count of 100, and one for the other domain which has a count of zero when a count of one is expected.

After the multibucket feature was merged, we do no longer see the anomaly for the missing domain. This definitely seems odd. @tveasey What are your thoughts on this one?

[benwtrent]

Doing some experiments, if the number of documents is 2 per bucket, then we do get both anomalies. Though the missing domain probability is extremely small.

{
   "job_id":"hrd-split-job",
   "result_type":"record",
   "probability":3.6470168589396805E-6,
   "record_score":60.098482474177416,
   "initial_record_score":60.098482474177416,
   "bucket_span":3600,
   "detector_index":0,
   "is_interim":false,
   "timestamp":1503460800000,
   "by_field_name":"domain_split",
   "by_field_value":"kerberos.http.192.168,62.222",
   "function":"count",
   "function_description":"count",
   "typical":[1.9850021750253195],
   "actual":[0.0],
   "domain_split":["kerberos.http.192.168,62.222"]
}

[tveasey]

Agreed this does seem odd. I think this warrants investigation as to what is happening inside the model. I suspect based on Ben's last comment that this might be a numerical issue: something underflowing and not being handled properly (probably when taking a log). I'll look into this.

[davidkyle]

This test still fails. Is it a problem with the test or a blocker for 6.5?

I updated the reproduce command above