[Security Solution][Detections] Error fetching preview when creating rule in index with simple documents #92434

achuguy · 2021-02-23T15:37:59Z

Describe the bug:
When using simple documents not conforming to ECS the preview for creating a query or threshold rule returns an error 'Error fetching preview'

Error fetching preview
EsError
Error: EsError
    at search_interceptor_EnhancedSearchInterceptor.handleSearchError (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/plugin/data/data.plugin.js:8:218143)
    at t.selector (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/plugin/dataEnhanced/dataEnhanced.plugin.js:2:29753)
    at t.error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:436:94089)
    at t._error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:436:134050)
    at t.error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:58213)
    at t._error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:58519)
    at t.error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:58213)
    at t._error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:58519)
    at t.error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21:58213)
    at t._error (https://1ffb4679fc5f4b18aeb8f724c4e7527f.us-east-1.aws.staging.foundit.no:9243/38938/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:436:134050)

The documents used are very simple, timestamp plus some test values:

I was able to confirm the preview works as expected with real endpoint data.

Kibana/Elasticsearch Stack version:
7.12 BC1

Server OS version:

Browser and Browser OS versions:

Elastic Endpoint version:

Original install method (e.g. download page, yum, from source, etc.):

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Steps to reproduce:

Index simple documents into an index i.e. @timestamp + value
Go to rule creation and use an index pattern matching the test index
Enter rules to match value
Preview rule

Current behavior:
Preview returns error if only simple documents in index

Expected behavior:
Preview should work regardless of document fields in index?

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context (logs, chat logs, magical formulas, etc.):

The text was updated successfully, but these errors were encountered:

elasticmachine · 2021-02-23T15:38:01Z

Pinging @elastic/security-solution (Team: SecuritySolution)

yctercero · 2021-03-16T18:14:02Z

@achuguy @peluja1012 So I tried recreating something similar and didn't hit the issue.

The docs in my made up index are here

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 13,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "basic-ytp",
        "_id" : "u-0PIHgBhcDDTrEEn-z5",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "suricata-sensor",
            "id" : "01296047-6f00-467c-ac60-f1a8d52bcac1"
          },
          "@timestamp" : "2021-03-04T17:10:05.438Z",
          "event" : {
            "severity" : 25,
            "action" : "socket_closed",
            "id" : "3ec71e6c-90cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "HAgTIHgBc4D54_cx6Oz0",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "suricata-sensor",
            "id" : "01296047-6f00-467c-ac60-f1a8d52bcac1"
          },
          "@timestamp" : "2021-03-09T17:10:05.438Z",
          "event" : {
            "severity" : 25,
            "action" : "socket_closed",
            "id" : "3ec71e6c-90cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "nQgPIHgBc4D54_cx3J8h",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "internal-ci-7",
            "id" : "a1d7b39c-f898-4dbe-a761-efb61939302d"
          },
          "@timestamp" : "2020-03-01T16:40:03.790Z",
          "event" : {
            "severity" : 50,
            "action" : "socket_closed",
            "id" : "4ec71e6c-50cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "U7wPIHgBIUXFwfmr_7QR",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "internal-ci-7",
            "id" : "a1d7b39c-f898-4dbe-a761-efb61939302d"
          },
          "@timestamp" : "2021-03-01T17:45:03.790Z",
          "event" : {
            "severity" : 75,
            "action" : "socket_closed",
            "id" : "4ec71e6c-50cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "-u4UIHgBhcDDTrEEFNRY",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "internal-ci-7",
            "id" : "a1d7b39c-f898-4dbe-a761-efb61939302d"
          },
          "@timestamp" : "2020-03-09T16:40:03.790Z",
          "event" : {
            "severity" : 50,
            "action" : "socket_closed",
            "id" : "4ec71e6c-50cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "V-4QIHgBhcDDTrEEFAOL",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "suricata-sensor",
            "id" : "01296047-6f00-467c-ac60-f1a8d52bcac1"
          },
          "@timestamp" : "2020-03-04T17:36:05.438Z",
          "event" : {
            "severity" : 25,
            "action" : "updated",
            "id" : "3ec71e6c-90cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "vu4QIHgBhcDDTrEElRzy",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "suricata-sensor",
            "id" : "01296047-6f00-467c-ac60-f1a8d52bcac1"
          },
          "@timestamp" : "2020-03-04T15:40:05.438Z",
          "event" : {
            "severity" : 50,
            "action" : "updated",
            "id" : "3ec71e6c-90cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "dbwQIHgBIUXFwfmru8JF",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "suricata-sensor",
            "id" : "01296047-6f00-467c-ac60-f1a8d52bcac1"
          },
          "@timestamp" : "2021-03-04T19:40:05.438Z",
          "event" : {
            "severity" : 25,
            "action" : "updated",
            "id" : "3ec71e6c-90cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "7bwQIHgBIUXFwfmr9cXJ",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "suricata-sensor",
            "id" : "01296047-6f00-467c-ac60-f1a8d52bcac1"
          },
          "@timestamp" : "2021-03-04T17:11:05.438Z",
          "event" : {
            "severity" : 50,
            "action" : "updated",
            "id" : "3ec71e6c-90cc-4aec-854f-004254a5d02b"
          }
        }
      },
      {
        "_index" : "basic-ytp",
        "_id" : "8wgQIHgBc4D54_cx_bX0",
        "_score" : 1.0,
        "_source" : {
          "agent" : {
            "name" : "siem-kibana",
            "id" : "01296047-6f00-467c-ac60-f1a8d52bcac1"
          },
          "@timestamp" : "2020-03-01T17:30:03.790Z",
          "event" : {
            "severity" : 75,
            "action" : "updated",
            "id" : "3ec71e6c-90cc-4aec-854f-004254a5d02b"
          }
        }
      }
    ]
  }
}

And here's what I got in preview:

If you're able to recreate what you saw, inspect in console and dig into the response, there should be more detail about what might be going wrong. Maybe what we should work on is digging out those details to make sure they show in the error modal.

MadameSheema · 2021-03-22T09:41:05Z

@achuguy is this issue still happening?

@spong @peluja1012 if so, can you please update the version label? Thanks :)

achuguy · 2021-03-23T14:19:29Z

@yctercero Looks like I'm not mapping my fields correctly and I'm using a search on a text field. I'm getting the following error from the console response:

reason: "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [event.category] in order to load field data by uninverting the inverted index. Note that this can use significant memory."

type: "illegal_argument_exception"

Definitely would help to bring that error up but not sure if this is user error on my part.

madirey · 2021-03-23T14:25:39Z

@achuguy Are you scripting it? The UI shouldn't allow you to use a .text (or any non-aggregatable field). But we don't protect against that at the API layer, as it would be a bit more complicated/expensive. cc @yctercero

achuguy · 2021-03-23T14:50:44Z

@madirey No scripting, just the UI. I've tried documents where I haven't set any mapping and also tried documents with fields set explicitly to keyword but I'm getting the same error regardless of the field used. I'm wondering if I'm indexing the documents incorrectly or something.

achuguy · 2021-03-23T14:56:15Z

I put some documents on https://p.elstc.co/paste/Y1mgeIbI#mvFUNu0myis4vubcBcGZuZWMdemaFYLPrNGEO9BZkuf in the index test-index that I'm running into this issue with

madirey · 2021-03-23T16:05:16Z

Whoops, I think I mixed this up with the other Preview issue. Thanks!

yctercero · 2021-03-23T22:10:33Z

@achuguy thanks! I'll revisit this this week.

spong · 2021-03-23T23:31:03Z

@MadameSheema Removing version as we continue to diagnose -- can then redetermine impact, but sounds like low based on a mapping issue (maybe just add an additional UI guard?).

peluja1012 · 2021-05-21T04:53:43Z

There seems to be an issue with Threshold rules and grouping by a field of type ip. I see the following error in 7.12 and in master. Tracking separately here.

7.12.0

master

To reproduce:

Create a test index with an ip field like this:

Dev Console code

PUT test-index-1
{
  "mappings": {
    "properties": {
      "source": {
        "properties": {
          "ip": {
            "type": "ip"
          }
        }
      }
  }
}

Index a few documents like this:

Dev Console code

POST test-index-1/_doc
{
  "@timestamp": "2021-05-20T18:10:49.337Z",
  "source": {
    "ip": "1.1.1.1"
  }
}

Begin to create a threshold rule and configure it like this:
Click the Preview Results button.

elasticmachine · 2021-07-26T08:03:13Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

MadameSheema · 2021-07-26T08:03:23Z

@peluja any update regarding the status of this bug? thanks :)

peluja1012 · 2021-11-09T01:29:39Z

Closing as fixed by this PR #105126

achuguy added bug Fixes for quality problems that affect the customer experience Feature:Detection Rules Anything related to Security Solution's Detection Rules v7.12.0 Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Feb 23, 2021

peluja1012 added the impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. label Feb 23, 2021

peluja1012 assigned yctercero Mar 9, 2021

spong removed the v7.12.0 label Mar 23, 2021

peluja1012 added v7.14.0 v7.13.1 Feature:Detection Rule Preview Security Solution Rule Preview and removed Feature:Detection Rules Anything related to Security Solution's Detection Rules labels May 21, 2021

peluja1012 unassigned yctercero Jul 8, 2021

MadameSheema added the Team:Detections and Resp Security Detection Response Team label Jul 26, 2021

MadameSheema removed v7.13.1 v7.14.0 labels Jul 28, 2021

peluja1012 added the Team:Detection Rule Management Security Detection Rule Management Team label Sep 15, 2021

peluja1012 closed this as completed Nov 9, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detections] Error fetching preview when creating rule in index with simple documents #92434

[Security Solution][Detections] Error fetching preview when creating rule in index with simple documents #92434

achuguy commented Feb 23, 2021 •

edited

Loading

elasticmachine commented Feb 23, 2021

yctercero commented Mar 16, 2021

MadameSheema commented Mar 22, 2021

achuguy commented Mar 23, 2021 •

edited

Loading

madirey commented Mar 23, 2021 •

edited

Loading

achuguy commented Mar 23, 2021

achuguy commented Mar 23, 2021 •

edited

Loading

madirey commented Mar 23, 2021

yctercero commented Mar 23, 2021

spong commented Mar 23, 2021

peluja1012 commented May 21, 2021 •

edited

Loading

elasticmachine commented Jul 26, 2021

MadameSheema commented Jul 26, 2021

peluja1012 commented Nov 9, 2021

[Security Solution][Detections] Error fetching preview when creating rule in index with simple documents #92434

[Security Solution][Detections] Error fetching preview when creating rule in index with simple documents #92434

Comments

achuguy commented Feb 23, 2021 • edited Loading

elasticmachine commented Feb 23, 2021

yctercero commented Mar 16, 2021

MadameSheema commented Mar 22, 2021

achuguy commented Mar 23, 2021 • edited Loading

madirey commented Mar 23, 2021 • edited Loading

achuguy commented Mar 23, 2021

achuguy commented Mar 23, 2021 • edited Loading

madirey commented Mar 23, 2021

yctercero commented Mar 23, 2021

spong commented Mar 23, 2021

peluja1012 commented May 21, 2021 • edited Loading

elasticmachine commented Jul 26, 2021

MadameSheema commented Jul 26, 2021

peluja1012 commented Nov 9, 2021

achuguy commented Feb 23, 2021 •

edited

Loading

achuguy commented Mar 23, 2021 •

edited

Loading

madirey commented Mar 23, 2021 •

edited

Loading

achuguy commented Mar 23, 2021 •

edited

Loading

peluja1012 commented May 21, 2021 •

edited

Loading