[Lens] New datasource for Elasticsearch queries as JSON

[wylieconlon]

The Kibana dev tools are popular with users who are familiar with the Elasticsearch REST API, but users can't use the REST API in visualizations because of the abstractions that we provide around querying. Users who want to use the REST API are forced to use Vega even for simple visualizations like a bar chart.

Lens could provide a new datasource that supports the search endpoints in Elasticsearch, and gives users a way to use this in a template.

This is similar to the request to support the query DSL in Canvas

---

Example

The user has documents using the nested mapping type, and wants to visualize the nested data in a bar chart. This is a valid Elasticsearch query, but not one that is supported by esaggs. This example is taken from the nested aggregation docs.

In this example, we are going to use similar templating as Vega to add filters using %context%

GET /products/_search
{
  "query" : %context%,
  "aggs" : {
    "resellers" : {
      "nested" : {
        "path" : "resellers"
      },
      "aggs" : {
        "min_price" : { "min" : { "field" : "resellers.price" } }
      }
    }
  }
}

The result of this query looks like:

{
  ...
  "aggregations": {
    "resellers": {
      "doc_count": 2,
      "min_price": {
        "value": 350
      }
    }
  }
}

The response would be analyzed and presented as a list of paths to be visualized. If any of the values are arrays, different visualizations would be suggested.

• aggregations.resellers.doc_count (Number, Length 1)
• aggregations.resellers.min_price.value (Number, Length 1)

Because these numbers only have length of 1, Lens would suggest that the user visualizes this as a Metric visualization. If the length was 2 or more, Lens would probably suggest the Table visualization.

---

Related to #6713

Related to #57708

Implemented in #69254

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[stratoula]

I am closing this as users now can create Lens charts with ES|QL queries so this is not on the roadmap for now