Skip to content

Commit

Permalink
[SavedObjects] Add aggregations support (#96292)
Browse files Browse the repository at this point in the history
* step 1 to add aggs in the find function of saved object

* setp 2 - add specific unit test to aggs + fix bug found during integrations

* step 3 - add security api_integration arounds aggs

* fix types

* unit test added for aggs_utils

* add documentation

* fix docs

* review I

* doc

* try to fix test

* add the new property to the saved object globaltype

* fix types

* delete old files

* fix types + test api integration

* type fix + test

* Update src/core/server/saved_objects/types.ts

Co-authored-by: Rudolf Meijering <[email protected]>

* review I

* change our validation to match discussion with Pierre and Rudolph

* Validate multiple items nested filter query through KueryNode

* remove unused import

* review + put back test

* migrate added tests to new TS file

* fix documentation

* fix license header

* move stuff

* duplicating test mappings

* rename some stuff

* move ALL the things

* cast to aggregation container

* update generated doc

* add deep nested validation

* rewrite the whole validation mechanism

* some cleanup

* minor cleanup

* update generated doc

* adapt telemetry client

* fix API integ tests

* fix doc

* TOTO-less

* remove xpack tests

* list supported / unsupported aggregations

* typo fix

* extract some validation function

* fix indent

* add some unit tests

* adapt FTR assertions

* update doc

* fix doc

* doc again

* cleanup test names

* improve tsdoc on validation functions

* perf nit

Co-authored-by: Xavier Mouligneau <[email protected]>
Co-authored-by: Rudolf Meijering <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
  • Loading branch information
4 people authored Apr 16, 2021
1 parent 15e8ca1 commit 106afd4
Show file tree
Hide file tree
Showing 34 changed files with 1,369 additions and 69 deletions.
11 changes: 8 additions & 3 deletions docs/api/saved-objects/find.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,14 @@ experimental[] Retrieve a paginated set of {kib} saved objects by various condit
(Optional, object) Filters to objects that have a relationship with the type and ID combination.

`filter`::
(Optional, string) The filter is a KQL string with the caveat that if you filter with an attribute from your type saved object.
It should look like that savedObjectType.attributes.title: "myTitle". However, If you used a direct attribute of a saved object like `updatedAt`,
you will have to define your filter like that savedObjectType.updatedAt > 2018-12-22.
(Optional, string) The filter is a KQL string with the caveat that if you filter with an attribute from your saved object type,
it should look like that: `savedObjectType.attributes.title: "myTitle"`. However, If you use a root attribute of a saved
object such as `updated_at`, you will have to define your filter like that: `savedObjectType.updated_at > 2018-12-22`.

`aggs`::
(Optional, string) **experimental** An aggregation structure, serialized as a string. The field format is similar to `filter`, meaning
that to use a saved object type attribute in the aggregation, the `savedObjectType.attributes.title`: "myTitle"` format
must be used. For root fields, the syntax is `savedObjectType.rootField`

NOTE: As objects change in {kib}, the results on each page of the response also
change. Use the find API for traditional paginated results, but avoid using it to export large amounts of data.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ Search for objects
<b>Signature:</b>

```typescript
find: <T = unknown>(options: SavedObjectsFindOptions) => Promise<SavedObjectsFindResponsePublic<T>>;
find: <T = unknown, A = unknown>(options: SavedObjectsFindOptions) => Promise<SavedObjectsFindResponsePublic<T, unknown>>;
```
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The constructor for this class is marked as internal. Third-party code should no
| [bulkGet](./kibana-plugin-core-public.savedobjectsclient.bulkget.md) | | <code>(objects?: Array&lt;{</code><br/><code> id: string;</code><br/><code> type: string;</code><br/><code> }&gt;) =&gt; Promise&lt;SavedObjectsBatchResponse&lt;unknown&gt;&gt;</code> | Returns an array of objects by id |
| [create](./kibana-plugin-core-public.savedobjectsclient.create.md) | | <code>&lt;T = unknown&gt;(type: string, attributes: T, options?: SavedObjectsCreateOptions) =&gt; Promise&lt;SimpleSavedObject&lt;T&gt;&gt;</code> | Persists an object |
| [delete](./kibana-plugin-core-public.savedobjectsclient.delete.md) | | <code>(type: string, id: string, options?: SavedObjectsDeleteOptions &#124; undefined) =&gt; ReturnType&lt;SavedObjectsApi['delete']&gt;</code> | Deletes an object |
| [find](./kibana-plugin-core-public.savedobjectsclient.find.md) | | <code>&lt;T = unknown&gt;(options: SavedObjectsFindOptions) =&gt; Promise&lt;SavedObjectsFindResponsePublic&lt;T&gt;&gt;</code> | Search for objects |
| [find](./kibana-plugin-core-public.savedobjectsclient.find.md) | | <code>&lt;T = unknown, A = unknown&gt;(options: SavedObjectsFindOptions) =&gt; Promise&lt;SavedObjectsFindResponsePublic&lt;T, unknown&gt;&gt;</code> | Search for objects |
| [get](./kibana-plugin-core-public.savedobjectsclient.get.md) | | <code>&lt;T = unknown&gt;(type: string, id: string) =&gt; Promise&lt;SimpleSavedObject&lt;T&gt;&gt;</code> | Fetches a single object |

## Methods
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-core-public](./kibana-plugin-core-public.md) &gt; [SavedObjectsFindResponsePublic](./kibana-plugin-core-public.savedobjectsfindresponsepublic.md) &gt; [aggregations](./kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md)

## SavedObjectsFindResponsePublic.aggregations property

<b>Signature:</b>

```typescript
aggregations?: A;
```
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,14 @@ Return type of the Saved Objects `find()` method.
<b>Signature:</b>

```typescript
export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T>
export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown> extends SavedObjectsBatchResponse<T>
```
## Properties
| Property | Type | Description |
| --- | --- | --- |
| [aggregations](./kibana-plugin-core-public.savedobjectsfindresponsepublic.aggregations.md) | <code>A</code> | |
| [page](./kibana-plugin-core-public.savedobjectsfindresponsepublic.page.md) | <code>number</code> | |
| [perPage](./kibana-plugin-core-public.savedobjectsfindresponsepublic.perpage.md) | <code>number</code> | |
| [total](./kibana-plugin-core-public.savedobjectsfindresponsepublic.total.md) | <code>number</code> | |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Find all SavedObjects matching the search query
<b>Signature:</b>

```typescript
find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T>>;
find<T = unknown, A = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T, A>>;
```
## Parameters
Expand All @@ -20,5 +20,5 @@ find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindRes
<b>Returns:</b>
`Promise<SavedObjectsFindResponse<T>>`
`Promise<SavedObjectsFindResponse<T, A>>`
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<!-- Do not edit this file. It is automatically generated by API Documenter. -->

[Home](./index.md) &gt; [kibana-plugin-core-server](./kibana-plugin-core-server.md) &gt; [SavedObjectsFindResponse](./kibana-plugin-core-server.savedobjectsfindresponse.md) &gt; [aggregations](./kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md)

## SavedObjectsFindResponse.aggregations property

<b>Signature:</b>

```typescript
aggregations?: A;
```
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,14 @@ Return type of the Saved Objects `find()` method.
<b>Signature:</b>

```typescript
export interface SavedObjectsFindResponse<T = unknown>
export interface SavedObjectsFindResponse<T = unknown, A = unknown>
```

## Properties

| Property | Type | Description |
| --- | --- | --- |
| [aggregations](./kibana-plugin-core-server.savedobjectsfindresponse.aggregations.md) | <code>A</code> | |
| [page](./kibana-plugin-core-server.savedobjectsfindresponse.page.md) | <code>number</code> | |
| [per\_page](./kibana-plugin-core-server.savedobjectsfindresponse.per_page.md) | <code>number</code> | |
| [pit\_id](./kibana-plugin-core-server.savedobjectsfindresponse.pit_id.md) | <code>string</code> | |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<b>Signature:</b>

```typescript
find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T>>;
find<T = unknown, A = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindResponse<T, A>>;
```
## Parameters
Expand All @@ -18,7 +18,7 @@ find<T = unknown>(options: SavedObjectsFindOptions): Promise<SavedObjectsFindRes
<b>Returns:</b>
`Promise<SavedObjectsFindResponse<T>>`
`Promise<SavedObjectsFindResponse<T, A>>`
{<!-- -->promise<!-- -->} - { saved\_objects: \[{ id, type, version, attributes }<!-- -->\], total, per\_page, page }
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ Creates an empty response for a find operation. This is only intended to be used
<b>Signature:</b>

```typescript
static createEmptyFindResponse: <T>({ page, perPage, }: SavedObjectsFindOptions) => SavedObjectsFindResponse<T>;
static createEmptyFindResponse: <T, A>({ page, perPage, }: SavedObjectsFindOptions) => SavedObjectsFindResponse<T, A>;
```
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ export declare class SavedObjectsUtils

| Property | Modifiers | Type | Description |
| --- | --- | --- | --- |
| [createEmptyFindResponse](./kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md) | <code>static</code> | <code>&lt;T&gt;({ page, perPage, }: SavedObjectsFindOptions) =&gt; SavedObjectsFindResponse&lt;T&gt;</code> | Creates an empty response for a find operation. This is only intended to be used by saved objects client wrappers. |
| [createEmptyFindResponse](./kibana-plugin-core-server.savedobjectsutils.createemptyfindresponse.md) | <code>static</code> | <code>&lt;T, A&gt;({ page, perPage, }: SavedObjectsFindOptions) =&gt; SavedObjectsFindResponse&lt;T, A&gt;</code> | Creates an empty response for a find operation. This is only intended to be used by saved objects client wrappers. |
| [namespaceIdToString](./kibana-plugin-core-server.savedobjectsutils.namespaceidtostring.md) | <code>static</code> | <code>(namespace?: string &#124; undefined) =&gt; string</code> | Converts a given saved object namespace ID to its string representation. All namespace IDs have an identical string representation, with the exception of the <code>undefined</code> namespace ID (which has a namespace string of <code>'default'</code>). |
| [namespaceStringToId](./kibana-plugin-core-server.savedobjectsutils.namespacestringtoid.md) | <code>static</code> | <code>(namespace: string) =&gt; string &#124; undefined</code> | Converts a given saved object namespace string to its ID representation. All namespace strings have an identical ID representation, with the exception of the <code>'default'</code> namespace string (which has a namespace ID of <code>undefined</code>). |

Expand Down
8 changes: 6 additions & 2 deletions src/core/public/public.api.md
Original file line number Diff line number Diff line change
Expand Up @@ -1224,7 +1224,7 @@ export class SavedObjectsClient {
// Warning: (ae-forgotten-export) The symbol "SavedObjectsClientContract" needs to be exported by the entry point index.d.ts
delete: (type: string, id: string, options?: SavedObjectsDeleteOptions | undefined) => ReturnType<SavedObjectsClientContract_2['delete']>;
// Warning: (ae-forgotten-export) The symbol "SavedObjectsFindOptions" needs to be exported by the entry point index.d.ts
find: <T = unknown>(options: SavedObjectsFindOptions_2) => Promise<SavedObjectsFindResponsePublic<T>>;
find: <T = unknown, A = unknown>(options: SavedObjectsFindOptions_2) => Promise<SavedObjectsFindResponsePublic<T, unknown>>;
get: <T = unknown>(type: string, id: string) => Promise<SimpleSavedObject<T>>;
update<T = unknown>(type: string, id: string, attributes: T, { version, migrationVersion, references }?: SavedObjectsUpdateOptions): Promise<SimpleSavedObject<T>>;
}
Expand All @@ -1244,6 +1244,8 @@ export interface SavedObjectsCreateOptions {

// @public (undocumented)
export interface SavedObjectsFindOptions {
// @alpha
aggs?: Record<string, estypes.AggregationContainer>;
defaultSearchOperator?: 'AND' | 'OR';
fields?: string[];
// Warning: (ae-forgotten-export) The symbol "KueryNode" needs to be exported by the entry point index.d.ts
Expand Down Expand Up @@ -1284,7 +1286,9 @@ export interface SavedObjectsFindOptionsReference {
}

// @public
export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> {
export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown> extends SavedObjectsBatchResponse<T> {
// (undocumented)
aggregations?: A;
// (undocumented)
page: number;
// (undocumented)
Expand Down
14 changes: 12 additions & 2 deletions src/core/public/saved_objects/saved_objects_client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,9 @@ export interface SavedObjectsDeleteOptions {
*
* @public
*/
export interface SavedObjectsFindResponsePublic<T = unknown> extends SavedObjectsBatchResponse<T> {
export interface SavedObjectsFindResponsePublic<T = unknown, A = unknown>
extends SavedObjectsBatchResponse<T> {
aggregations?: A;
total: number;
perPage: number;
page: number;
Expand Down Expand Up @@ -310,7 +312,7 @@ export class SavedObjectsClient {
* @property {object} [options.hasReference] - { type, id }
* @returns A find result with objects matching the specified search.
*/
public find = <T = unknown>(
public find = <T = unknown, A = unknown>(
options: SavedObjectsFindOptions
): Promise<SavedObjectsFindResponsePublic<T>> => {
const path = this.getPath(['_find']);
Expand All @@ -326,6 +328,7 @@ export class SavedObjectsClient {
sortField: 'sort_field',
type: 'type',
filter: 'filter',
aggs: 'aggs',
namespaces: 'namespaces',
preference: 'preference',
};
Expand All @@ -342,13 +345,20 @@ export class SavedObjectsClient {
query.has_reference = JSON.stringify(query.has_reference);
}

// `aggs` is a structured object. we need to stringify it before sending it, as `fetch`
// is not doing it implicitly.
if (query.aggs) {
query.aggs = JSON.stringify(query.aggs);
}

const request: ReturnType<SavedObjectsApi['find']> = this.savedObjectsFetch(path, {
method: 'GET',
query,
});
return request.then((resp) => {
return renameKeys<SavedObjectsFindResponse, SavedObjectsFindResponsePublic>(
{
aggregations: 'aggregations',
saved_objects: 'savedObjects',
total: 'total',
per_page: 'perPage',
Expand Down
16 changes: 16 additions & 0 deletions src/core/server/saved_objects/routes/find.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
has_reference_operator: searchOperatorSchema,
fields: schema.maybe(schema.oneOf([schema.string(), schema.arrayOf(schema.string())])),
filter: schema.maybe(schema.string()),
aggs: schema.maybe(schema.string()),
namespaces: schema.maybe(
schema.oneOf([schema.string(), schema.arrayOf(schema.string())])
),
Expand All @@ -59,6 +60,20 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
const usageStatsClient = coreUsageData.getClient();
usageStatsClient.incrementSavedObjectsFind({ request: req }).catch(() => {});

// manually validation to avoid using JSON.parse twice
let aggs;
if (query.aggs) {
try {
aggs = JSON.parse(query.aggs);
} catch (e) {
return res.badRequest({
body: {
message: 'invalid aggs value',
},
});
}
}

const result = await context.core.savedObjects.client.find({
perPage: query.per_page,
page: query.page,
Expand All @@ -72,6 +87,7 @@ export const registerFindRoute = (router: IRouter, { coreUsageData }: RouteDepen
hasReferenceOperator: query.has_reference_operator,
fields: typeof query.fields === 'string' ? [query.fields] : query.fields,
filter: query.filter,
aggs,
namespaces,
});

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { schema as s, ObjectType } from '@kbn/config-schema';

/**
* Schemas for the Bucket aggregations.
*
* Currently supported:
* - filter
* - histogram
* - terms
*
* Not implemented:
* - adjacency_matrix
* - auto_date_histogram
* - children
* - composite
* - date_histogram
* - date_range
* - diversified_sampler
* - filters
* - geo_distance
* - geohash_grid
* - geotile_grid
* - global
* - ip_range
* - missing
* - multi_terms
* - nested
* - parent
* - range
* - rare_terms
* - reverse_nested
* - sampler
* - significant_terms
* - significant_text
* - variable_width_histogram
*/
export const bucketAggsSchemas: Record<string, ObjectType> = {
filter: s.object({
term: s.recordOf(s.string(), s.oneOf([s.string(), s.boolean(), s.number()])),
}),
histogram: s.object({
field: s.maybe(s.string()),
interval: s.maybe(s.number()),
min_doc_count: s.maybe(s.number()),
extended_bounds: s.maybe(
s.object({
min: s.number(),
max: s.number(),
})
),
hard_bounds: s.maybe(
s.object({
min: s.number(),
max: s.number(),
})
),
missing: s.maybe(s.number()),
keyed: s.maybe(s.boolean()),
order: s.maybe(
s.object({
_count: s.string(),
_key: s.string(),
})
),
}),
terms: s.object({
field: s.maybe(s.string()),
collect_mode: s.maybe(s.string()),
exclude: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
include: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
execution_hint: s.maybe(s.string()),
missing: s.maybe(s.number()),
min_doc_count: s.maybe(s.number()),
size: s.maybe(s.number()),
show_term_doc_count_error: s.maybe(s.boolean()),
order: s.maybe(s.oneOf([s.literal('asc'), s.literal('desc')])),
}),
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { bucketAggsSchemas } from './bucket_aggs';
import { metricsAggsSchemas } from './metrics_aggs';

export const aggregationSchemas = {
...metricsAggsSchemas,
...bucketAggsSchemas,
};
Loading

0 comments on commit 106afd4

Please sign in to comment.