Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Infra] Change host count KPI query #188950

Merged
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { dateRt } from '@kbn/io-ts-utils';
import * as rt from 'io-ts';

const AssetTypeRT = rt.type({
assetType: rt.literal('host'),
});

export const GetInfraAssetCountRequestBodyPayloadRT = rt.intersection([
rt.partial({
query: rt.UnknownRecord,
}),
rt.type({
sourceId: rt.string,
from: dateRt,
to: dateRt,
}),
]);

export const GetInfraAssetCountRequestParamsPayloadRT = AssetTypeRT;

export const GetInfraAssetCountResponsePayloadRT = rt.intersection([
AssetTypeRT,
rt.type({
count: rt.number,
}),
]);

export type GetInfraAssetCountRequestParamsPayload = rt.TypeOf<
typeof GetInfraAssetCountRequestParamsPayloadRT
>;
export type GetInfraAssetCountRequestBodyPayload = Omit<
rt.TypeOf<typeof GetInfraAssetCountRequestBodyPayloadRT>,
'from' | 'to'
> & {
from: string;
to: string;
};

export type GetInfraAssetCountResponsePayload = rt.TypeOf<
typeof GetInfraAssetCountResponsePayloadRT
>;
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ export * from './metrics_api';
export * from './snapshot_api';
export * from './host_details';
export * from './infra';
export * from './asset_count_api';

/**
* Exporting versioned APIs types
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import { initNodeDetailsRoute } from './routes/node_details';
import { initOverviewRoute } from './routes/overview';
import { initProcessListRoute } from './routes/process_list';
import { initSnapshotRoute } from './routes/snapshot';
import { initInfraMetricsRoute } from './routes/infra';
import { initInfraAssetRoutes } from './routes/infra';
import { initMetricsExplorerViewRoutes } from './routes/metrics_explorer_views';
import { initProfilingRoutes } from './routes/profiling';
import { initServicesRoute } from './routes/services';
Expand Down Expand Up @@ -67,7 +67,7 @@ export const initInfraServer = (
initGetLogAlertsChartPreviewDataRoute(libs);
initProcessListRoute(libs);
initOverviewRoute(libs);
initInfraMetricsRoute(libs);
initInfraAssetRoutes(libs);
initProfilingRoutes(libs);
initServicesRoute(libs);
initCustomDashboardsRoutes(libs.framework);
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# Infra Hosts API
# Infra Assets API

This API returns a list of hosts and their metrics.
## **POST /api/metrics/infra**

**POST /api/metrics/infra**
parameters:
This endpoint returns a list of hosts and their metrics.

### Parameters:

- type: asset type. 'host' is the only one supported now
- metrics: list of metrics to be calculated and returned for each host
Expand All @@ -18,7 +19,7 @@ The response includes:
- metrics: object containing name of the metric and value
- metadata: object containing name of the metadata and value

## Examples
### Examples:

Request

Expand Down Expand Up @@ -113,3 +114,49 @@ Response
]
}
```

## **POST /api/infra/{assetType}/count**

This endpoint returns the count of the hosts monitored with the system integration.

### Parameters:

- type: asset type. 'host' is the only one supported now
- sourceId: sourceId to retrieve configuration such as index-pattern used to query the results
- from: Start date
- to: End date
- (optional) query: filter

The response includes:

- count: number - the count of the hosts monitored with the system integration
- type: string - the type of the asset **(currently only 'host' is supported)**

### Examples:

Request

```bash
curl --location -u elastic:changeme 'http://0.0.0.0:5601/ftw/api/infra/host/count' \
--header 'kbn-xsrf: xxxx' \
--header 'Content-Type: application/json' \
--data '{
"query": {
"bool": {
"must": [],
"filter": [],
"should": [],
"must_not": []
}
},
"from": "2024-07-23T11:34:11.640Z",
"to": "2024-07-23T11:49:11.640Z",
"sourceId": "default"
}'
```

Response

```json
{"type":"host","count":22}
```
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,27 @@
import Boom from '@hapi/boom';
import { createRouteValidationFunction } from '@kbn/io-ts-utils';

import type { BoolQuery } from '@kbn/es-query';
import {
type GetInfraMetricsRequestBodyPayload,
GetInfraMetricsRequestBodyPayloadRT,
GetInfraMetricsRequestBodyPayload,
GetInfraMetricsResponsePayloadRT,
} from '../../../common/http_api/infra';
import { InfraBackendLibs } from '../../lib/infra_types';
import {
type GetInfraAssetCountRequestBodyPayload,
type GetInfraAssetCountRequestParamsPayload,
GetInfraAssetCountRequestBodyPayloadRT,
GetInfraAssetCountResponsePayloadRT,
GetInfraAssetCountRequestParamsPayloadRT,
} from '../../../common/http_api/asset_count_api';
import type { InfraBackendLibs } from '../../lib/infra_types';
import { getInfraAlertsClient } from '../../lib/helpers/get_infra_alerts_client';
import { getHosts } from './lib/host/get_hosts';
import { getHostsCount } from './lib/host/get_hosts_count';
import { getInfraMetricsClient } from '../../lib/helpers/get_infra_metrics_client';

export const initInfraMetricsRoute = (libs: InfraBackendLibs) => {
const validateBody = createRouteValidationFunction(GetInfraMetricsRequestBodyPayloadRT);
export const initInfraAssetRoutes = (libs: InfraBackendLibs) => {
const validateMetricsBody = createRouteValidationFunction(GetInfraMetricsRequestBodyPayloadRT);

const { framework } = libs;

Expand All @@ -28,7 +37,7 @@ export const initInfraMetricsRoute = (libs: InfraBackendLibs) => {
method: 'post',
path: '/api/metrics/infra',
validate: {
body: validateBody,
body: validateMetricsBody,
},
},
async (requestContext, request, response) => {
Expand Down Expand Up @@ -74,4 +83,64 @@ export const initInfraMetricsRoute = (libs: InfraBackendLibs) => {
}
}
);

const validateCountBody = createRouteValidationFunction(GetInfraAssetCountRequestBodyPayloadRT);
const validateCountParams = createRouteValidationFunction(
GetInfraAssetCountRequestParamsPayloadRT
);

framework.registerRoute(
{
method: 'post',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we this could be a GET request.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to pass the query, filters, time range, etc. so we can even expand more the body if it is a POST. I am not sure how much will those params extend in the future so I was thinking that having the consistency with the other endpoints with similar functions can help here. But I don't have a strong opinion on that. Do you think it's better to have query params instead of the request body and convert to GET?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GET operation is preferable in this case, since nothing is being created. Infra uses POST for historical reasons, but most of them should be GET instead. The complicated part is on how to pass the filters in the query params.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed I created an issue to investigate the option to use GET instead of POST in our APIs #189170 There are some benefits and concerns described there and it would be nice also to keep them consistent and have GET requests where possible ( I totally agree that it will be better but we still need to check if the url limitations affect us) So this change won't be part of this PR

path: '/api/infra/{assetType}/count',
validate: {
body: validateCountBody,
params: validateCountParams,
},
},
async (requestContext, request, response) => {
const body: GetInfraAssetCountRequestBodyPayload = request.body;
const params: GetInfraAssetCountRequestParamsPayload = request.params;
const { assetType } = params;
const { query, from, to, sourceId } = body;

try {
const infraMetricsClient = await getInfraMetricsClient({
framework,
request,
infraSources: libs.sources,
requestContext,
sourceId,
});

const assetCount = await getHostsCount({
infraMetricsClient,
query: (query?.bool as BoolQuery) ?? undefined,
from,
to,
});

return response.ok({
body: GetInfraAssetCountResponsePayloadRT.encode({
assetType,
count: assetCount,
}),
});
} catch (err) {
if (Boom.isBoom(err)) {
return response.customError({
statusCode: err.output.statusCode,
body: { message: err.output.payload.message },
});
}

return response.customError({
statusCode: err.statusCode ?? 500,
body: {
message: err.message ?? 'An unexpected error occurred',
},
});
}
}
);
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { BoolQuery } from '@kbn/es-query';
import { InfraMetricsClient } from '../../../../lib/helpers/get_infra_metrics_client';
import { HOST_NAME_FIELD } from '../../../../../common/constants';

export async function getHostsCount({
infraMetricsClient,
query,
from,
to,
}: {
infraMetricsClient: InfraMetricsClient;
query?: BoolQuery;
from: string;
to: string;
}) {
const rangeQuery = [
{
range: {
'@timestamp': {
gte: from,
lte: to,
},
},
},
];
const queryFilter = query?.filter ?? [];
const queryBool = query ?? {};

const params = {
allow_no_indices: true,
ignore_unavailable: true,
body: {
size: 0,
track_total_hits: false,
query: {
bool: {
...queryBool,
filter: [
...queryFilter,
...rangeQuery,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use the rangeQuery helper function here.

Suggested change
...rangeQuery,
...rangeQuery(new Date(from).getTime(), new Date(to).getTime())

{
bool: {
should: [
{
term: {
'event.module': 'system',
},
},
{
term: {
'metricset.module': 'system',
},
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could use termQuery helper functions. Also, how about creating constants for the field names?

Suggested change
{
term: {
'event.module': 'system',
},
},
{
term: {
'metricset.module': 'system',
},
},
...termQuery(EVENT_MODULE, 'system')
...termQuery(METRICSET_MODULE, 'system'),

],
minimum_should_match: 1,
},
},
],
},
},
aggs: {
count: {
cardinality: {
field: HOST_NAME_FIELD,
},
},
},
},
};

const result = await infraMetricsClient.search(params);

return result.aggregations?.count.value ?? 0;
}
Loading