Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core.logging] Use host timezone in log records by default #90255

Conversation

TinaHeiligers
Copy link
Contributor

@TinaHeiligers TinaHeiligers commented Feb 4, 2021

Summary

Elasticsearch uses the host timezone as a logging default, whereas logging from the Kibana platform uses UTC as the default.

This PR aligns Kibana's logging settings in the Kibana platform with Elasticsearch, defaulting to the host timezone when none is specified for logging.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@TinaHeiligers TinaHeiligers added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:Logging v7.12.0 labels Feb 4, 2021
@TinaHeiligers
Copy link
Contributor Author

@elasticmachine merge upstream

kibanamachine and others added 21 commits February 4, 2021 09:35
…d" button (elastic#90015)

* Rename management button to "delete"

* fix jest

* Delete reload action from management

* Added both cancel and delete session

* Improve texts

* fix test

* ts

* doc

* fix jest
…source (elastic#89371)

* First pass at switching rules to depend on fields instead of _source

* Fix tests

* Change operator: excluded logic so missing fields are allowlisted

Co-authored-by: Kibana Machine <[email protected]>
test user assignment for test files under import geoJSON files

Co-authored-by: Kibana Machine <[email protected]>
)

* enabling yesterday option when 24 hours is selected

* addressing PR comments

* addressing PR comments

* enabling select box
(cherry picked from commit e564439)
Co-authored-by: Alejandro Fernández Haro <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
* Start moving some stuff

* Move some stuff around

* more

* Transactions label

* some snake casing

* i18n fix

* Remove unused ui filters endpoints

* Updates to select

* remove projections

* Use urlHelpers.push

* License change
…oad (elastic#90206)

* Update MockRouter to not pass/set a this.payload

- but instead intelligently validate payloads based on the request keys

* Fix relevance tuning API routes to not need a separate mock router for validating query & body

* Update all remaining tests to no longer pass a payload param to MockRouter
* Remove comment

Verified that this works as expected

* Replaces usage from SourceLogic to AddSourceLogic

* Remove unused duplicate code

Co-authored-by: Kibana Machine <[email protected]>
## Summary
Introduces the concept of a managed agent policy. Resolves most of the acceptance criteria from elastic#76843. Remaining to be done in follow up PRs

- [x] Define hosted Agent Policy concept in Fleet.
    - [x] Flag in policy? **_yes, added `is_managed: boolean`_ in agent policy SO**
    - [x] Should not built only for cloud, an admin should be able to set theses restrictions.
    - [x] We should have an API to configure it _**Can `POST` and `PUT` to  `/api/fleet/agent_policies/{policy_id}`**_
    - [x] Integration should be editable, we expect integration author to do the right thing and limit what can be edited.
- [x] Research if we can ensure the right behavior of Hosted Agent policy and restrict the super user.
- [ ] Capabilities restrictions
  - [ ] An Agent enrolled in an Hosted Agent policy should not be able to be upgraded.
  - [x] An Agent enrolled in an Hosted Agent policy should not be able to be unenrolled.
  - [ ] No Agents cannot be enrolled into this policy by the user.
      - Hide the enrollment key?
      - Need to figure out the workflow.
  - [x] An Agent enrolled in an Hosted Agent policy should not be able to be reassigned to a different configuration.
- [x] As a user I should be prevented to do theses action. _**No user-level checks. Only Agent Policy. No UI changes, but API errors are shown for failed actions like reassigning**_
- [x] As an API user I should receive error messages.
- [x] If making a single "flag" is easier/faster let's do it.  _**Currently single `is_managed` property on agent policy SO.**_

Checks are implemented in service layer (is agent enrolled in a managed policy?)

No UI-specific changes added but UI is affected because HTTP requests (like `api/fleet/agents/{agentId}/reassign`) can fail. See screenshots below.

Tests at service (`yarn test:jest`) and http (`yarn test ftr`) layers for each of create policy, update policy, unenroll agent, and reassign agent

Bulk actions currently filter out restricted items. A follow-up PR will change them to throw an error and cause the request to fail.


## Managed Policy
Can create (`POST`) and update (`PUT`) an agent policy with an `is_managed` property. Each new saved object will have an `is_managed` property (default `false`)

<details><summary>HTTP commands</summary>

#### Create (`is_managed: false` by default)
```
 curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"edc236a0-5cbb-11eb-ab2c-0134aecb4ce8","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-01-22T14:12:58.250Z","updated_by":"elastic"}}
```

#### Create with `is_managed: true`
```
 curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-02-03T14:45:06.059Z","updated_by":"elastic"}}
```

#### Update with `is_managed: true`
```
 curl --user elastic:changeme -X PUT  -H 'Content-Type: application/json' -H 'kbn-xsrf: 1234' localhost:5601/api/fleet/agent_policies/67c785b0-662e-11eb-bf6b-4790dc0178c0 -d '{ "name":"User created policy","namespace":"default","is_managed":true }'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":true,"revision":2,"updated_at":"2021-02-03T14:47:28.471Z","updated_by":"elastic","package_policies":[]}}
```
</details>

## Enroll behavior
is not changed/addressed in this PR. Agents can still be enrolled in managed policies

## Unenroll Agent from managed policy behavior
#### Enrolled in managed agent policy, cannot be unenrolled
```
curl --user elastic:changeme -X POST http://localhost:5601/api/fleet/agents/441d4a40-6710-11eb-8f57-db14e8e41cff/unenroll -H 'kbn-xsrf: 1234' | jq
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Cannot unenroll 441d4a40-6710-11eb-8f57-db14e8e41cff from a managed agent policy af9b4970-6701-11eb-b55a-899b78cb64da"
}
```

<details><summary>Screenshots for managed & unmanaged policies</summary>

#### Enrolled in managed agent policy, cannot be unenrolled
<img width="1931" alt="Screen Shot 2021-01-19 at 1 22 53 PM" src="https://user-images.githubusercontent.com/57655/105081614-67d05980-5a60-11eb-8faa-07e4e722a5b5.png">
<img width="1199" alt="Screen Shot 2021-01-19 at 1 30 26 PM" src="https://user-images.githubusercontent.com/57655/105081617-67d05980-5a60-11eb-9099-832dc6e04eca.png">
<img width="1971" alt="Screen Shot 2021-01-19 at 1 30 42 PM" src="https://user-images.githubusercontent.com/57655/105081618-67d05980-5a60-11eb-9a84-b80b6295ba19.png">

#### Enrolled agent policy is not managed, agent can be unenrolled<img width="1917" alt="Screen Shot 2021-01-19 at 1 44 12 PM" src="https://user-images.githubusercontent.com/57655/105081951-e3caa180-5a60-11eb-9308-7741b8986e8e.png">
<img width="2183" alt="Screen Shot 2021-01-19 at 1 44 19 PM" src="https://user-images.githubusercontent.com/57655/105081952-e3caa180-5a60-11eb-9833-1c721be0a107.png">

</details>


## Reassign agent 
#### No agent can be reassigned to a managed policy
```
 curl --user elastic:changeme -X 'PUT'  'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}' 
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Cannot reassign an agent to managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots</summary>

<img width="1350" alt="Screen Shot 2021-02-04 at 2 14 51 PM" src="https://user-images.githubusercontent.com/57655/106943490-8044a300-66f3-11eb-9d2c-4b1ceef2e783.png">

</details>

#### Enrolled in managed agent policy, cannot be reassigned
```
 curl --user elastic:changeme -X 'PUT'  'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}' 
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Cannot reassign an agent from managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```

<details><summary>Screenshots</summary>
<img width="1364" alt="Screen Shot 2021-01-19 at 2 58 38 PM" src="https://user-images.githubusercontent.com/57655/105086737-72dab800-5a67-11eb-8f5e-93cd7768b914.png">
<img width="1367" alt="Screen Shot 2021-01-19 at 2 58 44 PM" src="https://user-images.githubusercontent.com/57655/105086740-73734e80-5a67-11eb-8ef9-9c7005a0a4ea.png">
<img width="623" alt="Screen Shot 2021-01-19 at 2 59 27 PM" src="https://user-images.githubusercontent.com/57655/105086741-740be500-5a67-11eb-8fc2-721f8b5d178a.png">
</details>

#### Enrolled agent policy is unmanaged, agent can be reassigned to another unmanaged policy

<details><summary>Screenshots</summary>
<img width="1368" alt="Screen Shot 2021-01-19 at 3 00 01 PM" src="https://user-images.githubusercontent.com/57655/105086754-78d09900-5a67-11eb-86a5-9e3ac02d6e1f.png">
<img width="1363" alt="Screen Shot 2021-01-19 at 3 00 08 PM" src="https://user-images.githubusercontent.com/57655/105086761-7a01c600-5a67-11eb-991d-acf994e2a393.png">
<img width="625" alt="Screen Shot 2021-01-19 at 3 00 46 PM" src="https://user-images.githubusercontent.com/57655/105086764-7a9a5c80-5a67-11eb-8290-e79648d01579.png">
</details>

### Checklist

Delete any items that are not applicable to this PR.

- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…y plugins public services, types, and functionality (elastic#86704)

* wip RFC for API doc infra

* update

* update

* rfc

* rfc

* Update RFC

* Update RFC post Arch Review

* add pr link

* Update based on review feedback

* Update 0014_api_documentation.md

Co-authored-by: Kibana Machine <[email protected]>
…ligers/kibana into logging/es-compatible-tz-default
@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky


Test Failures

Kibana Pipeline / general / X-Pack API Integration Tests.x-pack/test/api_integration/apis/management/index_management/indices·js.apis management index management indices list should list all the indices with the expected properties and data enrichers

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/64473

[00:00:00]       │
[00:00:00]         └-: apis
[00:00:00]           └-> "before all" hook in "apis"
[00:04:08]           └-: management
[00:04:08]             └-> "before all" hook in "management"
[00:04:26]             └-: index management
[00:04:26]               └-> "before all" hook in "index management"
[00:04:26]               └-: indices
[00:04:26]                 └-> "before all" hook in "indices"
[00:04:27]                 └-: list
[00:04:27]                   └-> "before all" hook for "should list all the indices with the expected properties and data enrichers"
[00:04:27]                   └-> should list all the indices with the expected properties and data enrichers
[00:04:27]                     └-> "before each" hook: global before each for "should list all the indices with the expected properties and data enrichers"
[00:04:27]                     └- ✖ fail: apis management index management indices list should list all the indices with the expected properties and data enrichers
[00:04:27]                     │       Error: expected [ 'aliases',
[00:04:27]                     │   'data_stream',
[00:04:27]                     │   'documents',
[00:04:27]                     │   'health',
[00:04:27]                     │   'hidden',
[00:04:27]                     │   'ilm',
[00:04:27]                     │   'isFollowerIndex',
[00:04:27]                     │   'isFrozen',
[00:04:27]                     │   'isRollupIndex',
[00:04:27]                     │   'name',
[00:04:27]                     │   'primary',
[00:04:27]                     │   'replica',
[00:04:27]                     │   'size',
[00:04:27]                     │   'status',
[00:04:27]                     │   'uuid' ] to sort of equal [ 'aliases',
[00:04:27]                     │   'documents',
[00:04:27]                     │   'health',
[00:04:27]                     │   'hidden',
[00:04:27]                     │   'ilm',
[00:04:27]                     │   'isFollowerIndex',
[00:04:27]                     │   'isFrozen',
[00:04:27]                     │   'isRollupIndex',
[00:04:27]                     │   'name',
[00:04:27]                     │   'primary',
[00:04:27]                     │   'replica',
[00:04:27]                     │   'size',
[00:04:27]                     │   'status',
[00:04:27]                     │   'uuid' ]
[00:04:27]                     │       + expected - actual
[00:04:27]                     │ 
[00:04:27]                     │        [
[00:04:27]                     │          "aliases"
[00:04:27]                     │       -  "data_stream"
[00:04:27]                     │          "documents"
[00:04:27]                     │          "health"
[00:04:27]                     │          "hidden"
[00:04:27]                     │          "ilm"
[00:04:27]                     │       
[00:04:27]                     │       at Assertion.assert (/dev/shm/workspace/parallel/2/kibana/packages/kbn-expect/expect.js:100:11)
[00:04:27]                     │       at Assertion.eql (/dev/shm/workspace/parallel/2/kibana/packages/kbn-expect/expect.js:244:8)
[00:04:27]                     │       at Context.<anonymous> (test/api_integration/apis/management/index_management/indices.js:206:39)
[00:04:27]                     │       at Object.apply (/dev/shm/workspace/parallel/2/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16)
[00:04:27]                     │ 
[00:04:27]                     │ 

Stack Trace

Error: expected [ 'aliases',
  'data_stream',
  'documents',
  'health',
  'hidden',
  'ilm',
  'isFollowerIndex',
  'isFrozen',
  'isRollupIndex',
  'name',
  'primary',
  'replica',
  'size',
  'status',
  'uuid' ] to sort of equal [ 'aliases',
  'documents',
  'health',
  'hidden',
  'ilm',
  'isFollowerIndex',
  'isFrozen',
  'isRollupIndex',
  'name',
  'primary',
  'replica',
  'size',
  'status',
  'uuid' ]
    at Assertion.assert (/dev/shm/workspace/parallel/2/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/2/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.<anonymous> (test/api_integration/apis/management/index_management/indices.js:206:39)
    at Object.apply (/dev/shm/workspace/parallel/2/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16) {
  actual: '[\n' +
    '  "aliases"\n' +
    '  "data_stream"\n' +
    '  "documents"\n' +
    '  "health"\n' +
    '  "hidden"\n' +
    '  "ilm"\n' +
    '  "isFollowerIndex"\n' +
    '  "isFrozen"\n' +
    '  "isRollupIndex"\n' +
    '  "name"\n' +
    '  "primary"\n' +
    '  "replica"\n' +
    '  "size"\n' +
    '  "status"\n' +
    '  "uuid"\n' +
    ']',
  expected: '[\n' +
    '  "aliases"\n' +
    '  "documents"\n' +
    '  "health"\n' +
    '  "hidden"\n' +
    '  "ilm"\n' +
    '  "isFollowerIndex"\n' +
    '  "isFrozen"\n' +
    '  "isRollupIndex"\n' +
    '  "name"\n' +
    '  "primary"\n' +
    '  "replica"\n' +
    '  "size"\n' +
    '  "status"\n' +
    '  "uuid"\n' +
    ']',
  showDiff: true
}

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@TinaHeiligers TinaHeiligers deleted the logging/es-compatible-tz-default branch March 23, 2021 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Logging release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.12.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.