Beats enrollment subcommand #7182

exekias · 2018-05-27T22:12:45Z

This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations.

To test this:

Use the following branches:
- Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats
- Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats
Retrieve a valid enrollment token:

curl  \                             
  -u elastic \           
  -H 'kbn-xsrf: foobar'  \
  -H 'Content-Type: application/json' \
  -X POST \
  http://localhost:5601/api/beats/enrollment_tokens

Use it:

<beat> enroll http://localhost:5601 <enrollment_token>

Check agent is enrolled:

curl http://localhost:5601/api/beats/agents | jq

This is part of #7028, closes #7032

andrewkroh · 2018-05-28T11:27:28Z

libbeat/management/config.go

+
+// Load settings from its source file
+func (c *Config) Load() error {
+	path := paths.Resolve(paths.Data, "management.yaml")


I applaud your attempt to use the official YAML file extension, but we seem to use .yml for everything.

ruflin · 2018-05-28T14:44:08Z

libbeat/cmd/enroll.go

+func genEnrollCmd(name, version string) *cobra.Command {
+	keystoreCmd := cobra.Command{
+		Use:   "enroll <kibana_url> <enrollment_token>",
+		Short: "Enroll Kibana for Central Management",


Entroll in Kibana?

ruflin · 2018-05-28T14:46:01Z

libbeat/setup/kibana/client.go

+}
+
+// NewClientWithConfig creates and returns a kibana client using the given config
+func NewClientWithConfig(config *Config) (*Client, error) {


Not relate to this PR but it seems Kibana becomes more then just for setup. Perhaps we should move this into libbeat/kibana/client in the future.

I was planning to extract the kibana code to a different PR anyway, will move it 👍

ruflin · 2018-05-28T14:46:57Z

libbeat/management/config.go

+	}
+
+	// write temporary file first
+	// TODO this should be owned by the beats user, ensure that


I remember we even have some reusable code for this to make sure it's the correct user.

exekias · 2018-05-30T14:29:48Z

Moved kibana client change to #7211

exekias · 2018-08-21T16:41:59Z

Thank you for the reviews!

I've moved the code to the proper folders, this one should be ready now

We don't have permissions to retrieve that endpoint, so it would fail with a 401 error

ruflin · 2018-08-22T14:24:04Z

CHANGELOG.asciidoc

@@ -45,6 +45,8 @@ https://github.com/elastic/beats/compare/v6.4.0...master[Check the HEAD diff]
 *Auditbeat*

 - Fixed a crash in the file_integrity module under Linux. {issue}7753[7753]
+- Fixed a data race in the file_integrity module. {issue}8009[8009]


Probably from rebasing?

It seems this was a glitch from Github, I changed the last commit and it got fixed

ruflin · 2018-08-23T14:08:12Z

x-pack/libbeat/cmd/enroll.go

+	b, err := instance.NewBeat(name, "", version)
+
+	if err != nil {
+		return nil, fmt.Errorf("error initializing beat: %s", err)


error creating beat. To make sure the two error messages are different.

ruflin · 2018-08-23T14:09:04Z

x-pack/libbeat/cmd/enroll.go

+		Run: cli.RunWith(func(cmd *cobra.Command, args []string) error {
+			beat, err := getBeat(name, version)
+			kibanaURL := args[0]
+			enrollmentToken := args[1]


What happens if we don't have enough arguments?

cobra library handles that:

X1 :: beats/x-pack/metricbeat ‹enrollment*› » go run main.go -c ../../metricbeat/metricbeat.yml enroll 1 ↵ Error: accepts 2 arg(s), received 0 Usage: metricbeat enroll <kibana_url> <enrollment_token> [flags]

ruflin · 2018-08-23T14:10:00Z

x-pack/libbeat/cmd/enroll.go

+				return errors.Wrap(err, "Error while enrolling")
+			}
+
+			fmt.Println("Enrolled and ready to retrieve settings from Kibana")


As we know the Kibana url, we could share it here directly (without passwords)

I would need to parse it to remove credentials, having that the user has just inputted it for the command I'm not sure this is worth it, wdyt?

Nice to have, not a requirement :-)

ruflin · 2018-08-23T14:12:18Z

x-pack/libbeat/management/api/client.go

+		err = extractError(result)
+	} else {
+		if err = json.Unmarshal(result, message); err != nil {
+			return statusCode, errors.Wrap(err, "error unmarshaling response")


error unmarshaling Kibana response

ruflin · 2018-08-23T14:12:26Z

x-pack/libbeat/management/api/client.go

+		Message string
+	}
+	if err := json.Unmarshal(result, &kibanaResult); err != nil {
+		return errors.Wrap(err, "parsing kibana response")


nit Kibana

ruflin · 2018-08-23T14:13:56Z

x-pack/libbeat/management/api/enroll_test.go

+	"github.com/stretchr/testify/assert"
+)
+
+func TestEnrollValid(t *testing.T) {


Nice test 👍

ruflin · 2018-08-23T14:14:22Z

x-pack/libbeat/management/api/enroll_test.go

+		t.Fatal(err)
+	}
+
+	assert.Equal(t, "fooo", accessToken)


Perhaps make fooo a variable/const

ruflin · 2018-08-23T14:19:21Z

x-pack/libbeat/management/config.go

+		return err
+	}
+
+	// write temporary file first


Don't we have this code already in libbeat for Filebeat?

I didn't find any common code to perform this

~~I think this is what @ruflin is talking about, he wrotes it :)~~

https://github.com/elastic/beats/blob/870fffca8ad6590b8e794cc259cc58a18d0d9af0/libbeat/common/file/helper_other.go#L28-L46

I was too quick to comment here.

the code is os specific: https://github.com/elastic/beats/blob/870fffca8ad6590b8e794cc259cc58a18d0d9af0/libbeat/common/file/helper_windows.go The new / old part we only need on Windows.

Yep, I'm using that one, good to know it has specific implementation for windows 👍

I was actually hoping we have a generic function for just writeData that does also the open file magic, but it seems we just have the same code in multiple places :-(

ruflin · 2018-08-23T14:19:32Z

x-pack/libbeat/management/config.go

+
+// Load settings from its source file
+func (c *Config) Load() error {
+	path := paths.Resolve(paths.Data, "management.yml")


How is this file exactly used? The data received from Central Management will be written into this? I think I initially got confused by it that it's called config. Interesting further down you call it settings but probably that is similar to config.

Don't have a better proposal yet but would be nice to make it clear this is automtically generated data and not meant to be edited by the user (right?).

As you said, this file stores internal data for central management, like kibana endpoint settings or cache of retrieved configs. It's not meant to be edited by the user

What about naming the object Management? Perhaps it's just me but when I read Config and see unpack, I'm thinking of our beat.yml file :-)

I plan to add a Management/Manager object, as this is management.Config I think the namespace clarifies it. What about renaming it to Settings to avoid the Config term?

It could help but let's discuss it again when you add the management/manager object.

ruflin · 2018-08-23T14:21:58Z

x-pack/libbeat/management/enroll.go

+	}
+
+	// Enrolled, persist state
+	// TODO use beat.Keystore() for access_token


As soon as the keystore is used, will the management.yml file become obsolete or still used for the ohter data?

I think we would keep management.yml, as it contains structured data and state, keystore doesn't seem the best place for this as it may be too obscure?

I would consider this file similar to Filebeat registry, it's something internal, but good to have it around for debugging / understanding what's going on

Yes, agree we should keep it around. I wonder if we should in the future use the new "registry" writer for it, but still a different file. So all our state documents have the same structure (@urso FYI).

ph · 2018-08-23T18:52:04Z

x-pack/libbeat/management/api/enroll_test.go

+		assert.Equal(t, "6.3.0", request.Version)
+
+		fmt.Fprintf(w, `{"access_token": "fooo"}`)
+	}))


Thanks for the test @exekias I used the same pattern for the License checker! 👍

ruflin · 2018-08-24T12:29:52Z

x-pack/libbeat/management/config.go

+		return err
+	}
+
+	// write temporary file first


I was actually hoping we have a generic function for just writeData that does also the open file magic, but it seems we just have the same code in multiple places :-(

ruflin · 2018-08-24T12:32:41Z

Should I squash and use the PR message as commit message?

This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032

* Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog

* Beats enrollment subcommand (elastic#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of elastic#7028, closes elastic#7032 * Add API client to retrieve configurations from CM (elastic#8155) * Add central management service (elastic#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (elastic#8378) * Also send beat name when enrolling (elastic#8380) * Refactor how configs are stored (elastic#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (elastic#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (elastic#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3)

* Add Central Management feature (#8559) * Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3) * Fix monitoring registry usage

This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of elastic#7028, closes elastic#7032

exekias added in progress Pull request is currently in progress. review libbeat labels May 27, 2018

exekias force-pushed the enrollment branch 3 times, most recently from e678420 to be92b42 Compare May 27, 2018 22:32

andrewkroh reviewed May 28, 2018

View reviewed changes

ruflin reviewed May 28, 2018

View reviewed changes

exekias force-pushed the enrollment branch 3 times, most recently from 0bc1a3f to aa2c7f6 Compare July 12, 2018 15:32

exekias force-pushed the enrollment branch from aa2c7f6 to e20a58a Compare August 21, 2018 16:40

exekias removed the in progress Pull request is currently in progress. label Aug 21, 2018

Carlos Pérez-Aradros Herce added 7 commits August 22, 2018 11:41

Add central management API client

88e1fd2

Add enrollment client API and methods

a5ac0e5

Add enroll subcommand

36ac88c

Adapt to latest changes in master

24d00d9

Apply some review comments

30539fd

Move enroll command to xpack folder

d0718e3

Do not fetch Kibana version when enrolling

df24592

We don't have permissions to retrieve that endpoint, so it would fail with a 401 error

exekias force-pushed the enrollment branch from a40a031 to f7dd454 Compare August 22, 2018 09:41

ruflin reviewed Aug 22, 2018

View reviewed changes

Fix name

75893ce

exekias force-pushed the enrollment branch from f7dd454 to 75893ce Compare August 22, 2018 14:47

ruflin requested changes Aug 23, 2018

View reviewed changes

clarify error messages

2c9d937

ph reviewed Aug 23, 2018

View reviewed changes

ruflin approved these changes Aug 24, 2018

View reviewed changes

ruflin merged commit c1fb2f4 into elastic:feature/centralmgmt Aug 24, 2018

Beats enrollment subcommand #7182

Beats enrollment subcommand #7182

Conversation

exekias commented May 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

exekias commented May 30, 2018

exekias commented Aug 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph Aug 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented Aug 24, 2018

exekias commented May 27, 2018 •

edited

Loading

ph Aug 23, 2018 •

edited

Loading