Beats Management plugin: migrate server-side code #70930

pgayvallet · 2020-07-07T09:39:27Z

Summary

Checklist

Documentation was added for features that require explanation or tutorials.
Unit or functional tests were updated or added to match the most common scenarios

…anagement-server-2

pgayvallet

The FTR test suite (x-pack/test/api_integration/apis/beats) is passing, unfortunately it doesn't cover all the endpoints.

pgayvallet · 2020-07-07T15:23:41Z

x-pack/test/api_integration/apis/beats/enroll_beat.js

-        success: false,
-        error: { code: 400, message: 'Invalid enrollment token' },
+        statusCode: 400,
+        error: 'Bad Request',
+        message: 'Invalid enrollment token',


The legacy plugin was using custom body payload for error responses, which is not possible with KP. The client-side code actually don't care about this format changes, as the body of an error response is never accessed.

pgayvallet · 2020-07-07T15:25:36Z

x-pack/plugins/beats_management/server/routes/wrap_route_with_security.ts

+export function wrapRouteWithSecurity<P, Q, B>(
+  {
+    requiredLicense = [],


Extracted/adapted from x-pack/legacy/plugins/beats_management/server/lib/framework.ts#wrapRouteWithSecurity

pgayvallet · 2020-07-07T15:28:49Z

x-pack/plugins/beats_management/server/lib/adapters/framework/kibana_framework_adapter.ts

+export class KibanaBackendFrameworkAdapter implements BackendFrameworkAdapter {
+  public readonly internalUser = internalUser;
+  public info: null | FrameworkInfo = null;


Too many changes to keep file track, but this was x-pack/legacy/plugins/beats_management/server/lib/adapters/framework/kibana_framework_adapter.ts

pgayvallet · 2020-07-07T15:31:48Z

x-pack/plugins/beats_management/server/routes/beats/configuration.ts

+/*
+import Joi from 'joi';
+import { FrameworkRequest } from '../../lib/adapters/framework/adapter_types';
+import { CMServerLibs } from '../../lib/types';


I kept the legacy route registration commented in the route files to ease the diff when reviewing. I will remove them all before merging

pgayvallet · 2020-07-07T15:34:23Z

x-pack/plugins/beats_management/server/routes/beats/enroll.ts

+        // TODO: fixme eventually, need to access `info.remoteAddress` from KibanaRequest.
+        const legacyRequest = ensureRawRequest(request);


I need to access legacyRequest.info.remoteAddress, which is not exposed on KibanaRequest. I discussed with @restrry of the possibility to add it to core API. However as this plugin is the only one having this need and knowing that it will be removed for 8.0, I'm just using the private ensureRawRequest utility here.

I can implement the 'correct' way if that's preferable.

Josh added one in #71019

Ended up implementing it differently, so that PR no longer adds this API. I'm fine with this implementation for now and if other plugins (that aren't about to get deleted) also need this, we can re-evaluate adding it then.

pgayvallet · 2020-07-07T15:38:28Z

x-pack/plugins/beats_management/server/routes/beats/events.ts

+      headers: Joi.object({
+        'kbn-beats-access-token': Joi.string().required(),
+      }).options({ allowUnknown: true }),


This misleading kbn-beats-access-token token validation is actually doing nothing, as the object is not .required(). Also, the header is not used in the route handler, and neither the FTR test suite nor the client-side API are sending the header, which make me think this was just bad copypasta.

All that to say: the fact that this validation is not present on the KP handler is not a mistake.

pgayvallet · 2020-07-07T15:41:56Z

x-pack/plugins/beats_management/server/routes/configurations/upsert.ts

+  // TODO: write to Kibana audit log file
+  router.put(


I just kept the TODO comments unchanged.

pgayvallet · 2020-07-07T15:43:25Z

x-pack/plugins/beats_management/server/routes/tokens/create.ts

+        body: schema.nullable(
+          schema.object({
+            num_tokens: schema.number({ defaultValue: DEFAULT_NUM_TOKENS, min: 1 }),
+          })
+        ),


TIL an empty payload is null, not undefined for hapi.

elasticmachine · 2020-07-07T15:45:28Z

Pinging @elastic/kibana-platform (Team:Platform)

pgayvallet · 2020-07-07T15:49:13Z

@ph @justinkambic would you mind sparing some time to perform a quick user testing of the plugin's functionalities?

ph · 2020-07-08T12:53:14Z

This look targetting for 7.10 so I will give it a shot after FF? is that okay with you?/

pgayvallet · 2020-07-08T18:45:48Z

@ph yea, it's planned for 7.10, so after FF is perfectly fine. Thanks a lot.

…anagement-server-2

pgayvallet · 2020-07-23T09:05:12Z

@ph @justinkambic do you think you'll have the capacity to take a look in the next few weeks?

…anagement-server-2

justinkambic

Review summary

I primarily reviewed the front-end changes, which are more-or-less an update to the imports to account for file migration. Nothing stood out to me on that account. I did look over server code as well, everything seems to be working as I'd expect, and the code migration largely made sense.

I attempted to utilize each of this plugin's routes while doing my functional review as well.

Functional summary

My functional review consisted of:

Enroll a beat
Create a tag
- Create config block for Filebeat to tail a log
- Create an output config block for ES access
Attach a tag to a beat
Create and attach multiple tags to beats
Remove tag from beat
Run Filebeat and ingest data, ensure my logs are indexed
Delete a tag
Unenroll the beat

Bugs

I did notice some weirdness when it came to deleting tags. It seems that when a tag is selected and deleted from the list, all the other tags are dropped from the list except the one deleted. Re-fetching of the available tags does show that the deleted one is gone and the rest remain, but it's definitely an unsettling UX. I'm not sure if this is how the table already behaves in master.

A similar behavior when unenrolling beats. I unenrolled my Filebeat and the UI didn't reflect the change appropriately. After refresh the list displayed no items, like I would expect. Again, this bug might already be in master and not a side effect of this change.

docs/developer/architecture/code-exploration.asciidoc

…anagement-server-2

pgayvallet · 2020-08-17T13:34:24Z

@justinkambic thanks a lot of the review. Regarding the refresh bugs, I just checked master, and can confirm they are already present on our master branch. Looking at the client-side migration PR from @joshdover , it looks like it was even present before we migrated the client-side code, as the action handlers for the various table did not really change. So I guess this is 'alright'.

I can open a follow-up issue to track this if we feel like it's necessary.

joshdover

Reviewed the key integration points & all route code. Wasn't able to find inconsistencies with the old routes that could create issues. Of course there's plenty of opportunity for improvement in this plugin, but not worth the effort since it will be sunset soon. Integration with Core APIs LGTM.

Do you mind go ahead and renaming this plugin's id to beatsManagement so we can do away with that deprecation log?

joshdover · 2020-08-17T18:51:27Z

x-pack/plugins/beats_management/server/routes/wrap_route_with_security.ts

+      if (user.kind !== 'authenticated') {
+        return response.forbidden({
+          body: {
+            message: `Request must be authenticated`,
+          },
+        });
+      }


I'm not even sure this is necessary since this is the default behavior? Maybe not worth changing the original implementation though.

Yea, honestly I tried to reduce the changes as much as possible, so the 'security' handler wrapper may have some redundancy with core's defaults, but I feel like it's not worth doing any additional changes.

…anagement-server-2

justinkambic · 2020-08-17T19:19:24Z

I can open a follow-up issue to track this if we feel like it's necessary.

I think this is a question more for the Ingest team if they want someone to fix it. Given that we know these issues exist, perhaps it would be a good idea to track them in an issue and make Ingest aware of them.

kibanamachine · 2020-08-17T21:05:12Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: c4c984c

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
beats_management	-	-307	307
beatsManagement	307	+307	-
total		-0

async chunks size

id	value	diff	baseline
beats_management	-	-409.3KB	409.3KB
beatsManagement	409.3KB	⚠️ +409.3KB	-
total		-4.0B

page load bundle size

id	value	diff	baseline
beats_management	-	-581.6KB	581.6KB
beatsManagement	581.6KB	⚠️ +581.6KB	-
total		-8.0B

History

💔 Build #68552 failed f444fac
💔 Build #68425 failed 9a71840
💔 Build #68412 failed 8221a63
💔 Build #67365 failed fbfdf3a
💚 Build #63829 succeeded 543df15

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* move all the legacy codebase to KP plugin * fix imports * add empty handler context & rename routes folder * start migrating routes * migrate rest of the routes * migrate adapters * remove beats from legacy xpack plugin * use nullable + replace response.custom * fix wrapRouteWithSecurity, remove incorrect header validation * remove comment * updating generated plugin list * fix typos in doc * adapt readme too. * remove old commented routes from route files * remove eslint disabling * use camel case for plugin id * update generated doc Co-authored-by: Josh Dover <[email protected]> # Conflicts: # x-pack/legacy/plugins/beats_management/server/rest_api/beats/configuration.ts # x-pack/legacy/plugins/beats_management/server/rest_api/configurations/upsert.ts

pgayvallet added 6 commits July 6, 2020 15:30

move all the legacy codebase to KP plugin

c51112b

fix imports

7ea9273

add empty handler context & rename routes folder

c7dc6a7

start migrating routes

dfb3dc6

migrate rest of the routes

8222306

migrate adapters

82f59b8

pgayvallet added chore Feature:Legacy Removal Issues related to removing legacy Kibana Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v7.10.0 v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels Jul 7, 2020

pgayvallet added 5 commits July 7, 2020 11:56

remove beats from legacy xpack plugin

283cb08

use nullable + replace response.custom

e1b333f

Merge remote-tracking branch 'upstream/master' into kbn-67980-beats-m…

5c1c6c4

…anagement-server-2

fix wrapRouteWithSecurity, remove incorrect header validation

fd158be

remove comment

a7f7eeb

pgayvallet commented Jul 7, 2020

View reviewed changes

pgayvallet marked this pull request as ready for review July 7, 2020 15:45

pgayvallet requested review from a team, justinkambic and ph July 7, 2020 15:49

Merge remote-tracking branch 'upstream/master' into kbn-67980-beats-m…

2faba13

…anagement-server-2

pgayvallet added 3 commits July 20, 2020 09:03

Merge remote-tracking branch 'upstream/master' into kbn-67980-beats-m…

6842a6b

…anagement-server-2

updating generated plugin list

9025c5f

Merge remote-tracking branch 'upstream/master' into kbn-67980-beats-m…

543df15

…anagement-server-2

pgayvallet mentioned this pull request Jul 23, 2020

Remove legacy plugin discovery #71927

Closed

14 tasks

Merge remote-tracking branch 'upstream/master' into kbn-67980-beats-m…

fbfdf3a

…anagement-server-2

justinkambic reviewed Aug 11, 2020

View reviewed changes

docs/developer/architecture/code-exploration.asciidoc Outdated Show resolved Hide resolved

pgayvallet added 4 commits August 17, 2020 14:15

Merge remote-tracking branch 'upstream/master' into kbn-67980-beats-m…

c0fe0b8

…anagement-server-2

fix typos in doc

8221a63

adapt readme too.

f9ed333

remove old commented routes from route files

9a71840

remove eslint disabling

bf0df27

joshdover approved these changes Aug 17, 2020

View reviewed changes

pgayvallet added 3 commits August 17, 2020 21:06

Merge remote-tracking branch 'upstream/master' into kbn-67980-beats-m…

def77dd

…anagement-server-2

use camel case for plugin id

f444fac

update generated doc

c4c984c

pgayvallet merged commit 890fc31 into elastic:master Aug 18, 2020

pgayvallet mentioned this pull request Aug 18, 2020

[7.x] Beats Management plugin: migrate server-side code (#70930) #75269

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Beats Management plugin: migrate server-side code #70930

Beats Management plugin: migrate server-side code #70930

pgayvallet commented Jul 7, 2020 •

edited

Loading

pgayvallet left a comment

pgayvallet Jul 7, 2020

pgayvallet Jul 7, 2020

pgayvallet Jul 7, 2020

pgayvallet Jul 7, 2020

pgayvallet Jul 7, 2020

mshustov Jul 8, 2020

joshdover Aug 17, 2020

pgayvallet Jul 7, 2020

pgayvallet Jul 7, 2020

pgayvallet Jul 7, 2020

elasticmachine commented Jul 7, 2020

pgayvallet commented Jul 7, 2020

ph commented Jul 8, 2020 •

edited

Loading

pgayvallet commented Jul 8, 2020

pgayvallet commented Jul 23, 2020

justinkambic left a comment •

edited

Loading

pgayvallet commented Aug 17, 2020

joshdover left a comment

joshdover Aug 17, 2020

pgayvallet Aug 17, 2020

justinkambic commented Aug 17, 2020

kibanamachine commented Aug 17, 2020

		// TODO: fixme eventually, need to access `info.remoteAddress` from KibanaRequest.
		const legacyRequest = ensureRawRequest(request);

Beats Management plugin: migrate server-side code #70930

Beats Management plugin: migrate server-side code #70930

Conversation

pgayvallet commented Jul 7, 2020 • edited Loading

Summary

Checklist

pgayvallet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jul 7, 2020

pgayvallet commented Jul 7, 2020

ph commented Jul 8, 2020 • edited Loading

pgayvallet commented Jul 8, 2020

pgayvallet commented Jul 23, 2020

justinkambic left a comment • edited Loading

Choose a reason for hiding this comment

Review summary

Functional summary

Bugs

pgayvallet commented Aug 17, 2020

joshdover left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinkambic commented Aug 17, 2020

kibanamachine commented Aug 17, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

async chunks size

page load bundle size

History

pgayvallet commented Jul 7, 2020 •

edited

Loading

ph commented Jul 8, 2020 •

edited

Loading

justinkambic left a comment •

edited

Loading