[Fleet] Fleet reinstalls non-managed packages on Kibana boot

[joshdover]

In 8.0+ we moved Fleet setup to happen Kibana boot and wanted to ensure that managed packages are rolled back if needed.

We use the force: true flag to accomplish this here:

kibana/x-pack/plugins/fleet/server/services/preconfiguration.ts

Lines 173 to 181 in 3b516b6

	// Preinstall packages specified in Kibana config
	const preconfiguredPackages = await bulkInstallPackages({
	savedObjectsClient: soClient,
	esClient,
	packagesToInstall: packages.map((pkg) =>
	pkg.version === PRECONFIGURATION_LATEST_KEYWORD ? pkg.name : pkg
	),
	force: true, // Always force outdated packages to be installed if a later version isn't installed
	});

One issue I noticed after testing 7.16 -> 8.0 upgrades is that packages that were installed prior to the upgrade are getting re-installed (though not upgraded), even though they are not managed packages or preconfigured packages.

1. Start ES and Kibana 7.16.0
2. Run Fleet setup
3. Install the nginx package (I used 1.1.0 so I could see if it was also being upgraded to the latest 1.2.1 version)
4. Shutdown Kibana and ES
5. Start ES 8.1.0 (make sure to use the same path.data as the 7.16.0 instance)
6. Start Kibana from main
7. Watch debug logs and observe this log message:

[2021-12-03T16:53:27.973+01:00][DEBUG][plugins.fleet] kicking off install of nginx-1.1.0 from registry

Interestingly, this only appears to happen the first time and does not happen on subsequent reboots. This may make this less of a problem, however it could make Fleet setup take a very long time for long-time users of 7.16.0 who upgrade to 8.x

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

I had a check on this, and the logic that runs on upgrade is in Fleet setup code.
It looks like some assets are being created as part of the upgrade, and that triggers install on existing packages.

According to the code comment, this is done to "update the index template and the datastream write indices".
I'm not really familiar with this part, so I don't know if reinstall is actually necessary on upgrade or not.

When I comment out installPackage from this logic, nginx integration looks fine on UI, but it might be that references to assets have to be updated?

kibana/x-pack/plugins/fleet/server/services/setup.ts

Lines 144 to 163 in e298474

	if (globalAssetsRes.some((asset) => asset.isCreated)) {
	// Update existing index template
	const packages = await getInstallations(soClient);
	await Promise.all(
	packages.saved_objects.map(async ({ attributes: installation }) => {
	if (installation.install_source !== 'registry') {
	logger.error(
	`Package needs to be manually reinstalled ${installation.name} after installing Fleet global assets`
	);
	return;
	}
	await installPackage({
	installSource: installation.install_source,
	savedObjectsClient: soClient,
	pkgkey: pkgToPkgKey({ name: installation.name, version: installation.version }),
	esClient,
	spaceId: DEFAULT_SPACE_ID,
	// Force install the package will update the index template and the datastream write indices
	force: true,

[joshdover]

Ah so I think this is in place to ensure that we are using the latest versions of the shared global ingest pipeline that Fleet installs. I think it would make sense to revisit this as part of #121099. @juliaElastic does that make sense to you? If so, I think we can close this and make a note to revisit this as part of that issue.

[juliaElastic]

Ah so I think this is in place to ensure that we are using the latest versions of the shared global ingest pipeline that Fleet installs. I think it would make sense to revisit this as part of #121099. @juliaElastic does that make sense to you? If so, I think we can close this and make a note to revisit this as part of that issue.

So are you saying that as part of the other issue, Fleet will reinstall packages on stack upgrade? So that the setup code no longer has to check and reinstall packages? If so, I agree that we can close and revisit with the other issue.

[joshdover]

So are you saying that as part of the other issue, Fleet will reinstall packages on stack upgrade? So that the setup code no longer has to check and reinstall packages? If so, I agree that we can close and revisit with the other issue.

Yes, that's the idea. Great, I'll close this and consider it as part of the design of #121099