[Security Solution] Remove filesystem detection rules in favor of a bundled Fleet package

[xcrzx]

Related to: elastic/detection-rules#2386

Summary

Since the support bundling Fleet packages with Kibana source has been added recently (see this PR), we can remove the filesystem distribution method of prebuilt detection rules.

• Add thesecurity_detection_engine package to the list of bundled packages:
  

  kibana/fleet_packages.json

  Lines 14 to 18 in 711ff3a

  	[
  	{
  	"name": "apm",
  	"version": "8.4.0"
  	},

• Remove the filesystem rules: /x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_rules
• Deprecate associated config options as they become noop:
  • xpack.securitySolution.prebuiltRulesFromSavedObjects
  • xpack.securitySolution.prebuiltRulesFromFileSystem
• Update the detection rules release process:
  • Remove the part where a Kibana PR with detection rules is created
  • Add bundled package version update instructions (see a PR that updates package versions for example)
• Ensure that tests run against the latest prebuilt rules package

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[terrancedejesus]

Reading over this, I have a few questions that will help me have a better understanding of what actions TRADE needs to take on our end.

1. Will the latest pre-built detection rules package need to be at the CDN prior to creating a PR to Kibana to update the fleet_packages.json file? Typically we update Kibana and then the integration package, so we I want to confirm we need to re-arrange the order of this in our process and documentation.
2. We do split-releases for 7.x and 8.x, therefore do we need to add two entries into this JSON file for each or can we use forceAlignStackVersion to force Kibana to look for the latest compatible version?
3. If we do an OOB update, do we then need to always update this JSON file with the latest version?
4. Are we able to update this now with the latest 8.x series package being v8.3.1?
5. Probably the most important question for TRADE regarding this is testing. At the moment, when we start the PR with the updated x-pack package for pre-built detection rules, testing is automatically done in backend CI jobs in buildkite I believe. Will testing still take place to ensure the package is pulled, rules are installed and the build is successful? Example PR from latest.

I believe with these questions answered I can then go ahead and start working on our end to address the updates for the process, documentation and CLI commands. Thanks in advance!

[xcrzx]

@terrancedejesus @Mikaayenson Thank you for taking the time to discuss that issue. Here's a summary of our sync:

• Will the latest pre-built detection rules package need to be at the CDN prior to creating a PR to Kibana to update the fleet_packages.json file? Typically we update Kibana and then the integration package, so we I want to confirm we need to re-arrange the order of this in our process and documentation.

Yes, the release process order will change a little bit. In short, it will consist of the following steps:

1. Build the integration package
2. Create an integrations repo PR
3. Create a package-storage PR under the snapshot branch
4. Create a Kibana PR that bumps the version in the fleet_packages.json
   • @xcrzx needs to find a way to run Kibana tests against the latest version of the detection rules package in epr-snapshot.elastic.co. Probably we could use a tag to override xpack.fleet.registryUrl: https://epr.elastic.co Kibana setting.
5. If Kibana CI is green, promote the rules package to production

• We do split-releases for 7.x and 8.x, therefore do we need to add two entries into this JSON file for each or can we use forceAlignStackVersion to force Kibana to look for the latest compatible version?

Unfortunately, bundling of Fleet packages is not supported in 7.x. So the release process for that branch will remain unchanged. The changes described in this ticket will affect only 8.x releases.

• If we do an OOB update, do we then need to always update this JSON file with the latest version?

No, we don't need to update the JSON for OOB updates. Updating the JSON file is required only before stack releases.

• Are we able to update this now with the latest 8.x series package being v8.3.1?

Yes, we can create a test branch to see how that approach works. However, it would be better to merge it only after we've removed the filesystem rules from Kibana to avoid bundling detection rules twice.

• Probably the most important question for TRADE regarding this is testing. At the moment, when we start the PR with the updated x-pack package for pre-built detection rules, testing is automatically done in backend CI jobs in buildkite I believe. Will testing still take place to ensure the package is pulled, rules are installed and the build is successful? Example PR from latest.

Yes, we will need to ensure that Kibana is tested with the latest rule versions before promoting the package as described above.

[xcrzx]

Note on testing prepackaged Fleet integrations locally (that could be helpful for development purposes):

1. Add the security_detection_engine package to the fleet_packages.json

   {
    "name": "security_detection_engine",
    "version": "8.2.1"
   }

2. Bundle Fleet packages by running node scripts/build. You could speed up the process by commenting out all tasks in src/dev/build/build_distributables.ts but Tasks.BundleFleetPackages.
3. Point Fleet to the folder with bundled packages by adding xpack.fleet.developer.bundledPackageLocation: build/kibana/x-pack/plugins/fleet/target/bundled_packages to kibana.dev.yaml.
4. Navigate to any Detection Engine page that triggers the Prebuilt Security Detection Rules package installation. Check that the latest available version of the package was installed (that could be done on the integrations page).

To imitate an air-gapped environment:

1. Uninstall the Prebuilt Security Detection Rules integration manually in Kibana
2. Point Fleet to an unexisting registry by adding xpack.fleet.registryUrl: https://localhost:8080 to kibana.dev.yaml.
3. Navigate to any Detection Engine page to trigger the Prebuilt Security Detection Rules package installation. Check that the prebundled version of the package was installed.

[terrancedejesus]

@banderror and @xcrzx below is our issue to track the work related to this for TRaDE.

• [FR] Update existing release process for removed file system rules detection-rules#2386