feat(wasm): support bundled filters

[flrgh]

Summary

An upcoming Kong release will ship with at least one bundled Wasm filter.

Previously there was no concept of "bundled" filters: all filters must be provided by the user in the wasm_filters_path directory.

This introduces a new wasm_filters kong.conf property. It is a comma-delimited array that functions quite similarly to the plugins property with the addition of a special user value:

# all filters disabled
wasm_filters = off

# all bundled filters enabled
wasm_filters = bundled

# all user filters (in `wasm_filters_path`) enabled
wasm_filters = user

# all available filters enabled
wasm_filters = bundled, user

# specific filters enabled
wasm_filters = some_bundled_filter, some_user_supplied_filter

In the case of a collision between bundled and user filters, the conf loader will prefer the user filter and log a warning.

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

KAG-4211

[flrgh]

The code for this is AFAICT feature-complete and should be considered ready-for-review, but this should not be merged until after the distribution work for bundled filters is done so that I can update the tests accordingly.

(Also I have no idea if my hack for enabling bundled filters in local dev is going to work yet.)

[flrgh]

Currently rebased onto #12853. Once that is merged this PR will get a bit smaller.

[kikito]

This pr has some debug messages on it, I am moving it to Draft until that is addressed. @flrgh please move it back to "Ready to review" when ready.

[brentos]

I'm a bit late for adding my $0.02 on this, but I think some sort of namespacing of filters could nicely solve the collision issue. For example:

kong.datakit
kong.response_transformer
foocompany.datakit
foocompany.response_transfomer

Then we could enable with

wasm_filters = kong.datakit, foocompany.datakit

In addition, setting up some standard file locations for these such that we don't have to configure a base path, and these filters could be installed at:

<default_filter_dir>/kong/datakit
<default_filter_dir>/foocompany/datakit

[hishamhm]

@brentos Users can name filters as they like, so they can definitely use namespacing to avoid collisions. 👍 This is more about allowing for "intentional" collisions, that is, for when a user wants to override a bundled filter that is shipped with Kong (for example, with a more recent version), without having to disable bundled filters altogether.

[flrgh]

re: all of these timeout changes

This is to account for the increased NGINX startup time incurred from loading the new datakit module. Locally I'm seeing about ~3.6s pretty consistently, but timing in CI (where we are often saturating the CPU) is slower and very inconsistent, so a large timeout is selected to keep the test from being flaky.

There is some work in progress to mitigate the slow startup time: Kong/ngx_wasm_module#536

[flrgh]

update: because module loading is an init_worker thing, the privileged agent worker is also delayed on startup, so this affects how long it takes for the DP to establish a connection to the CP. I believe that is why this test suite is struggling so much more than others. Changes:

1. added some more logfile assertions to watch for a module load event (technically this only guarantees that 1/N processes has finished init_worker, but it's still somewhat helpful).
2. adjusted the worker procs to 2
3. disabled bundled filters in this test suite because datakit is the biggest source of slow init_worker

[flrgh]

investigating something, please do not merge

edit: okay I think the tests should be a lot more stable now, good to merge when 🟢

[team-gateway-bot]

Successfully created cherry-pick PR for master:

• https://github.com/kong/kong-ee/pull/8918