[7.x](backport #24621) Add cluster level add_kubernetes_metadata support for centralized enrichment

[mergify]

This is an automatic backport of pull request #24621 done by Mergify.

---

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

• @Mergifyio refresh will re-evaluate the rules
• @Mergifyio rebase will rebase this PR on its base branch
• @Mergifyio update will merge the base branch into this PR
• @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

• look at your merge queues
• generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

[botelastic]

This pull request doesn't have a Team:<team> label.

[elasticmachine]

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-09-28T07:58:50.557+0000

• Duration: 71 min 28 sec

• Commit: 73ca847

Test stats 🧪

Test	Results
Failed	2
Passed	20612
Skipped	1515
Total	22129

Test errors

Expand to view the tests failures

Build&Test / x-pack/metricbeat-pythonIntegTest / test_qmgr – x-pack.metricbeat.module.ibmmq.test_ibmmq.Test

Expand to view the error details

 failed on setup with "compose.service.BuildError: (<Service: ibmmq>, 'manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown')" 

Expand to view the stacktrace

 self = <Service: ibmmq>, no_cache = False, pull = False, force_rm = False
memory = None, build_args_override = None, gzip = False, rm = True
silent = False, cli = False, progress = None

    def build(self, no_cache=False, pull=False, force_rm=False, memory=None, build_args_override=None,
              gzip=False, rm=True, silent=False, cli=False, progress=None):
        output_stream = open(os.devnull, 'w')
        if not silent:
            output_stream = sys.stdout
            log.info('Building %s' % self.name)
    
        build_opts = self.options.get('build', {})
    
        build_args = build_opts.get('args', {}).copy()
        if build_args_override:
            build_args.update(build_args_override)
    
        for k, v in self._parse_proxy_config().items():
            build_args.setdefault(k, v)
    
        path = rewrite_build_path(build_opts.get('context'))
        if self.platform and version_lt(self.client.api_version, '1.35'):
            raise OperationFailedError(
                'Impossible to perform platform-targeted builds for API version < 1.35'
            )
    
        builder = self.client if not cli else _CLIBuilder(progress)
        build_output = builder.build(
            path=path,
            tag=self.image_name,
            rm=rm,
            forcerm=force_rm,
            pull=pull,
            nocache=no_cache,
            dockerfile=build_opts.get('dockerfile', None),
            cache_from=self.get_cache_from(build_opts),
            labels=build_opts.get('labels', None),
            buildargs=build_args,
            network_mode=build_opts.get('network', None),
            target=build_opts.get('target', None),
            shmsize=parse_bytes(build_opts.get('shm_size')) if build_opts.get('shm_size') else None,
            extra_hosts=build_opts.get('extra_hosts', None),
            container_limits={
                'memory': parse_bytes(memory) if memory else None
            },
            gzip=gzip,
            isolation=build_opts.get('isolation', self.options.get('isolation', None)),
            platform=self.platform,
        )
    
        try:
>           all_events = list(stream_output(build_output, output_stream))

../../build/ve/docker/lib/python3.7/site-packages/compose/service.py:1110: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

output = <generator object APIClient._stream_helper at 0x7f64c3bc4c78>
stream = <_io.TextIOWrapper name="<_io.FileIO name=6 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>

    def stream_output(output, stream):
        is_terminal = hasattr(stream, 'isatty') and stream.isatty()
        stream = utils.get_output_stream(stream)
        lines = {}
        diff = 0
    
        for event in utils.json_stream(output):
            yield event
            is_progress_event = 'progress' in event or 'progressDetail' in event
    
            if not is_progress_event:
>               print_output_event(event, stream, is_terminal)

../../build/ve/docker/lib/python3.7/site-packages/compose/progress_stream.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

event = {'error': 'manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown', 'errorDetail': {'message': 'manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown'}}
stream = <_io.TextIOWrapper name="<_io.FileIO name=6 mode='rb+' closefd=True>" mode='r+' encoding='utf-8'>
is_terminal = False

    def print_output_event(event, stream, is_terminal):
        if 'errorDetail' in event:
>           raise StreamOutputError(event['errorDetail']['message'])
E           compose.progress_stream.StreamOutputError: manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown

../../build/ve/docker/lib/python3.7/site-packages/compose/progress_stream.py:62: StreamOutputError

During handling of the above exception, another exception occurred:

self = <class 'test_ibmmq.Test'>

    @classmethod
    def setUpClass(self):
        self.beat_name = "metricbeat"
        self.beat_path = os.path.abspath(
            os.path.join(os.path.dirname(__file__), "../../"))
    
        self.template_paths = [
            os.path.abspath(os.path.join(self.beat_path, "../../metricbeat")),
            os.path.abspath(os.path.join(self.beat_path, "../../libbeat")),
        ]
    
>       super(XPackTest, self).setUpClass()

tests/system/xpack_metricbeat.py:19: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../metricbeat/tests/system/metricbeat.py:28: in setUpClass
    super(BaseTest, self).setUpClass()
../../libbeat/tests/system/beat/beat.py:148: in setUpClass
    self.compose_up_with_retries()
../../libbeat/tests/system/beat/beat.py:163: in compose_up_with_retries
    raise e
../../libbeat/tests/system/beat/beat.py:159: in compose_up_with_retries
    self.compose_up()
../../libbeat/tests/system/beat/compose.py:69: in compose_up
    timeout=30)
../../build/ve/docker/lib/python3.7/site-packages/compose/project.py:548: in up
    svc.ensure_image_exists(do_build=do_build, silent=silent, cli=cli)
../../build/ve/docker/lib/python3.7/site-packages/compose/service.py:367: in ensure_image_exists
    self.build(cli=cli)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Service: ibmmq>, no_cache = False, pull = False, force_rm = False
memory = None, build_args_override = None, gzip = False, rm = True
silent = False, cli = False, progress = None

    def build(self, no_cache=False, pull=False, force_rm=False, memory=None, build_args_override=None,
              gzip=False, rm=True, silent=False, cli=False, progress=None):
        output_stream = open(os.devnull, 'w')
        if not silent:
            output_stream = sys.stdout
            log.info('Building %s' % self.name)
    
        build_opts = self.options.get('build', {})
    
        build_args = build_opts.get('args', {}).copy()
        if build_args_override:
            build_args.update(build_args_override)
    
        for k, v in self._parse_proxy_config().items():
            build_args.setdefault(k, v)
    
        path = rewrite_build_path(build_opts.get('context'))
        if self.platform and version_lt(self.client.api_version, '1.35'):
            raise OperationFailedError(
                'Impossible to perform platform-targeted builds for API version < 1.35'
            )
    
        builder = self.client if not cli else _CLIBuilder(progress)
        build_output = builder.build(
            path=path,
            tag=self.image_name,
            rm=rm,
            forcerm=force_rm,
            pull=pull,
            nocache=no_cache,
            dockerfile=build_opts.get('dockerfile', None),
            cache_from=self.get_cache_from(build_opts),
            labels=build_opts.get('labels', None),
            buildargs=build_args,
            network_mode=build_opts.get('network', None),
            target=build_opts.get('target', None),
            shmsize=parse_bytes(build_opts.get('shm_size')) if build_opts.get('shm_size') else None,
            extra_hosts=build_opts.get('extra_hosts', None),
            container_limits={
                'memory': parse_bytes(memory) if memory else None
            },
            gzip=gzip,
            isolation=build_opts.get('isolation', self.options.get('isolation', None)),
            platform=self.platform,
        )
    
        try:
            all_events = list(stream_output(build_output, output_stream))
        except StreamOutputError as e:
>           raise BuildError(self, six.text_type(e))
E           compose.service.BuildError: (<Service: ibmmq>, 'manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown')

../../build/ve/docker/lib/python3.7/site-packages/compose/service.py:1112: BuildError 

Build&Test / x-pack/metricbeat-goIntegTest / TestFetch – github.com/elastic/beats/v7/x-pack/metricbeat/module/ibmmq/qmgr

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestFetch
Found orphan containers (metricbeat_7_16_0_73ca8470f4-snapshot_coredns_1, metricbeat_7_16_0_73ca8470f4-snapshot_cockroachdb_1, metricbeat_7_16_0_73ca8470f4-snapshot_appsearch_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Building ibmmq
Step 1/7 : ARG IBMMQ_VERSION
Step 2/7 : FROM ibmcom/mq:${IBMMQ_VERSION}
Service 'ibmmq' failed to build: manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown
    qmgr_integration_test.go:31: failed to start service 'ibmmq: exit status 1
Found orphan containers (metricbeat_7_16_0_73ca8470f4-snapshot_coredns_1, metricbeat_7_16_0_73ca8470f4-snapshot_cockroachdb_1, metricbeat_7_16_0_73ca8470f4-snapshot_appsearch_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Building ibmmq
Step 1/7 : ARG IBMMQ_VERSION
Step 2/7 : FROM ibmcom/mq:${IBMMQ_VERSION}
Service 'ibmmq' failed to build: manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown
    qmgr_integration_test.go:31: failed to start service 'ibmmq: exit status 1
Found orphan containers (metricbeat_7_16_0_73ca8470f4-snapshot_coredns_1, metricbeat_7_16_0_73ca8470f4-snapshot_cockroachdb_1, metricbeat_7_16_0_73ca8470f4-snapshot_appsearch_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Building ibmmq
Step 1/7 : ARG IBMMQ_VERSION
Step 2/7 : FROM ibmcom/mq:${IBMMQ_VERSION}
Service 'ibmmq' failed to build: manifest for ibmcom/mq:9.1.4.0-r1-amd64 not found: manifest unknown: manifest unknown
    qmgr_integration_test.go:31: failed to start service 'ibmmq: exit status 1
    qmgr_integration_test.go:31: getting host for ibmmq: no container running for service
--- FAIL: TestFetch (5.84s)
 

Steps errors

Expand to view the steps failures

filebeat-goIntegTest - mage goIntegTest

• Took 5 min 43 sec . View more details here
• Description: mage goIntegTest

x-pack/metricbeat-goIntegTest - mage goIntegTest

• Took 13 min 24 sec . View more details here
• Description: mage goIntegTest

x-pack/metricbeat-goIntegTest - mage goIntegTest

• Took 10 min 47 sec . View more details here
• Description: mage goIntegTest

x-pack/metricbeat-goIntegTest - mage goIntegTest

• Took 8 min 20 sec . View more details here
• Description: mage goIntegTest

List files to upload

• Took 0 min 0 sec . View more details here
• Description: ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

• Took 12 min 41 sec . View more details here
• Description: mage pythonIntegTest

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

• Took 6 min 19 sec . View more details here
• Description: mage pythonIntegTest

x-pack/metricbeat-pythonIntegTest - mage pythonIntegTest

• Took 6 min 50 sec . View more details here
• Description: mage pythonIntegTest

List files to upload

• Took 0 min 0 sec . View more details here
• Description: ls -l src/github.com/elastic/beats/build/system-tests-*.tar.gz

Error signal

• Took 0 min 0 sec . View more details here
• Description: Error 'hudson.AbortException: script returned exit code 1'

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

• Name: Build&Test / x-pack/metricbeat-pythonIntegTest / test_qmgr – x-pack.metricbeat.module.ibmmq.test_ibmmq.Test
• Name: Build&Test / x-pack/metricbeat-goIntegTest / TestFetch – github.com/elastic/beats/v7/x-pack/metricbeat/module/ibmmq/qmgr

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.