Flaky prometheus remote write system test

[jsoriano]

Flaky Test

• Test Name: metricbeat/module/prometheus/test_prometheus.py::TestRemoteWrite::test_remote_write
• Link:

  beats/metricbeat/module/prometheus/test_prometheus.py

  Line 88 in 9807eb1

  self.wait_until(lambda: self.output_lines() > 0, 60)

• Branch: Seen in master and several branches.
• Notes: Seems to timeout waiting for output, happening since a couple of days ago (Sept 1st).

Stack Trace

From https://beats-ci.elastic.co/job/Beats/job/beats/job/PR-20926/3/testReport/junit/metricbeat.module.prometheus.test_prometheus/TestRemoteWrite/Build_and_Test___Metricbeat_OSS_Python_Integration_tests___test_remote_write/

self = <test_prometheus.TestRemoteWrite testMethod=test_remote_write>

    @unittest.skipUnless(metricbeat.INTEGRATION_TESTS, "integration test")
    def test_remote_write(self):
        """
        prometheus remote_write test
        """
        self.render_config_template(modules=[{
            "name": "prometheus",
            "metricsets": ["remote_write"],
            "period": "5s",
            "host": "localhost",
            "port": "9201"
        }])
        proc = self.start_beat()
    
        self.wait_until(lambda: self.log_contains("Starting HTTP"))
    
>       self.wait_until(lambda: self.output_lines() > 0, 60)

module/prometheus/test_prometheus.py:88: 

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)