Autodiscover based on services doesn't stop monitors when service is deleted

[jsoriano]

In 7.7 and 7.8, when a service is deleted, no autodiscover event seems to happen, so the monitor keeps running.

Since 7.9, the autodiscover event happens, and the monitor seems to be stopped, but something keeps running:

2020-08-11T12:06:57.646Z	DEBUG	[scheduler]	scheduler/scheduler.go:195	Job 'auto-http-0XADF69AB61E75FE7F' returned at 2020-08-11 12:06:57.646943114 +0000 UTC m=+15.631587534

They seem to be two different issues:

• Autodiscover event not produced when a service is deleted, seems to be fixed in 7.x branch, but not in 7.7 and 7.8. This probably affects all Beats. @ChrsMark do you know what could be the fix? Should we have backported it to 7.8?
  • It was fixed by Add better handling for Kubernetes Update and Delete watcher events #18882.
• Scheduler keeps running after stopping the monitor. @andrewvc is this expected?

For confirmed bugs, please report:

• Version: At least since 7.7.
• Steps to Reproduce:
  • Enable kubernetes autodiscover based on services on Heartbeat.
  • Create a service, wait for a monitor to be started.
  • Delete the service, some things keep running.

A configuration like this one can be used to reproduce this:

  heartbeat.autodiscover:
    providers:
      - type: kubernetes
        resource: service
        scope: cluster
        cleanup_timeout: 0s
        templates:
          - condition:
              equals:
                kubernetes.service.name: test
            config:
              - type: http
                urls: ["https://${data.host}:${data.port}"]
                schedule: "@every 30s"
                check.response.status: 200
                timeout: 10s
                wait: 1s

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[ChrsMark]

🤔 I cannot recall any change that could fix sth like this in 7.x.

[ChrsMark]

@jsoriano it should be #18882 and more specifically https://github.com/elastic/beats/pull/18882/files#diff-becb7cd8554f76895e450cdb5557331aR208.

This one was back-ported to 7.8.2 but it seems that 7.8.2 never happened.

[jsoriano]

@ChrsMark thanks for the investigation! I guess we are covered then, I will update the description.

We would still need to confirm if the running scheduler is expected.