Remove queue.Consumer

[faec]

What does this PR do?

Removes the queue.Consumer interface, and replaces its Get call with one directly on the queue interface itself.

Why is it important?

The current Beats queue interface defines a notion of a "consumer" targeting a queue, and which is the exclusive object through which the queue can be read. There can be an arbitrary number of consumers operating in parallel, which can be closed independently of the queue and of each other.

Supporting the queue consumer interface and its current implementations is an obstacle to many planned changes, particularly elastic/elastic-agent-shipper#27 where we are trying to propagate consistent ordered event IDs through all queue types.

While we could use more complicated workarounds to support this regardless, there are good reasons to instead remove the concept of a queue consumer entirely:

• Unlike queue.Producer, the consumer has no special configuration of its own that affects its behavior, which means it's questionable whether there's any advantage to creating multiple consumers (as opposed to calling directly into the queue from multiple goroutines).
• Outside of the queue tests there is only one line in the entire Beats codebase that creates a queue consumer, and it never creates more than one.
• Removing the extra layer of indirection prepares the way for simpler unified handling of events and event batches between the traditional Beats pipeline and the in-progress Shipper prototype.

This PR should functionally be a no-op; the only functional change is that certain places that used to pass a queue consumer now pass the queue itself instead.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

LGTM, code seems clearer after this as well.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-05-12T14:39:10.802+0000

• Duration: 71 min 11 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22556
Skipped	1938
Total	24494

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[fearful-symmetry]

Yay, looks good to me, with what queue knowledge I have.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b queue-consumer upstream/queue-consumer
git merge upstream/main
git push upstream queue-consumer