Clean up python-related system tests

[fearful-symmetry]

What does this PR do?

This PR is the result of me fighting with the python unit tests every time I refactor part of the system module. These tests are pretty old, and in most cases haven't been touched in 6 years, so my goal was to refactor them just enough to both A) not make my linter light up the entire file B) make it easier to adapt the tests as we continue to refactor the system.

However, I was a little unsure of just how far I should take this cleanup. The tests are really old and a lot of the code is kind of strange, but if someone is going to come along in a few months and rewrite the entire tests in go/mage (as has been suggested a few times), it's not worth it to do a more aggressive cleanup. However, if this code is going to stick around for a while, it might be worth it to refactor and modernize the rest of the python test code, of which there is a lot. Right now this PR is basically geared towards making things easier for the system module, and I'd like the opinion of others as to if it's worth it to expand the scope of this PR to the rest of the python test infrastructure.

Why is it important?

The python tests have proven to be something of a burden when refactoring the system code, as their fast-and-loose approach to running beats and fetching data doesn't quite work with the refactored code's stricter reporting.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-02-16T04:35:02.745+0000

• Duration: 132 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	37803
Skipped	3168
Total	40971

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label. Could you fix it @fearful-symmetry? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[cmacknz]

This looks reasonable to me, although there are a lot of changes so maybe I missed something.

I wouldn't carry this through the rest of the system tests unless there is some benefit to you to do so. I don't know that we have a plan for the system tests yet, beyond knowing that nobody likes them. The easiest option is to only refactor the tests you touch regularly.

You could check in the lint+format configuration you used for this to run on the files you touched as a way of preventing them from regressing if you wanted to.