[WIP][From @nzlosh] Interoperability updates for py3.8, py3.9 and py3.10

[FileMagic]

Based on the updates done in #6157, I merged in the changes that @cognifloyd did on master into the changes that @nzlosh was working on. From here I am going to update things until unit tests work properly on python 3.8-3.10.

[CLAassistant]

All committers have signed the CLA.

[FileMagic]

pls ignore this for a bit. I am more setting this up for ci-cd for after I figure out this race condition, and will be working with @nzlosh to rebase any changes into his pr to be shipped afterwards.

[FileMagic]

From inital looks, it seems that tooz is not configured properly to use a datastore during pytest which may be causing race conditions????? I am not quite sure yet, however, we are clearly getting race conditions while testing, and the issue is coming from

WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
AUDIT    st2.st2common.services.action:action.py:254 The status of action execution is changed from requested to scheduled. <LiveAction.id=6695750428019631d0419e7f, ActionExecution.id=6695750428019631d0419e80>
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
AUDIT    st2.st2actions.worker:worker.py:195 Launching action execution.
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
AUDIT    st2.st2common.services.action:action.py:254 The status of action execution is changed from requested to scheduled. <LiveAction.id=6695750428019631d0419e8d, ActionExecution.id=6695750428019631d0419e8e>
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
AUDIT    st2.st2actions.worker:worker.py:195 Launching action execution.
AUDIT    st2.st2common.services.access:access.py:91 Access granted to "stanley" with the token set to expire at "2024-07-16T19:14:12.940541Z".
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
AUDIT    st2.st2actions.container.base:base.py:173 Liveaction completed
AUDIT    st2.st2common.services.action:action.py:201 Action execution requested. LiveAction.id=6695750428019631d0419e8d, ActionExecution.id=6695750428019631d0419e8e
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
AUDIT    st2.st2actions.container.base:base.py:173 Liveaction completed
AUDIT    st2.st2common.services.action:action.py:201 Action execution requested. LiveAction.id=6695750428019631d0419e7f, ActionExecution.id=6695750428019631d0419e80
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
ERROR    st2.st2common.services.workflows:workflows.py:71 [6695750428019631d0419e80] Workflow execution completed with errors.
ERROR    st2.st2common.services.workflows:workflows.py:71 [6695750428019631d0419e80] {'type': 'error', 'message': 'Execution failed. See result for details.', 'task_id': 'task1', 'result': {'failed': True, 'succeeded': False, 'return_code': 1, 'stdout': '', 'stderr': 'sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper'}}
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
ERROR    st2.st2common.services.workflows:workflows.py:71 [6695750428019631d0419e80] Workflow execution completed with errors.
ERROR    st2.st2common.services.workflows:workflows.py:71 [6695750428019631d0419e80] {'type': 'error', 'message': 'Execution failed. See result for details.', 'task_id': 'task1', 'result': {'failed': True, 'succeeded': False, 'return_code': 1, 'stdout': '', 'stderr': 'sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper'}}
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.
ERROR    st2.st2common.services.workflows:workflows.py:71 [6695750428019631d0419e80] Workflow execution completed with errors.
ERROR    st2.st2common.services.workflows:workflows.py:71 [6695750428019631d0419e80] {'type': 'error', 'message': 'Execution failed. See result for details.', 'task_id': 'task1', 'result': {'failed': True, 'succeeded': False, 'return_code': 1, 'stdout': '', 'stderr': 'sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper'}}
WARNING  st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible.

Every time something fails, we get WARNING st2.st2common.services.coordination:coordination.py:244 Coordination backend is not configured. Code paths which use coordination service will use best effort approach and race conditions are possible. coming up quite a few times. I wonder if I change the tests to auto setup a different backend for tooz to coordinate off of if it would fix this issue (I have no clue what it is using now I am assuming a file rn).

[FileMagic]

I read the test, my last comment was dumb. We do noop(s) on tooz calls. Still no clue what is happening on that end though.

[ajjonen]

The race conditions are due to noop being used instead of actual coordination in the unit testing. Setting up redis and coordination will fix these.

[nzlosh]

+1 @ajjonen. Redis wasn't a hard dependency initially but has been for a couple of years now, I think it makes sense to use redis as the tooz coordinator to run tests.

[FileMagic]

Okay makes sense to me. I'll do that then. Thanks for the heads up good to see I wasn't as crazy as I initially thought with this #6223 (comment)