Resolve work stealing deadlock caused by race in move_task_confirm

[fjetter]

This refactors the move_task_confirm and resolves a few minor bugs

• self.scheduler.remove_worker after an unccesseful event called the function improperly (argument mismatch) and would raise either way. Removing the worker is not necessary since the batchedcomm takes care of this and trigger all relevant key reschedulings
• similarly the rescheduling trigger only needed to act if the steal was confirmed and is now a bit more precise

The most important bug is a deadlock caused by a concurrent race condition of stealing requests. I do not prohibit concurrent steals for the same key but rather make the system robust to concurrency if it should happen. The concurrency control I'm applying is an ETag like transaction confirmation using a stealing stimulus ID which is further passed through the transitions on worker side such that it can be traced. Every received response will now be logged other than before were certain code branches were ignored making debugging harder.

• Closes Race condition in work stealing resulting in deadlock #5370
• Tests added / passed
• Passes pre-commit run --all-files

[fjetter]

New tests seem to be a bit brittle and there is a cythonization error. Will need to look into it

[fjetter]

Most of the test failures are unrelated

• Fix regression where unknown tasks were allowed to be stolen #5392
• test_worker_reconnects_mid_compute_multiple_states_on_scheduler
• test_failing_worker_with_additional_replicas_on_cluster
• test_stress_creation_and_deletion

However, there appears to be a genuine failure in test_reschedule_concurrent_requests_deadlock I will need to investigate (timeout; probably a deadlock)

[crusaderky]

I think these constant names generate confusion with Worker.status.
Could you change them e.g. to _TASKSTATE_STATE_CONFIRM etc.?

On a related note, this IMHO highlights that TaskState.state direly needs to become an Enum - it's just too easy to miss a state here, now or in the future.

[fjetter]

Could you change them e.g. to _TASKSTATE_STATE_CONFIRM etc.?

sure

this IMHO highlights that TaskState.state direly needs to become an Enum

agreed. I would prefer doing this in another PR if you don't mind

[jrbourbeau]

Sorry, I missed this outstanding request. @fjetter would you mind pushing up a separate PR with the updated name @crusaderky suggested?

[crusaderky]

agreed. I would prefer doing this in another PR

Yes obviously

[fjetter]

Test failures appear to be unrelated. There is again the cythonization error #5406 which I will address in a follow up PR

[jrbourbeau]

Thanks @fjetter for fixing and @crusaderky for reviewing