[3005.x] backport syndic communication fix from #63428

[hurzhurz]

What does this PR do?

Syndic communication is broken in 3005.
This backports the fix from #63428 to 3005.

What issues does this PR fix or reference?

Should probably be the resolution for all these issues:
#62577
#62781
#62933
#63715
#63979

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[garethgreenaway]

@hurzhurz Thanks for the PR, unfortunately we will only be doing 3005 releases for security fixes. That being said, 3006 is expected to be released very soon.

[szjur]

@hurzhurz @garethgreenaway From what I see I can say is that _send_req_async() is not async at all in 3005 - it can easily block for many seconds sending result of just 1 job and while that is happening the syndic does not process new job results received by the local master, adding them to self.job_rets. I have trace log evidence for that. This is not how I understand async code and not how my own async code works.

Syndic performance in 3005 is very poor under load - worse than 3003. Looking at 3003 code I have doubts that this code was truely async there either but it performed better.

[szjur]

Besides, self.async_req_channel = salt.channel.client.ReqChannel.factory(self.opts) and

class ReqChannel:
"""
Factory class to create a sychronous communication channels to the master's
ReqServer. ReqChannels use transports to connect to the ReqServer.
"""

@staticmethod
def factory(opts, **kwargs):
    return SyncWrapper(

class SyncWrapper:
"""
A wrapper to make Async classes synchronous

How can using a "sychronous communication channel" possibly result in something asynchronous? If you ask me _send_req_async() and _send_req_sync() added by this patch are pretty much the same - the former doesn't do anything asynchronously. And since _forward_events() is not a coroutine either I'm not sure why is the syndic code taking all that trouble to pretend this part of the code is asynchronous.

_send_req_async() in Salt 3003 used AsyncReqChannel.factory().