[BUG] 3005.1 salt-syndic: [ERROR ] Unable to connect pusher: Stream is closed

[cbalan58]

Description
After upgrading to 3005.1 the syndic servers are not able to talk to master (at same Salt version).
This means that highstate run from master is not reaching any minions.

Setup
1 master, 3 syndics and ~5000 minions. Master and syndics are at salt v3005.1, minions at older versions (3004.2, 3003 etc).
Master has 8Gi of memory and syndic 16, 2vcpu for master and 8 for syndic, the network runs on 25G ports.

• on-prem machine
• VM (VMWare)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

• from master:
  /usr/bin/salt '*' state.highstate
  of minions targeted: 3
  of minions returned: 0
  of minions that did not return: 3
  of minions with errors: 0

---

ERROR: Minions returned with non-zero exit code

Expected behavior
Highstate should return minions' executed states but it stops at syndics (the 3 minions from above output).

Versions Report

salt --versions-report

-- on master: Salt Version: Salt: 3005.1

Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 8 2022, 11:32:15)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 20.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: rocky 8.7 Green Obsidian
locale: UTF-8
machine: x86_64
release: 4.18.0-425.10.1.el8_7.x86_64
system: Linux
version: Rocky Linux 8.7 Green Obsidian
-- on syndicate:
Salt Version:
Salt: 3005.1

Dependency Versions:
cffi: 1.11.5
cherrypy: Not Installed
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.10.1
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.14
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Nov 8 2022, 11:32:15)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 20.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: rocky 8.7 Green Obsidian
locale: UTF-8
machine: x86_64
release: 4.18.0-425.10.1.el8_7.x86_64
system: Linux
version: Rocky Linux 8.7 Green Obsidian

Additional context
systemctl status salt-syndic
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Unable to call _fire_master on x.x.x.x
[CRITICAL] Unable to call _fire_master on any masters!
[ERROR ] Unable to connect pusher: Stream is closed
[ERROR ] Unable to connect pusher: Stream is closed

-- from salt-syndic with log-level 'all'
[DEBUG ] Decrypting the current master AES key
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] salt.crypt.verify_signature: Loading public key
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] salt.crypt.verify_signature: Verifying signature
[DEBUG ] <salt.crypt.AsyncAuth object at 0x7f3699643828> Got new master aes key.
[DEBUG ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_188cb2711b_pub.ipc
[DEBUG ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_188cb2711b_pull.ipc
[TRACE ] IPCClient: Connecting to socket: /var/run/salt/minion/minion_event_188cb2711b_pull.ipc
[TRACE ] IPCClient: Connecting to socket: /var/run/salt/minion/minion_event_188cb2711b_pull.ipc
[ERROR ] Unable to connect pusher: Stream is closed
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/utils/event.py", line 427, in connect_pull
self.pusher.connect(timeout=timeout)
File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/transport/ipc.py", line 345, in _connect
yield self.stream.connect(sock_addr)
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
[DEBUG ] Closing AsyncReqChannel instance

-- telnet to master on ports 4505 and 4506 works OK, they are VMs in same location and network (to keep things easier, am using only the 1 syndic that is in same network with master, but same error is present on all 3 syndic servers)
-- i moved the /etc/salt/pki/minion/syndic_master.pub and after service restart it got back OK, diff with older file showed nothing.
-- top processes with nr of opened files seems to not be too high:
-- on master:
13213 salt-mast
12518 glusterfs
666 glusterd
-- on syndic:
19369 salt-mast
10761 glusterfs
1135 salt-synd
696 polkitd

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

Duplicate: #62577