[BUG] Syndication still broken in 3005.1 (?)

[Heiko-san]

Description

Hi, I know there have been a lot of tickets around the issue, which got closed as duplicates of #62577 .
However #62577 was closed as resolved itself, but still we run into the issue.
Maybe we are doing something wrong, but we installed the latest saltstack version of onedir installation from just a few minutes ago, so it seems there is still an issue(?)

Syndication works on the previous version (3004.2).
With 3005.1 we see the errors named in #62577 in the syndic logs.

2023-02-14 11:57:01,786 [salt.utils.event                                :430 ][ERROR   ][5936] Unable to connect pusher: Stream is closed

As far as I can tell from the other issue, it seems to be related to pyzmq version?
How can that be with the onedir installation, I thought it comes with its own python, am I wrong?

The behavior we can see is:

• Doing a salt '*' test.ping on a downstream master, succeeds on its minions (3004.2 & 3005.1).
• Doing a salt '*' test.ping on MoM at 3004.2 succeeds on all minions.
• Doing a salt '*' test.ping on MoM at 3005.1 fails, it reports timeout of the downstream masters (which themselves don't run salt-minion).

Setup

• We have many different Minions (Debian, Centos, Ubuntu, OEL, all different OS Releases), all of them run saltstack as the new onedir installation with version 3005.1
• These Minions connect to different masters which are running the same onedir installation of salt-master at version 3005.1. They run syndic, but no salt-minion itself.
• These masters connect to one MoM, which itself runs the very same installation of onedir installation of salt-master at version 3005.1.
• We installed these packages (where the python3 packages should not be used by salt onedir, I think (?)

apt-get -y install git python3-pip python3-git python3-setuptools salt-api salt-master salt-syndic

• We used this Repo, so the onedir installation.

deb https://repo.saltproject.io/salt/py3/ubuntu/20.04/amd64/3005 focal main

• The last build (of the master servers and MoM), which still has the error was just a jew minutes ago / today.

Versions Report

• salt --versions-report from MoM

Salt Version:
          Salt: 3005.1
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.16 (main, Jan  6 2023, 22:49:56)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 4.15.0-202-generic
        system: Linux
       version: Ubuntu 20.04 focal

• salt --versions-report from one of the downstream masters

Salt Version:
          Salt: 3005.1
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.16 (main, Jan  6 2023, 22:49:56)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 4.15.0-202-generic
        system: Linux
       version: Ubuntu 20.04 focal

• salt-call --versions-report from one of the minions

Salt Version:
          Salt: 3005.1
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.16 (main, Jan  6 2023, 22:51:58)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: oracle 7.9 
        locale: utf-8
       machine: x86_64
       release: 4.14.35-2047.521.4.el7uek.x86_64
        system: Linux
       version: Oracle Linux Server 7.9

[lkubb]

Note that the latest releases do not contain the mentioned commit. It will be included in v3006.

Edit: Additionally, the PR linked in the issue seems to be off. It should have been #63428 from what I can tell.