[BUG] 3006 minion returns data to both masters in active/active multimaster setup

[vkotarov]

Description
When running jobs against v3006 minions, I can see job returns coming back to both masters.
When running the same job against v3004 minions, the job returns are coming back only to the master that initiated the job.

Setup
Multimaster setup, Active/Active, master version 3004.2

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

In this environment the minion named host01 is running version 3006.2 while the minion on master01 is running version 3004.2

[user@master01 ~]$ salt -L "host01 master01" test.ping
host01:
    True
master01:
    True

-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 2
# of minions returned: 2
# of minions that did not return: 0
# of minions with errors: 0
-------------------------------------------

Expected behavior
Minions send responses only to the master that published the job.

Screenshots

[user@master01 ~]$ salt-run state.event "*job*"
salt/job/20230824094922174054/new	{"jid": "20230824094922174054", "tgt_type": "list", "tgt": ["host01", "master01"], "user": "sudo_user", "fun": "test.ping", "arg": [], "minions": ["host01", "master01"], "missing": [], "_stamp": "2023-08-24T09:49:22.576850"}
salt/job/20230824094922174054/ret/host01	{"cmd": "_return", "id": "host01", "success": true, "return": true, "retcode": 0, "jid": "20230824094922174054", "fun": "test.ping", "fun_args": [], "master_id": "master01.saltstack-environment.com", "_stamp": "2023-08-24T09:49:22.983361"}
salt/job/20230824094922174054/returning	{"_stamp": "2023-08-24T09:49:22.995504"}
salt/job/20230824094922174054/ret/master01	{"cmd": "_return", "id": "master01", "success": true, "return": true, "retcode": 0, "jid": "20230824094922174054", "fun": "test.ping", "fun_args": [], "master_id": "master01.saltstack-environment.com", "_stamp": "2023-08-24T09:49:23.020168"}
salt/job/20230824094922174054/complete	{"returned": ["host01", "master01"], "missing": [], "fun": "test.ping", "arg": [], "_stamp": "2023-08-24T09:49:23.025504"}


[user@master02 ~]$ salt-run state.event "*job*"
salt/job/20230824094922174054/ret/host01	{"cmd": "_return", "id": "host01", "success": true, "return": true, "retcode": 0, "jid": "20230824094922174054", "fun": "test.ping", "fun_args": [], "master_id": "master01.saltstack-environment.com", "_stamp": "2023-08-24T09:49:22.954435"}
salt/job/20230824094922174054/returning	{"_stamp": "2023-08-24T09:49:22.960451"}
salt/job/20230824094922174054/complete	{"returned": ["host01"], "missing": [], "fun": "", "arg": "", "_stamp": "2023-08-24T09:49:22.965321"}

Versions Report

salt --versions-report

    Salt Version:
              Salt: 3006.2
     
    Python Version:
            Python: 3.10.12 (main, Aug  3 2023, 21:47:10) [GCC 11.2.0]
     
    Dependency Versions:
              cffi: 1.14.6
          cherrypy: unknown
          dateutil: 2.8.1
         docker-py: Not Installed
             gitdb: Not Installed
         gitpython: Not Installed
            Jinja2: 3.1.2
           libgit2: Not Installed
      looseversion: 1.0.2
          M2Crypto: Not Installed
              Mako: Not Installed
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         packaging: 22.0
         pycparser: 2.21
          pycrypto: Not Installed
      pycryptodome: 3.9.8
            pygit2: Not Installed
      python-gnupg: 0.4.8
            PyYAML: 6.0.1
             PyZMQ: 23.2.0
            relenv: 0.13.3
             smmap: Not Installed
           timelib: 0.2.4
           Tornado: 4.5.3
               ZMQ: 4.3.4
     
    System Versions:
              dist: oracle 7.9 
            locale: utf-8
           machine: x86_64
           release: 4.1.12-124.60.1.el7uek.x86_64
            system: Linux
           version: Oracle Linux Server 7.9

Additional context
Saltstack Config also installed and integrated. It clearly shows multiple rows with responses which is confusing as the amount of rows is higher than number of targeted minions and one and the same information (return) is duplicated.

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[frebib]

Duplicate of #62834

[whytewolf]

closing as a duplicate of #62834