[BUG] Salt-call hangs on Windows - version 3005.1

[AGirin]

Description
salt-call now hangs on Windows and needs to be killed manually. Observed on many servers that we built with our bootstrap process. Windows 2019 and 2022 were tested.

Setup
Any state - salt-call hangs.

Versions Report

          Salt: 3005.1

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.4
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.8.14 (tags/v3.8.14:f43e767, Sep 30 2022, 10:51:05) [MSC v.1929 64 bit (AMD64)]
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 19.0.0
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist:
        locale: cp1252
       machine: AMD64
       release: 2022Server
        system: Windows
       version: 2022Server 10.0.20348 SP0 Multiprocessor Free

[AGirin]

[OrangeDog]

Onedir or classic packaging?

[AGirin]

Classic I believe.

[OrangeDog]

Does it look like the same causes as #62550? But that should be fixed in 3005.1

[AGirin]

I saw that one... It looks like it was addressed for Ubuntu.. Not Windows.. We are running 3005.1...

[AGirin]

I added kill statement to our bootstrap process in the meantime...

[whytewolf]

in your highstate are you using any imports? I'm trying to track this down and think i have a lead but just want to be sure. but I am not sure why it is only happening on windows.

when i say imports i mean jinja that imports a map.jinja or any other code that does import <"filename"> as <variable> or from <"filename"> import variable [with|without] [context]

[twangboy]

@AGirin Could you post the exact salt-call command you are running? Are you using --local?

[whytewolf]

@twangboy I was able to replicate this also.

here is the setup i did to get the fastest replication of the issue

root@salt00:/srv/salt/tests# cat test.sls
{% import "map.jinja" as settings%}

always-passes-with-any-kwarg:
  test.nop:
    - name: foo
    - something: else
    - foo: bar

always-passes:
  test.succeed_without_changes:
    - name: foo

always-fails:
  test.fail_without_changes:
    - name: foo

always-changes-and-succeeds:
  test.succeed_with_changes:
    - name: foo

always-changes-and-fails:
  test.fail_with_changes:
    - name: foo

my-custom-combo:
  test.configurable_test_state:
    - name: foo
    - changes: True
    - result: False
    - comment: bar.baz
    - warnings: A warning

root@salt00:/srv/salt/tests# cat /srv/salt/map.jinja
{% set settings = ["this"] %}

and the command i ran on the wndows host was

salt-call slsutil.renderer salt://tests/tests.sls -l all

which gave the following output

windows hang log.txt
hanging at that last line.

same hang also happens when run with state.apply instead of the slsutil.renderer

[whytewolf]

as a test. i just built same setup local to the windows minion and using --local and it doesn't hang with --local and local files.

[whytewolf]

this does look to leave hanging processes in a running minion also.

[twangboy]

Upgrading to pyzmq to 22.0.3 fixes the issue

[AGirin]

How do we update this on Windows? And sorry for late reply.

[AGirin]

For that matter we do not use --local.

[AGirin]

And really any state hagngs...

salt-call.bat state.apply grains.roles or salt-call state.apply --> both hang at the end.

[AGirin]

Will salt pip install --upgrade pyzmq work?

[AGirin]

Previous command does not work... What is the correct way to do this on Windows?

[twangboy]

So, you can only update pyzmq with the original packages for Windows. The OneDir/Tiamat packages don't pick up the change. We're probably going to have to re-package for Windows with the updated pyzmq. We're trying to decide if that means a re-tag or if we can fix Tiamat to pick up the updated pyzmq.

[twangboy]

If you installed Salt using the Classic Package, you can upgrade pyzmq by doing the following:

Observe the version of pyzmq that Salt is using:

# from the master
salt <minion id> test.versions

# on the minion
salt-call --local test.versions

Look for the following in the return:

PyZMQ: 18.0.1

Upgrade pyzmq:

# From the master
salt <minion id> pip.install pyzmq==22.0.3

# from the minion
salt-call --local pip.install pyzmq==22.0.3

Then check the pyzmq version again to make sure it is updated.

You may need to restart the minion.

salt <minion id> service.restart salt-minion
#or
salt-call --local service.restart salt-minion

We will be fixing this in either a 3005.2 release or a 3005.1-2 build for Windows packages.

[twangboy]

You could also run pip directly from the Salt python environment. From a command line:

# CD to bin directory
cd "C:\Program Files\Salt Project\Salt\bin"

# Upgrade pyzmq
python.exe -m pip install pyzmq==22.0.3

# Verify the installed version
python.exe -m pip show pyzmq

# Look for
Version: 22.0.3

[twangboy]

The above commands will only work if you have installed the Classic Package of Salt. If you are using a OneDir/Tiamat package, you will have to wait for the fix (either a patched build, or a new release).

[seandjones92]

Hi, just wanted to post here and confirm that in another environment upgrading pyzmq to version 22.0.3 addressed the behavior as noted

[AGirin]

Let me try the commands you provided.

[geisingerDev]

is this fix only for the classic package of the salt-minion?