[BUG] salt-ssh gpg pillar rendering failure

[edgan]

Description
I use gpg encrypted pillar files. This has worked for many versions of salt, going back years. Now the same setup with 3003 throws errors on state.highstate. If I downgrade to 3002.6, and run the same command it works. Even 3003 with pillar.items works.

The file is encrypted with a key in /home/user/.gnupg.

Setup
./Saltfile:

salt-ssh:
  config_dir: .
  max_procs: 1
  ssh_wipe: True

./master:

gpg_keydir: /home/user/.gnupg

file_roots:
  base:
    - ..
    - ../../salt-grains

pillar_roots:
  base:
    - ../../salt-pillars
    - ../../salt-grains

salt-pillars/top.sls:

base:
  '*':
    - kube.foo.ssl.ca.key

raw:
salt-pillars/kube/foo/ssl/ca/key/init.sls:

kube:
  foo:
    ssl:
      ca:
        key:
          data: |
            -----BEGIN EC PRIVATE KEY-----
            ...
            ...
            ...
            -----END EC PRIVATE KEY-----

encrypted:
salt-pillars/kube/foo/ssl/ca/key/init.sls:

#!gpg|yaml
-----BEGIN PGP MESSAGE-----

...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
-----END PGP MESSAGE-----

Errors

[WARNING ] Could not decrypt cipher b'-----BEGIN PGP MESSAGE-----\n\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n................................................................\n............................\n.....
\n-----END PGP MESSAGE-----', received: b"gpg: keyblock resource '/var/tmp/.user_ab4926_salt/gpgkeys/pubring.kbx': No such file or directory\n[GNUPG:] ERROR add_keyblock_resource 33587281\n[GNUPG:] ENC_TO A0251D47D061704C 1 0\n[GNUPG:] ERROR keydb_search 33554445\n[GNUPG:] ERROR keydb_search 33554445\ngpg: encrypted with RSA key, ID A0251D47D061704C\n[GNUPG:] NO_SECKEY A0251D47D061704C\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed: No secret key\n[GNUPG:] END_DECRYPTION\n"
[ERROR   ] SLS 'kube.foo.ssl.ca.key' does not render to a dictionary
[ERROR   ] The rendered pillar sls file, 'kube.foo.ssl.ca.key' state did not return the expected data format. This is a sign of a malformed pillar sls file. Returned errors: 'SLS 'kube.foo.ssl.ca.key' does not render to a dictionary'
[CRITICAL] Pillar render error: SLS 'kube.foo.ssl.ca.key' does not render to a dictionary

Steps to Reproduce the behavior
3003 fails:
salt-ssh -i fqdn state.highstate

3003 works:
salt-ssh -i fqdn pillar.items

3002.6 works:
salt-ssh -i fqdn state.highstate

Expected behavior
The encrypt pillars to decrypt properly running an salt-ssh state.highstate

GnuPG version

gnupg2-2.2.25-2.fc33.x86_64

Versions Report

salt-ssh --versions-report
Salt Version:
          Salt: 3003

Dependency Versions:
          cffi: 1.14.1
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: 1.1.3
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: 2.6.1
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 20 2021, 00:00:00)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 22.0.3
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: fedora 33
        locale: utf-8
       machine: x86_64
       release: 5.11.10-200.fc33.x86_64
        system: Linux
       version: Fedora 33

[xuhcc]

I'm getting similar errors with salt-ssh 3003.1:

gpg: keyblock resource '/var/tmp/.***_salt/gpgkeys/pubring.kbx': No such file or directory

salt-ssh 3002.6 works fine.

[lapfrank12]

Same issue still happening with 3003.2 which was just released. Just intrigued, since this is a P2, which release can we expect this to be fixed on?

[lapfrank12]

Salt 3004 was released but no mention of this issue being fixed yet. Any updates?

[kfmak]

It looks like we have the same issue on our platform. We are using Ubuntu 20.04 with these packages installed on our master:

• salt-master 3004.2+ds-1
• gpg 2.2.19-3ubuntu2.2
• libgpgme11:amd64 1.13.1-7ubuntu2
• libgpg-error0:amd64 1.37-1

On minion side, we firstly add a version issue (3003 installed). After the upgrade, it did not solve the issue. Here the packages installed:

• salt-minion 3004.2+ds-1
• gpg 2.2.19-3ubuntu2.2
• libgpgme11:amd64 1.13.1-7ubuntu2
• libgpg-error0:amd64 1.37-1

The issues appeared after we update our server to the latest packages versions. Here the generated message on our master when running state.apply. Same behaviors can be generated on all minions connected to the master.

[WARNING ] Could not decrypt cipher '-----BEGIN PGP MESSAGE-----\n...\n-----END PGP MESSAGE-----', received: "gpg: keyblock resource '/var/tmp/.ubuntu_3998c6_salt/gpgkeys/pubring.kbx': No such file or directory\n[GNUPG:] ERROR add_keyblock_resource 33587281\n[GNUPG:
] ENC_TO XXXXXXXXXXXX 1 0\n[GNUPG:] ERROR keydb_search 33554445\n[GNUPG:] ERROR keydb_search 33554445\ngpg: encrypted with RSA key, ID XXXXXXXXXXXX\n[GNUPG:] NO_SECKEY XXXXXXXXXXXX\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed:
 No secret key\n[GNUPG:] END_DECRYPTION\n"

When creating an encrypted message with gpg and decipher it manually, it works without any issue. When reusing thefailing encrypted payload from the warning message, it works as well.

$ echo test | gpg -a -e -r salt-master
$ cat test.gpg | sudo gpg --homedir /etc/salt/gpgkeys/ -d -a
gpg: encrypted with 3072-bit RSA key, ID XXXXXXXXXXXX, created 2021-09-14
      "salt-master"
test

A gpg-agent is correctly running on our servers using this command: gpg-agent --homedir /etc/salt/gpgkeys --use-standard-socket --daemon.

[mbochenk]

Running into the same issue with salt-ssh on 3006.1, 3004.2 and 3005 (whatever the latest is right now).

[wwimberly]

I'm seeing the same issue on 3006.2. Is there a workaround to this other than not using gpg encrypted pillar?

[lkubb]

I found the cause for this, see #36796 (comment). Will submit a test + fix for this shortly (if this does not turn into a yak shaving situation at least).

[dwoz]

Closing as fixed per #65067 (comment)