Salt-master doesn't renew vault token

[thenoid]

Description of Issue/Question

Vault allows for an issued token to be renewed while it is still active. When the salt-master is configured to use token auth, it does not attempt to renew the token, instead, it lets it expire and then fail.

This seems like poor behavior, especially since the other auth method approle appears to validate if it's expired and request a new token. (older versions of salt do not support approle).

It would be beneficial if the salt-master monitored the remaining time to live of it's configured token and renew'd it.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

/etc/salt/master.d/vault.conf
vault:
  url: http://avault.server:8200
  auth:
    method: token
    token: randomjunkhere

Steps to Reproduce Issue

Provision a token with the appropriate polices, wait until the ttl expires, be unable to access vault.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

salt --versions-report
Salt Version:
           Salt: 2017.7.8

Dependency Versions:
           cffi: 1.5.2
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.10
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Nov 12 2018, 14:36:49)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-1075-aws
         system: Linux
        version: Ubuntu 16.04 xenial

[Ch3LL]

ping @carlpett seems you added a lot of the vault features. do you have any ideas here?

[carlpett]

It has been a few years, so somewhat foggy memory, but if I remember correctly, at the time of the initial implementation, we required a root token since Vault did not support granting some privilege needed. Since root tokens do not expire, renewal wasn't required.

Since it seems this is no longer the case, it would be a very reasonable addition to ensure non-root tokens work well, in order to discourage using them. I'm myself no longer in a position where I use either Vault or Saltstack, sadly, so my ability to contribute this is very limited.

[Ch3LL]

thanks for the input i really appreciate it :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[idontwanttosignin]

Reviving this as it is still an issue we are affected by.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[ptitdoc]

In the meantime, maybe it can be good to improve documentation of salt/vault module setup.

In my case, I'm using method: approle but I'm still providing a secret_id along with the role_id which expires (when running vault write -f auth/approle/role/saltmaster/secret-id, it shows a TTL of 768h).

I also face secret_id expiration in my case and salt starts logging messages as following when the secret_id expire :
Failed to get token from master! An error was returned: Bad Request

[JeremyHutchings]

@ptitdoc Did you find any resolution to this ? As I cannot get salt working with vault with token or approle at the moment due to this issue.

[ptitdoc]

I's currently working, I only need to reset the secret-id every month because of the TTL using vault write -f auth/approle/role/saltmaster/secret-id

hashicorp/vault#5445

So there is two ways around:

1/ Change the secret_id_ttl by providing a ttl in seconds but it is buggy. If you exceed 999999 seconds, it will just show 768h of ttl when writing a new secret-id:
vault write auth/approle/role/saltmaster secret_id_ttl=999999
vault write -f auth/approle/role/saltmaster/secret-id

2/ Set the secret_id_ttl to zero to disable completely ttl:
vault write auth/approle/role/saltmaster secret_id_ttl=0
vault write -f auth/approle/role/saltmaster/secret-id

[JeremyHutchings]

@ptitdoc thanks, still trying to get this working myself.

What do you have in /etc/salt/master.d/vault.conf to get this working and what's the policy you have in vault for the role?

I'm getting the message of "Forbidden" when the minion is requesting a token from the master and am unset if it's the master refusing the minion or Vault upstream.

[ptitdoc]

Yeah that is hard to troubleshoot, and I struggled to make it work because hashicorp vault configuration is somehow hard to understand.

My vault.conf look like this:

vault:
  url: https://myurl:8200
  #verify: /etc/ssl/certs/ca-certificates.crt
  #role_name: minion_role
  auth:
    # Need to be able to replay token 3 times in our usecase (pass read, create, write)
    uses: 30
    ttl: 30
    method: approle
    role_id:  my_role_id
    secret_id: my_secret_id
  policies:
    - saltmaster

I loaded the following policy for the approle:

$ cat /etc/vault/saltmaster.hcl 
# This section grants all access on "salt/*". 
# Further restrictions can be applied to this broad policy, as shown below.

# Minimum required access for KV secrets engine V1 and V2

# V1
path "salt/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "salt/" {
  capabilities = ["list"]
}

# V2
path "salt/data/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "salt/data/" {
  capabilities = ["list"]
}

# THIS ENTRY ALLOW SALTUTIL.PILLAR_REFRESH TO WORK PROPERLY
path "auth/token/create" {
 capabilities = ["create", "read", "update"]
}

[JeremyHutchings]

@ptitdoc Thanks. How did you set up that policy and approle in vault to get salt to be happy with it ?

I've copied yours to test and salt is reporting "Bad: Request" with no other debug I can find atm.

[ptitdoc]

I don't really remember but there are several ways, including pushing a vault policy using salt states

I followed a mix of several tutorials and vault documentation such as:
https://binbash.fr/topic/64/vault-ssh-et-secrets-de-saltstack-dans-vault
https://backbeat.tech/blog/secure-servers-with-saltstack-and-vault-part-1
https://www.vaultproject.io/docs/auth/approle
https://www.vaultproject.io/docs/concepts/policies
https://docs.saltproject.io/en/latest/ref/states/all/salt.states.vault.html

Importing a policy file can be done like this:
vault policy write policy-name policy-file.hcl

Associating policies if your already created your approle can be done using:
vault write auth/approle/role/saltmaster policies=saltmaster

[JeremyHutchings]

@ptitdoc Thanks, I've been though most of them, though I think there are differences between a locally installed vault and https://portal.cloud.hashicorp.com as no policy I've tried is allowing the creation of tokens it seems

[dgengtek]

In my case, I'm using method: approle but I'm still providing a secret_id along with the role_id which expires (when running vault write -f auth/approle/role/saltmaster/secret-id, it shows a TTL of 768h).

@ptitdoc vault is not buggy, you need to increase the max_lease_ttl for your auth backend in vault if you want to get longer valid secrets than the default 768h

This issue is talking about renewal of tokens in case the ttl of the token is less than the possible max_ttl of the given auth backend so that saltstack could request to extend the lifetime of the token. Expired secrets(the token when using token method or the secret_id when using approle method) should be rotated by some other mechanism and not handled by saltstack itself in my opinion.

This seems like poor behavior, especially since the other auth method approle appears to validate if it's expired and request a new token. (older versions of salt do not support approle).

I don't think this is an issue but rather a feature request. The approle method does not renew anything it just does a login to vault with the given role_id:secret_id and requests a new token. The secret_id in the approle method is equivalent to the token method.

[darkpixel]

Unfortunately there are no good docs for setting up Salt with vault.
It's been a nightmare on my end. Over the past few years I've spun up test environments about 9 times now, and every single time I manage to run into "Salt's vault integration is just broken" or "Salt's documentation is terrible and you sorta have to mash together 15 different tutorials from a bunch of different websites to possibly get it working".

The debug output on the master when things don't work is dismal.

Currently using approle for authentication, running salt-call -l debug vault.read_secret somepath/somesecret throws the following on the master:

[salt.utils.lazy                                                    :99  ][DEBUG   ][28099] LazyLoaded vault.generate_token
[salt.utils.event                                                   :817 ][DEBUG   ][28099] Sending event: tag = salt/run/20220620220902705208/new; data = {'fun': 'runner.vault.generate_token', 'jid': '20220620220902705208', 'user': 'root', 'fun_args': ['saltmaster.redacted.tld', 'NzSGnbat8B2nZJiFo0c62qaegb1AMT/Snz0ODjwTFd3aU+DnJnHmXSCC1GIBh1YtnEs2drCGCHf8LDe4HxWdMF/Xjkv39RZV8vIg5rKMYadKWHLF1+cqM+CuyNiZi/tsnx/ArRVlnoZd4u45R19/enomuusnSPTPi9GD9np8LcRNoHYM3KgJdrd6RKlQ73VbTh+guriQe8OMWcR6eE6zLz1rhYMeLf2Kg84eHE+u665tPPALhgzBkkivzKdoZV/nGEEWUjBCMugM+u68m9ebtP9vzdTTzAD1HQIATvrlhmIYZXFdGbPVj8Wtdl4vbW/8WO1nJiB6PdqQaTXmWK9FSQ==', False, None, None], '_stamp': '2022-06-20T22:09:03.159122'}
[salt.loaded.int.runners.vault                                      :46  ][DEBUG   ][28099] Token generation request for saltmaster.redacted.tld (impersonated by master: False)
[urllib3.connectionpool                                             :1001][DEBUG   ][28108] Starting new HTTPS connection (1): vault-server.redacted.tld:443
[urllib3.connectionpool                                             :456 ][DEBUG   ][28108] https://vault-server.redacted.tld:443 "POST /v1/auth/approle/login HTTP/1.1" 200 692
[salt.loaded.int.runners.vault                                      :68  ][DEBUG   ][28099] Vault token expired. Recreating one
[salt.loaded.int.runners.vault                                      :220 ][DEBUG   ][28099] saltmaster.redacted.tld policies: ['salt-master']
[urllib3.connectionpool                                             :1001][DEBUG   ][28108] Starting new HTTPS connection (1): vault-server.redacted.tld:443                                                                                                                                                                                                │
[urllib3.connectionpool                                             :456 ][DEBUG   ][28108] https://vault-server.redacted.tld:443 "POST /v1/auth/token/create/salt-master HTTP/1.1" 400 40
[salt.utils.event                                                   :817 ][DEBUG   ][28099] Sending event: tag = salt/run/20220620220902705208/ret; data = {'fun': 'runner.vault.generate_token', 'jid': '20220620220902705208', 'user': 'root', 'fun_args': ['saltmaster.redacted.tld', 'NzSGnbat8B2nZJiFo0c62qaegb1AMT/Snz0ODjwTFd3aU+DnJnHmXSCC1GIBh1YtnEs2drCGCHf8LDe4HxWdMF/Xjkv39RZV8vIg5rKMYadKWHLF1+cqM+CuyNiZi/tsnx/ArRVlnoZd4u45R19/enomuusnSPTPi9GD9np8LcRNoHYM3KgJdrd6RKlQ73VbTh+guriQe8OMWcR6eE6zLz1rhYMeLf2Kg84eHE+u665tPPALhgzBkkivzKdoZV/nGEEWUjBCMugM+u68m9ebtP9vzdTTzAD1HQIATvrlhmIYZXFdGbPVj8Wtdl4vbW/8WO1nJiB6PdqQaTXmWK9FSQ==', False, None, None], '_stamp': '2022-06-20T22:09:03.343232', 'return': {'error': 'Bad Request'}, 'success': True}

It seems like it's having a problem creating a new token, but using the vault command line tool with the approle credentials works just fine.

I think salt's broken, but there isn't enough debug output to tell what's going on exactly, and I'm not interested in downgrading the vault server to use HTTP so I can tcpdump it.

[darkpixel]

I should add that I've put it into token auth mode and tried both an issued token as well as the root token and I get the same results.

[hmalinov]

...
[urllib3.connectionpool                                             :456 ][DEBUG   ][28108] https://vault-server.redacted.tld:443 "POST /v1/auth/token/create/salt-master HTTP/1.1" 400 40
[salt.utils.event                                                   :817 ][DEBUG   ][28099] Sending event: tag = salt/run/20220620220902705208/ret; data = {'fun': 'runner.vault.generate_token', 'jid': '20220620220902705208', 'user': 'root', 'fun_args': ['saltmaster.redacted.tld', 'NzSGnbat8B2nZJiFo0c62qaegb1AMT/Snz0ODjwTFd3aU+DnJnHmXSCC1GIBh1YtnEs2drCGCHf8LDe4HxWdMF/Xjkv39RZV8vIg5rKMYadKWHLF1+cqM+CuyNiZi/tsnx/ArRVlnoZd4u45R19/enomuusnSPTPi9GD9np8LcRNoHYM3KgJdrd6RKlQ73VbTh+guriQe8OMWcR6eE6zLz1rhYMeLf2Kg84eHE+u665tPPALhgzBkkivzKdoZV/nGEEWUjBCMugM+u68m9ebtP9vzdTTzAD1HQIATvrlhmIYZXFdGbPVj8Wtdl4vbW/8WO1nJiB6PdqQaTXmWK9FSQ==', False, None, None], '_stamp': '2022-06-20T22:09:03.343232', 'return': {'error': 'Bad Request'}, 'success': True}

@darkpixel The fact that you have salt-master after /v1/auth/token/create looks like you have in the configuration:

vault:
  url: https://<url>
  role_name: salt-master

My investigation lead that this makes the token creation to fail with an error "unknown role .."
Remove the role_name and try again. It seems that with approle it is not needed.