You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description
Using only s3fs for a fileserver_backend does not work properly (cannot find top.sls) without the addition of roots to fileserver_backend and then pointing file_roots to the s3cache in /var/cache/salt/minion/s3cache/$S3_BUCKET.
Setup
Tested on a AWS t2.micro EC2 instance (IMDSv2) running AL2022 with 5.15.57-30.131.amzn2022.x86_64 kernel
Linux ip-10-0-0-233.ec2.internal 5.15.57-30.131.amzn2022.x86_64 #1 SMP Wed Aug 17 23:50:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Full non-working minion configuration (see error below)
top_file_merging_strategy: same
##### File Directory Settings #####
##########################################
# The Salt Minion can redirect all file server operations to a local directory,
# this allows for the same state tree that is on the master to be used if
# copied completely onto the minion. This is a literal copy of the settings on
# the master but used to reference a local directory on the minion.
# Set the file client. The client defaults to looking on the master server for
# files, but can be directed to look at the local file directory setting
# defined below by setting it to "local". Setting a local file_client runs the
# minion in masterless mode.
file_client: local
fileserver_backend:
- s3fs
s3.keyid: REDACTED
s3.key: REDACTED
s3.location: us-east-1
s3.service_url: s3.us-east-1.amazonaws.com
s3.buckets:
- salt-config-676XXXXXXXX
s3.https_enable: true
s3.verify_ssl: True
# The file directory works on environments passed to the minion, each environment
# can have multiple root directories, the subdirectories in the multiple file
# roots cannot match, otherwise the downloaded files will not be able to be
# reliably ensured. A base environment is required to house the top file.
# Example:
#file_roots:
# base:
# - /srv/salt
Workaround required to get s3fs working
top_file_merging_strategy: same
##### File Directory Settings #####
##########################################
# The Salt Minion can redirect all file server operations to a local directory,
# this allows for the same state tree that is on the master to be used if
# copied completely onto the minion. This is a literal copy of the settings on
# the master but used to reference a local directory on the minion.
# Set the file client. The client defaults to looking on the master server for
# files, but can be directed to look at the local file directory setting
# defined below by setting it to "local". Setting a local file_client runs the
# minion in masterless mode.
file_client: local
fileserver_backend:
- s3fs
- roots
s3.keyid: REDACTED
s3.key: REDACTED
s3.location: us-east-1
s3.service_url: s3.us-east-1.amazonaws.com
s3.buckets:
- salt-config-676XXXXXXXX
s3.https_enable: true
s3.verify_ssl: True
# The file directory works on environments passed to the minion, each environment
# can have multiple root directories, the subdirectories in the multiple file
# roots cannot match, otherwise the downloaded files will not be able to be
# reliably ensured. A base environment is required to house the top file.
# Example:
file_roots:
base:
- /var/cache/salt/minion/s3cache/salt-config-676XXXXXXXX
Steps to Reproduce the behavior
Launch an EC2 instance and create an S3 bucket to serve salt files
Configure minion using the above configs and point to the proper bucket
Attempt to apply state with salt-call state.apply or salt-call --local state.apply
Both of these produce the following error (debug logs below for greater detail):
[ERROR ] The 'base' saltenv has no top file, and the fallback saltenv specified by default_top (base) also has no top file
local:
----------
ID: states
Function: no.None
Result: False
Comment: No Top file or master_tops data matches found. Please see master log for details.
Changes:
Full debug using salt-call state.apply -l debug
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ip-10-0-0-233.ec2.internal
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG ] Configuration file path: /etc/salt/minion
[DEBUG ] Grains refresh requested. Refreshing grains.
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load dns
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load napalm
[DEBUG ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pythoncom
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load win32com
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load win32api
[DEBUG ] Override __utils__: <module 'salt.loaded.int.grains.zfs' from '/opt/saltstack/salt/run/salt/grains/zfs.py'>
[DEBUG ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Elapsed time getting FQDNs: 0.024873018264770508 seconds
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load _bootlocale
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load lsb_release
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded zfs.is_supported
[DEBUG ] Determining pillar cache
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded state.apply
[DEBUG ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded direct_call.execute
[DEBUG ] Override __grains__: <module 'salt.loaded.int.module.grains' from '/opt/saltstack/salt/run/salt/modules/grains.py'>
[DEBUG ] The functions from module 'grains' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded grains.get
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load immutables
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch_helpers
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load backports_abc
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load backports
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load saltwinshell
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded saltutil.is_running
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load smbprotocol
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pypsexec
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load winrm
[DEBUG ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded config.get
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load git
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pygit2
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Updating s3fs fileserver cache
[DEBUG ] Refreshing buckets cache file
[DEBUG ] The functions from module 's3' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3.query
[DEBUG ] S3 Request: https://salt-config-67XXXXXXXXXXX.s3.us-east-1.amazonaws.com/?marker=
[DEBUG ] S3 Headers::
[DEBUG ] Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=26bcff00305dad8ac2b1f7ab2e1f3bb8120d36f376ea6d2475d112c0ce96c158
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load netrc
[DEBUG ] Starting new HTTPS connection (1): salt-config-676XXXXXXXX .s3.us-east-1.amazonaws.com:443
[DEBUG ] https://salt-config-676XXXXXXXXXXX.s3.us-east-1.amazonaws.com:443 "GET /?marker= HTTP/1.1" 200 None
[DEBUG ] S3 Response Status Code: 200
[DEBUG ] Writing buckets cache file
[INFO ] Syncing local cache from S3...
[INFO ] salt-config-676XXXXXXXX - : base.sls
[INFO ] salt-config-676XXXXXXXX - : top.sls
[INFO ] Sync local cache from S3 completed.
[DEBUG ] Gathering pillar data for state run
[DEBUG ] Determining pillar cache
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] Finished gathering pillar data for state run
[INFO ] Loading fresh modules for state activity
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[DEBUG ] Could not find file 'salt://top.sls' in saltenv 'base'
[DEBUG ] No contents loaded for saltenv 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[DEBUG ] In saltenv '', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG ] In saltenv '', ** considering ** path '/var/cache/salt/minion/files/top.sls' to resolve 'salt://top.sls'
[DEBUG ] Fetching file from saltenv '', ** attempting ** 'salt://top.sls'
[DEBUG ] No dest file found
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[INFO ] Fetching file from saltenv '', ** done ** 'top.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/op.sls
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'jinja' renderer: 0.003148794174194336
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/op.sls:
base:
'*':
- base
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'yaml' renderer: 0.0002684593200683594
[ERROR ] The 'base' saltenv has no top file, and the fallback saltenv specified by default_top (base) also has no top file
[DEBUG ] Section for saltenv 'base' in the '' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'same' and the saltenvs do not match
[DEBUG ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded state.check_result
[DEBUG ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded highstate.output
local:
----------
ID: states
Function: no.None
Result: False
Comment: No Top file or master_tops data matches found. Please see master log for details.
Changes:
Summary for local
------------
Succeeded: 0
Failed: 1
------------
Total states run: 1
Total run time: 0.000 ms
[DEBUG ] Restoring sys.argv to: ['/opt/saltstack/salt/run/run', 'state.apply', '-l', 'debug']
Note that even though the above fails, the s3cache is still created and populated:
Debug output of workaround using fileserver_backend roots workaround
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: ip-10-0-0-233.ec2.internal
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load log4mongo
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load raven
[DEBUG ] Configuration file path: /etc/salt/minion
[DEBUG ] Grains refresh requested. Refreshing grains.
[DEBUG ] Reading configuration from /etc/salt/minion
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load dns
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load tldextract
[DEBUG ] The functions from module 'core' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'disks' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'extra' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'lvm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'mdadm' are being loaded by dir() on the loaded module
[DEBUG ] The functions from module 'minion_process' are being loaded by dir() on the loaded module
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load napalm
[DEBUG ] The functions from module 'opts' are being loaded by dir() on the loaded module
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pythoncom
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load win32com
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load win32api
[DEBUG ] Override __utils__: <module 'salt.loaded.int.grains.zfs' from '/opt/saltstack/salt/run/salt/grains/zfs.py'>
[DEBUG ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Elapsed time getting FQDNs: 0.029502391815185547 seconds
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load _bootlocale
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load lsb_release
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] User provided environment variable 'LD_LIBRARY_PATH' with value '/opt/saltstack/salt/run' which is the value that PyInstaller set's. Removing it
[DEBUG ] Setting environment variable 'LD_LIBRARY_PATH' to an empty string
[DEBUG ] The functions from module 'zfs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded zfs.is_supported
[DEBUG ] Determining pillar cache
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded state.apply
[DEBUG ] The functions from module 'direct_call' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded direct_call.execute
[DEBUG ] Override __grains__: <module 'salt.loaded.int.module.grains' from '/opt/saltstack/salt/run/salt/modules/grains.py'>
[DEBUG ] The functions from module 'grains' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded grains.get
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pywintypes
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load immutables
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load singledispatch_helpers
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load backports_abc
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load backports
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load saltwinshell
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded saltutil.is_running
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load smbprotocol
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pypsexec
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load winrm
[DEBUG ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded config.get
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load git
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pygit2
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[DEBUG ] Updating s3fs fileserver cache
[DEBUG ] Refreshing buckets cache file
[DEBUG ] The functions from module 's3' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3.query
[DEBUG ] S3 Request: https://salt-config-676XXXXXXXX.s3.us-east-1.amazonaws.com/?marker=
[DEBUG ] S3 Headers::
[DEBUG ] Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=3a2b68ef747aacfda018c43cf2713fb1d5b244b0e3b0012427fb3d272deb2846
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load netrc
[DEBUG ] Starting new HTTPS connection (1): salt-config-676XXXXXXXX .s3.us-east-1.amazonaws.com:443
[DEBUG ] https://salt-config-676XXXXXXXXXXX.s3.us-east-1.amazonaws.com:443 "GET /?marker= HTTP/1.1" 200 None
[DEBUG ] S3 Response Status Code: 200
[DEBUG ] Writing buckets cache file
[INFO ] Syncing local cache from S3...
[INFO ] salt-config-676XXXXXXXX - : base.sls
[INFO ] salt-config-676XXXXXXXX - : top.sls
[INFO ] Sync local cache from S3 completed.
[DEBUG ] Updating roots fileserver cache
[DEBUG ] Gathering pillar data for state run
[DEBUG ] Determining pillar cache
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] Finished gathering pillar data for state run
[INFO ] Loading fresh modules for state activity
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[DEBUG ] In saltenv 'base', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/top.sls' to resolve 'salt://top.sls'
[DEBUG ] Fetching file from saltenv 'base', ** attempting ** 'salt://top.sls'
[DEBUG ] No dest file found
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[INFO ] Fetching file from saltenv 'base', ** done ** 'top.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/top.sls
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/top.sls' using 'jinja' renderer: 0.05261063575744629
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/top.sls:
base:
'*':
- base
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/top.sls' using 'yaml' renderer: 0.00021910667419433594
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[DEBUG ] In saltenv '', looking at rel_path 'top.sls' to resolve 'salt://top.sls'
[DEBUG ] In saltenv '', ** considering ** path '/var/cache/salt/minion/files/top.sls' to resolve 'salt://top.sls'
[DEBUG ] Fetching file from saltenv '', ** attempting ** 'salt://top.sls'
[DEBUG ] No dest file found
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[INFO ] Fetching file from saltenv '', ** done ** 'top.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/op.sls
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'jinja' renderer: 0.000751495361328125
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/op.sls:
base:
'*':
- base
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/op.sls' using 'yaml' renderer: 0.00015544891357421875
[DEBUG ] Section for saltenv 'base' in the '' saltenv's top file will be ignored, as the top_file_merging_strategy is set to 'same' and the saltenvs do not match
[DEBUG ] The functions from module 'confirm_top' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded confirm_top.confirm_top
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load seco
[DEBUG ] The functions from module 'compound_match' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded compound_match.match
[DEBUG ] compound_match: ip-10-0-0-233.ec2.internal ? *
[DEBUG ] The functions from module 'glob_match' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded glob_match.match
[DEBUG ] compound_match ip-10-0-0-233.ec2.internal ? "*" => "True"
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded saltutil.sync_all
[DEBUG ] Syncing all
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/clouds'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing clouds for environment 'base'
[INFO ] Loading cache from salt://_clouds, for base
[INFO ] Caching directory '_clouds/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_clouds'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/beacons'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing beacons for environment 'base'
[INFO ] Loading cache from salt://_beacons, for base
[INFO ] Caching directory '_beacons/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_beacons'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/modules'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing modules for environment 'base'
[INFO ] Loading cache from salt://_modules, for base
[INFO ] Caching directory '_modules/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_modules'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/states'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing states for environment 'base'
[INFO ] Loading cache from salt://_states, for base
[INFO ] Caching directory '_states/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_states'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/sdb'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing sdb for environment 'base'
[INFO ] Loading cache from salt://_sdb, for base
[INFO ] Caching directory '_sdb/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_sdb'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/grains'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing grains for environment 'base'
[INFO ] Loading cache from salt://_grains, for base
[INFO ] Caching directory '_grains/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_grains'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/renderers'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing renderers for environment 'base'
[INFO ] Loading cache from salt://_renderers, for base
[INFO ] Caching directory '_renderers/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_renderers'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/returners'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing returners for environment 'base'
[INFO ] Loading cache from salt://_returners, for base
[INFO ] Caching directory '_returners/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_returners'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/output'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing output for environment 'base'
[INFO ] Loading cache from salt://_output, for base
[INFO ] Caching directory '_output/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=0 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_output'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/utils'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing utils for environment 'base'
[INFO ] Loading cache from salt://_utils, for base
[INFO ] Caching directory '_utils/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_utils'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/log_handlers'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing log_handlers for environment 'base'
[INFO ] Loading cache from salt://_log_handlers, for base
[INFO ] Caching directory '_log_handlers/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_log_handlers'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/executors'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing executors for environment 'base'
[INFO ] Loading cache from salt://_executors, for base
[INFO ] Caching directory '_executors/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_executors'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/proxy'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing proxy for environment 'base'
[INFO ] Loading cache from salt://_proxy, for base
[INFO ] Caching directory '_proxy/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_proxy'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/engines'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing engines for environment 'base'
[INFO ] Loading cache from salt://_engines, for base
[INFO ] Caching directory '_engines/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_engines'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/thorium'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing thorium for environment 'base'
[INFO ] Loading cache from salt://_thorium, for base
[INFO ] Caching directory '_thorium/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_thorium'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/serializers'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing serializers for environment 'base'
[INFO ] Loading cache from salt://_serializers, for base
[INFO ] Caching directory '_serializers/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_serializers'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/matchers'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing matchers for environment 'base'
[INFO ] Loading cache from salt://_matchers, for base
[INFO ] Caching directory '_matchers/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_matchers'
[INFO ] Creating module dir '/var/cache/salt/minion/extmods/pillar'
[DEBUG ] The functions from module 's3fs' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded s3fs.envs
[DEBUG ] The functions from module 'roots' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded roots.envs
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load azure
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load hglib
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load pysvn
[DEBUG ] Could not LazyLoad s3fs.init: 's3fs.init' is not available.
[DEBUG ] Could not LazyLoad roots.init: 'roots.init' is not available.
[INFO ] Syncing pillar for environment 'base'
[INFO ] Loading cache from salt://_pillar, for base
[INFO ] Caching directory '_pillar/' for environment 'base'
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=1 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Local cache dir: '/var/cache/salt/minion/files/base/_pillar'
[DEBUG ] Refreshing modules...
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load site
[INFO ] Loading fresh modules for state activity
[DEBUG ] The functions from module 'jinja' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] The functions from module 'yaml' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load esky
[DEBUG ] The functions from module 'saltutil' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded saltutil.refresh_modules
[DEBUG ] The functions from module 'event' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded event.fire
[DEBUG ] SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_01eb525e9d_pub.ipc
[DEBUG ] SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_01eb525e9d_pull.ipc
[ERROR ] Unable to connect pusher: Stream is closed
Traceback (most recent call last):
File "salt/utils/event.py", line 427, in connect_pull
self.pusher.connect(timeout=timeout)
File "salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "salt/transport/ipc.py", line 345, in _connect
yield self.stream.connect(sock_addr)
File "salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
[DEBUG ] Reading buckets cache file
[DEBUG ] Returning file list from cache: age=2 cache_time=20 /var/cache/salt/minion/file_lists/roots/base.p
[DEBUG ] Reading buckets cache file
[DEBUG ] Reading buckets cache file
[DEBUG ] In saltenv 'base', looking at rel_path 'base.sls' to resolve 'salt://base.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/base.sls' to resolve 'salt://base.sls'
[DEBUG ] Fetching file from saltenv 'base', ** attempting ** 'salt://base.sls'
[DEBUG ] No dest file found
[DEBUG ] Reading buckets cache file
[DEBUG ] Closing IPCMessageClient instance
[DEBUG ] Reading buckets cache file
[INFO ] Fetching file from saltenv 'base', ** done ** 'base.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/base.sls
[DEBUG ] Jinja search path: ['/var/cache/salt/minion/files/base']
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/base.sls' using 'jinja' renderer: 0.0013020038604736328
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/base.sls:
/etc/host:
file.managed:
- replace: False
- user: root
- group: root
- mode: '0644'
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/base.sls' using 'yaml' renderer: 0.00031065940856933594
[DEBUG ] The functions from module 'config' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded config.option
[DEBUG ] The functions from module 'file' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded file.managed
[INFO ] Running state [/etc/host] at time 16:51:27.596731
[INFO ] Executing state file.managed for [/etc/host]
[DEBUG ] TiamatPipPathFinder.find_spec: First time trying to load mmap
[DEBUG ] The functions from module 'file' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded file.user_to_uid
[INFO ] File /etc/host exists with proper permissions. No changes made.
[INFO ] Completed state [/etc/host] at time 16:51:27.637698 (duration_in_ms=40.968)
[DEBUG ] File /var/cache/salt/minion/accumulator/133644863202016 does not exist, no need to cleanup
[DEBUG ] The functions from module 'state' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded state.check_result
[DEBUG ] The functions from module 'highstate' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded highstate.output
local:
----------
ID: /etc/host
Function: file.managed
Result: True
Comment: File /etc/host exists with proper permissions. No changes made.
Started: 16:51:27.596730
Duration: 40.968 ms
Changes:
Summary for local
------------
Succeeded: 1
Failed: 0
------------
Total states run: 1
Total run time: 40.968 ms
[DEBUG ] Restoring sys.argv to: ['/opt/saltstack/salt/run/run', 'state.apply', '-l', 'debug']
Expected behavior
Using only s3fs should be sufficient and not require any roots configured on the host. Using an s3fs only configuration would ensure that all configurations applied come directly from S3 instead using the workaround of pulling states/files from S3 to disk and then applying configurations that are read from disk.
Versions Report
<details><summary>
Salt Version:
Salt: 3005
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
Python: 3.9.13 (main, Aug 23 2022, 18:33:16)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: amzn 2022
locale: utf-8
machine: x86_64
release: 5.15.57-30.131.amzn2022.x86_64
system: Linux
version: Amazon Linux 2022
</summary>
</details>
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!
Description
Using only s3fs for a fileserver_backend does not work properly (cannot find top.sls) without the addition of roots to fileserver_backend and then pointing file_roots to the s3cache in /var/cache/salt/minion/s3cache/$S3_BUCKET.
Setup
Tested on a AWS t2.micro EC2 instance (IMDSv2) running AL2022 with 5.15.57-30.131.amzn2022.x86_64 kernel
Full non-working minion configuration (see error below)
Workaround required to get s3fs working
Steps to Reproduce the behavior
salt-call state.apply
orsalt-call --local state.apply
Both of these produce the following error (debug logs below for greater detail):
Full debug using
salt-call state.apply -l debug
Note that even though the above fails, the s3cache is still created and populated:
Debug output of workaround using fileserver_backend roots workaround
Expected behavior
Using only s3fs should be sufficient and not require any roots configured on the host. Using an s3fs only configuration would ensure that all configurations applied come directly from S3 instead using the workaround of pulling states/files from S3 to disk and then applying configurations that are read from disk.
Versions Report
Additional context
Workaround appears to work but this does not feel like the expected configuration. Also broached in the Salt Slack channel here:
https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1663115884126479
The text was updated successfully, but these errors were encountered: