s3_sync file_root parameter does not accept file path #689

pthall · 2021-08-13T23:16:22Z

Summary

Hello, the s3_sync module documentation for file_root parameter comment states:

File/directory path for synchronization.

To me, this suggests that an individual file can be specified, ex: /home/me/my_archive.tar.gz

When an individual file path is provided, no file is detected by the file globbing. Thus, nothing is uploaded:

TASK [ia-home-archive : upload to S3 Glacier] *****************************************************
ok: [127.0.0.1] => {
    "changed": false,
    "filelist_actionable": [],
    "filelist_initial": [],
    "filelist_local_etag": [],
    "filelist_s3": [],
    "filelist_typed": [],
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "bucket": "ia-home-archive",
            "cache_control": "",
            "debug_botocore_endpoint_logs": false,
            "delete": false,
            "ec2_url": null,
            "exclude": ".*",
            "file_change_strategy": "date_size",
            "file_root": "/home/me/my_archive.tar.gz",
            "include": "*",
            "key_prefix": "",
            "mime_map": null,
            "mode": "push",
            "permission": "private",
            "profile": "2fa",
            "region": "us-west-2",
            "retries": null,
            "security_token": null,
            "storage_class": "DEEP_ARCHIVE",
            "validate_certs": true
        }
    },
    "uploads": []
}

Once I changed the file_root parameter to /home/me and include parameter to my_archive.tar.gz, I got the desired result.

It seems the documentation could be revised to clarify that file_root can only be a directory, or the module could be further improved to support a file path in the file_root parameter.

Issue Type

Bug Report

Component Name

s3_sync

Ansible Version

$ ansible --version
ansible [core 2.11.3]
  config file = None
  configured module search path = ['/home/peter.hall/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/share/virtualenvs/aws/lib/python3.6/site-packages/ansible
  ansible collection location = /home/peter.hall/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/share/virtualenvs/aws/bin/ansible
  python version = 3.6.8 (default, Nov 16 2020, 16:55:22) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
  jinja version = 3.0.1
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
Collection                    Version
----------------------------- -------
amazon.aws                    1.5.0
ansible.netcommon             2.3.0
ansible.posix                 1.2.0
ansible.utils                 2.3.1
ansible.windows               1.7.2
arista.eos                    2.2.0
awx.awx                       19.2.2
azure.azcollection            1.8.0
check_point.mgmt              2.0.0
chocolatey.chocolatey         1.1.0
cisco.aci                     2.0.0
cisco.asa                     2.0.2
cisco.intersight              1.0.16
cisco.ios                     2.3.1
cisco.iosxr                   2.4.0
cisco.meraki                  2.4.2
cisco.mso                     1.2.0
cisco.nso                     1.0.3
cisco.nxos                    2.5.0
cisco.ucs                     1.6.0
cloudscale_ch.cloud           2.2.0
community.aws                 1.5.0
community.azure               1.0.0
community.crypto              1.8.0
community.digitalocean        1.8.0
community.docker              1.9.0
community.fortios             1.0.0
community.general             3.5.0
community.google              1.0.0
community.grafana             1.2.1
community.hashi_vault         1.3.2
community.hrobot              1.1.1
community.kubernetes          1.2.1
community.kubevirt            1.0.0
community.libvirt             1.0.2
community.mongodb             1.3.0
community.mysql               2.1.0
community.network             3.0.0
community.okd                 1.1.2
community.postgresql          1.4.0
community.proxysql            1.1.0
community.rabbitmq            1.1.0
community.routeros            1.2.0
community.skydive             1.0.0
community.sops                1.1.0
community.vmware              1.12.0
community.windows             1.6.0
community.zabbix              1.4.0
containers.podman             1.6.2
cyberark.conjur               1.1.0
cyberark.pas                  1.0.7
dellemc.enterprise_sonic      1.1.0
dellemc.openmanage            3.6.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.11.0
fortinet.fortimanager         2.1.3
fortinet.fortios              2.1.2
frr.frr                       1.0.3
gluster.gluster               1.0.1
google.cloud                  1.0.2
hetzner.hcloud                1.4.4
hpe.nimble                    1.1.3
ibm.qradar                    1.0.3
infinidat.infinibox           1.2.4
inspur.sm                     1.2.0
junipernetworks.junos         2.4.0
kubernetes.core               1.2.1
mellanox.onyx                 1.0.0
netapp.aws                    21.6.0
netapp.azure                  21.8.1
netapp.cloudmanager           21.9.0
netapp.elementsw              21.6.1
netapp.ontap                  21.9.0
netapp.um_info                21.7.0
netapp_eseries.santricity     1.2.13
netbox.netbox                 3.1.1
ngine_io.cloudstack           2.1.0
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.0
openstack.cloud               1.5.0
openvswitch.openvswitch       2.0.0
ovirt.ovirt                   1.5.4
purestorage.flasharray        1.10.0
purestorage.flashblade        1.6.0
sensu.sensu_go                1.11.1
servicenow.servicenow         1.0.6
splunk.es                     1.0.2
t_systems_mms.icinga_director 1.20.0
theforeman.foreman            2.1.2
vyos.vyos                     2.5.0
wti.remote                    1.0.1

AWS SDK versions

$ pip show boto boto3 botocore
Name: boto
Version: 2.49.0
Summary: Amazon Web Services Library
Home-page: https://github.com/boto/boto/
Author: Mitch Garnaat
Author-email: [email protected]
License: MIT
Location: /usr/local/share/virtualenvs/aws/lib/python3.6/site-packages
Requires:
Required-by:
---
Name: boto3
Version: 1.18.18
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/share/virtualenvs/aws/lib/python3.6/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.21.18
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /usr/local/share/virtualenvs/aws/lib/python3.6/site-packages
Requires: urllib3, jmespath, python-dateutil
Required-by: s3transfer, boto3

Configuration

$ ansible-config dump --only-changed
CALLBACKS_ENABLED(/home/peter.hall/ansible/ansible.cfg) = ['profile_tasks']
DEFAULT_HOST_LIST(/home/peter.hall/ansible/ansible.cfg) = ['/home/peter.hall/repos
DEFAULT_STDOUT_CALLBACK(/home/peter.hall/ansible/ansible.cfg) = debug
DEFAULT_TIMEOUT(/home/peter.hall/ansible/ansible.cfg) = 30

OS / Environment

CentOS 7

Steps to Reproduce

- name: upload to S3 Glacier
  community.aws.s3_sync:
    bucket:  home-archive
    file_root: /home/me/my_archive.tar.gz
    storage_class: DEEP_ARCHIVE
    permission: private
    profile: 2fa
    region: us-west-2

Expected Results

file specified by file_root: /home/me/my_archive.tar.gz is uploaded

Actual Results

File globbing does not detect any file.

TASK [ia-home-archive : upload to S3 Glacier] *****************************************************
task path: /home/peter.hall/repositories/do-ansible/roles/ia-home-archive/tasks/main_loop.yml:13
Friday 13 August 2021  15:45:07 -0700 (0:00:01.783)       0:00:10.643 *********
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: peter.hall
<127.0.0.1> EXEC /bin/sh -c 'echo ~peter.hall && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/peter.hall/.ansible/tmp `"&& mkdir "` echo /home/peter.hall/.ansible/tmp/ansible-tmp-1628894707.9170668-3601-171520442577912 `" && echo ansible-tmp-1628894707.9170668-3601-171520442577912="` echo /home/peter.hall/.ansible/tmp/ansible-tmp-1628894707.9170668-3601-171520442577912 `" ) && sleep 0'
Loading collection amazon.aws from /usr/local/share/virtualenvs/aws/lib/python3.6/site-packages/ansible_collections/amazon/aws
Including module_utils file ansible/__init__.py
Including module_utils file ansible/module_utils/__init__.py
Including module_utils file ansible/module_utils/_text.py
Including module_utils file ansible/module_utils/basic.py
Including module_utils file ansible/module_utils/common/_collections_compat.py
Including module_utils file ansible/module_utils/common/__init__.py
Including module_utils file ansible/module_utils/common/_json_compat.py
Including module_utils file ansible/module_utils/common/_utils.py
Including module_utils file ansible/module_utils/common/arg_spec.py
Including module_utils file ansible/module_utils/common/file.py
Including module_utils file ansible/module_utils/common/parameters.py
Including module_utils file ansible/module_utils/common/collections.py
Including module_utils file ansible/module_utils/common/process.py
Including module_utils file ansible/module_utils/common/sys_info.py
Including module_utils file ansible/module_utils/common/text/converters.py
Including module_utils file ansible/module_utils/common/text/__init__.py
Including module_utils file ansible/module_utils/common/text/formatters.py
Including module_utils file ansible/module_utils/common/validation.py
Including module_utils file ansible/module_utils/common/warnings.py
Including module_utils file ansible/module_utils/compat/selectors.py
Including module_utils file ansible/module_utils/compat/__init__.py
Including module_utils file ansible/module_utils/compat/_selectors2.py
Including module_utils file ansible/module_utils/compat/selinux.py
Including module_utils file ansible/module_utils/distro/__init__.py
Including module_utils file ansible/module_utils/distro/_distro.py
Including module_utils file ansible/module_utils/errors.py
Including module_utils file ansible/module_utils/parsing/convert_bool.py
Including module_utils file ansible/module_utils/parsing/__init__.py
Including module_utils file ansible/module_utils/pycompat24.py
Including module_utils file ansible/module_utils/six/__init__.py
Including module_utils file ansible_collections/amazon/aws/plugins/module_utils/core.py
Including module_utils file ansible/module_utils/common/dict_transformations.py
Including module_utils file ansible_collections/__init__.py
Including module_utils file ansible_collections/amazon/__init__.py
Including module_utils file ansible_collections/amazon/aws/__init__.py
Including module_utils file ansible_collections/amazon/aws/plugins/__init__.py
Including module_utils file ansible_collections/amazon/aws/plugins/module_utils/__init__.py
Including module_utils file ansible_collections/amazon/aws/plugins/module_utils/ec2.py
Including module_utils file ansible/module_utils/ansible_release.py
Including module_utils file ansible_collections/amazon/aws/plugins/module_utils/cloud.py
Using module file /usr/local/share/virtualenvs/aws/lib/python3.6/site-packages/ansible_collections/community/aws/plugins/modules/s3_sync.py
<127.0.0.1> PUT /home/peter.hall/.ansible/tmp/ansible-local-3507r12mon4r/tmpvuqi_l11 TO /home/peter.hall/.ansible/tmp/ansible-tmp-1628894707.9170668-3601-171520442577912/AnsiballZ_s3_sync.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/peter.hall/.ansible/tmp/ansible-tmp-1628894707.9170668-3601-171520442577912/ /home/peter.hall/.ansible/tmp/ansible-tmp-1628894707.9170668-3601-171520442577912/AnsiballZ_s3_sync.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/local/share/virtualenvs/aws/bin/python /home/peter.hall/.ansible/tmp/ansible-tmp-1628894707.9170668-3601-171520442577912/AnsiballZ_s3_sync.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/peter.hall/.ansible/tmp/ansible-tmp-1628894707.9170668-3601-171520442577912/ > /dev/null 2>&1 && sleep 0'
ok: [127.0.0.1] => {
    "changed": false,
    "filelist_actionable": [],
    "filelist_initial": [],
    "filelist_local_etag": [],
    "filelist_s3": [],
    "filelist_typed": [],
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "bucket": "ia-home-archive",
            "cache_control": "",
            "debug_botocore_endpoint_logs": false,
            "delete": false,
            "ec2_url": null,
            "exclude": ".*",
            "file_change_strategy": "date_size",
            "file_root": "/home/me/my_archive.tar.gz",
            "include": "*",
            "key_prefix": "",
            "mime_map": null,
            "mode": "push",
            "permission": "private",
            "profile": "2fa",
            "region": "us-west-2",
            "retries": null,
            "security_token": null,
            "storage_class": "DEEP_ARCHIVE",
            "validate_certs": true
        }
    },
    "uploads": []
}

Code of Conduct

I agree to follow the Ansible Code of Conduct

The text was updated successfully, but these errors were encountered:

ansibullbot · 2021-08-13T23:25:10Z

Files identified in the description:

[plugins/modules/s3_sync.py](https://github.com/['ansible-collections/amazon.aws', 'ansible-collections/community.aws', 'ansible-collections/community.vmware']/blob/main/plugins/modules/s3_sync.py)

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

ansibullbot · 2021-08-13T23:25:10Z

cc @jillr @markuman @s-hertel @tremble @wimnat
click here for bot help

markuman · 2021-08-18T12:36:46Z

@pthall Thanks for your report. I can confirm this bug/behaviour.

According to the documentation, file_root must also accept a single file.
The issue is, that it is using os.walk(file_root) only, which will return nothing for a single file.

A fix for that is possibly easy.
Just check os.path.isfile(fileroot) before and return the expected object for the single file.

@pthall do you have some time and energy to provide a PR for this issue?

pthall · 2021-08-19T21:10:11Z

Thanks @markuman, when I can set up a suitable test environment I will develop/test a patch for this.

markuman · 2021-08-20T05:07:14Z

@pthall it's already in progress :) #692
but we got many open issues. You can pick up one of your interest if you like.

s3_sync: file_root parameter - handle individual file path SUMMARY Added ability to handle individual file path and upload an individual file to s3 bucket. Fixes #689. ISSUE TYPE Bugfix Pull Request COMPONENT NAME s3_sync Reviewed-by: Markus Bergholz <[email protected]> Reviewed-by: None <None> Reviewed-by: Mark Chappell <None> Reviewed-by: None <None>

Fix deprecation warning scope SUMMARY Fix deprecation warning scope to be not displayed while terminating instances. Fixes ansible-collections#689 ISSUE TYPE Bugfix Pull Request COMPONENT NAME ec2_instance Reviewed-by: Joseph Torcasso <None> Reviewed-by: Alina Buzachis <None> Reviewed-by: Mandar Kulkarni <[email protected]>

ansibullbot added bug This issue/PR relates to a bug module module needs_triage plugins plugin (any type) python3 labels Aug 13, 2021

markuman added easyfix Good for new comers and easy to start with contribution good first issue and removed needs_triage labels Aug 18, 2021

mandar242 mentioned this issue Aug 19, 2021

s3_sync: file_root parameter - handle individual file path #692

Merged

abikouo added the has_pr label Aug 19, 2021

ansible-zuul bot closed this as completed in #692 Aug 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3_sync file_root parameter does not accept file path #689

s3_sync file_root parameter does not accept file path #689

pthall commented Aug 13, 2021

ansibullbot commented Aug 13, 2021

ansibullbot commented Aug 13, 2021

markuman commented Aug 18, 2021 •

edited

Loading

pthall commented Aug 19, 2021 •

edited

Loading

markuman commented Aug 20, 2021

s3_sync file_root parameter does not accept file path #689

s3_sync file_root parameter does not accept file path #689

Comments

pthall commented Aug 13, 2021

Summary

Issue Type

Component Name

Ansible Version

Collection Versions

AWS SDK versions

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

Code of Conduct

ansibullbot commented Aug 13, 2021

ansibullbot commented Aug 13, 2021

markuman commented Aug 18, 2021 • edited Loading

pthall commented Aug 19, 2021 • edited Loading

markuman commented Aug 20, 2021

markuman commented Aug 18, 2021 •

edited

Loading

pthall commented Aug 19, 2021 •

edited

Loading