roles.py: script to replace Ansible Galaxy #5

jakubgs · 2024-03-12T12:04:18Z

Usage:

usage: roles.py [-h] [-f FILTER] [-w WORKERS] [-r REQUIREMENTS] [-s ROLES_SYMLINK] [-l LOG_LEVEL] [-d] [-a] [-i | -c | -u]

This tool managed Ansible roles as Git repositories.
It is both faster and simpler than Ansible Galaxy.

By default ~/.ansible/roles is symlinked to ~/work.
Override it using --roles-symlink or ROLES_SYMLINK.

Installation behavior:
- If no version is specified newest is pulled.
- If version is matching nothing is done.
- If repo is dirty or detached nothing is done.
- If version is newer user is notified.

options:
  -h, --help            show this help message and exit
  -f FILTER, --filter FILTER
                        Filter role repo names.
  -w WORKERS, --workers WORKERS
                        Max workers to run in parallel.
  -r REQUIREMENTS, --requirements REQUIREMENTS
                        Location of requirements.yml file.
  -s ROLES_SYMLINK, --roles-symlink ROLES_SYMLINK
                        Actual location of installed roles.
  -l LOG_LEVEL, --log-level LOG_LEVEL
                        Logging level.
  -d, --fail-dirty      Fail if repo is dirty.
  -a, --fail-detached   Fail if repo has detached head.
  -i, --install         Clone and update required roles.
  -c, --check           Only check roles, no installing.
  -u, --update          Update requirements with current commits.

Examples:
./roles.py --install
./roles.py --check
./roles.py --update

Examples:

yakimant · 2024-05-22T12:48:27Z

❯ roles.py --check -d
...
Traceback (most recent call last):
  File "/Users/status/bin/roles.py", line 337, in <module>
    main()
  File "/Users/status/bin/roles.py", line 332, in main
    fail_states.append(State.DIRTY)
    ^^^^^^^^^^^^^^^^^^
AttributeError: 'set' object has no attribute 'append'

yakimant · 2024-05-22T12:49:38Z

❯ roles.py --check -f infra-role-bootstrap-linux
[WARNING] [infra-role-bootstrap-linux ] - exists(): state = EXISTS
[WARNING] [infra-role-bootstrap-linux ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-bootstrap-linux ] - is_dirty(): state = DIRTY
infra-role-bootstrap-linux     ---         DIRTY (Git: 1022d516 | Req: ANY)
Traceback (most recent call last):
  File "/Users/status/bin/roles.py", line 337, in <module>
    main()
  File "/Users/status/bin/roles.py", line 320, in main
    role = roles[req['name']]
           ~~~~~^^^^^^^^^^^^^
KeyError: 'infra-role-wireguard'

yakimant · 2024-05-22T12:51:37Z

❯ roles.py --check 2>&1 | grep infra-role-bootstrap-linux
[WARNING] [infra-role-bootstrap-linux ] - exists(): state = EXISTS
[WARNING] [infra-role-bootstrap-linux ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-bootstrap-linux ] - is_dirty(): state = DIRTY
infra-role-bootstrap-linux     ---         DIRTY (Git: 1022d516 | Req: ANY)

Is it WRONG_VERSION because required ANY?

yakimant · 2024-05-22T12:53:05Z

❯ roles.py --check
[WARNING] [infra-role-bootstrap-linux ] - exists(): state = EXISTS
[WARNING] [infra-role-wireguard       ] - exists(): state = EXISTS
[WARNING] [infra-role-open-ports      ] - exists(): state = EXISTS
[WARNING] [infra-role-swap-file       ] - exists(): state = EXISTS
[WARNING] [infra-role-consul-service  ] - exists(): state = EXISTS
[WARNING] [infra-role-systemd-timer   ] - exists(): state = EXISTS
[WARNING] [infra-role-nextcloud       ] - exists(): state = EXISTS
[WARNING] [infra-role-coturn          ] - exists(): state = EXISTS
[WARNING] [infra-role-open-ports      ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-bootstrap-linux ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-swap-file       ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-wireguard       ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-systemd-timer   ] - valid_version(): state = VALID
[WARNING] [infra-role-consul-service  ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-restic-backups  ] - exists(): state = EXISTS
[WARNING] [infra-role-nextcloud       ] - valid_version(): state = VALID
[WARNING] [infra-role-keycloak        ] - exists(): state = EXISTS
[WARNING] [infra-role-coturn          ] - valid_version(): state = VALID
[WARNING] [infra-role-postgres-ha     ] - exists(): state = EXISTS
[WARNING] [infra-role-open-ports      ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-nginx           ] - exists(): state = EXISTS
[WARNING] [infra-role-bootstrap-linux ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-nginx-metrics   ] - exists(): state = EXISTS
[WARNING] [infra-role-wireguard       ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-swap-file       ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-openkm          ] - exists(): state = EXISTS
[WARNING] [infra-role-jitsi           ] - exists(): state = EXISTS
[WARNING] [infra-role-consul-service  ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-keycloak        ] - valid_version(): state = VALID
[WARNING] [infra-role-restic-backups  ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-postgres-ha     ] - valid_version(): state = VALID
[WARNING] [infra-role-nginx-metrics   ] - valid_version(): state = VALID
[WARNING] [infra-role-nginx           ] - valid_version(): state = VALID
[WARNING] [infra-role-openkm          ] - valid_version(): state = VALID
[WARNING] [infra-role-restic-backups  ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-jitsi           ] - valid_version(): state = VALID
infra-role-bootstrap-linux     ---         DIRTY (Git: 1022d516 | Req: ANY)
infra-role-wireguard           ---         DIRTY (Git: 0653ac64 | Req: ANY)
infra-role-open-ports          ---         DIRTY (Git: ec9b3527 | Req: ANY)
infra-role-swap-file           ---         DIRTY (Git: ada5240a | Req: ANY)
infra-role-consul-service      ---         DIRTY (Git: 4d7c9c60 | Req: d6299306)
infra-role-systemd-timer       ---         VALID (Git: 50575b23 | Req: 50575b23)
infra-role-nextcloud           ---         VALID (Git: 4dfadb58 | Req: 4dfadb58)
infra-role-coturn              ---         VALID (Git: 64629f56 | Req: 64629f56)
infra-role-restic-backups      ---         DIRTY (Git: 789a9c25 | Req: af6b0d7c)
infra-role-keycloak            ---         VALID (Git: 2d2af930 | Req: 2d2af930)
infra-role-postgres-ha         ---         VALID (Git: 9fbe3851 | Req: 9fbe3851)
infra-role-nginx               ---         VALID (Git: 3043c998 | Req: 3043c998)
infra-role-nginx-metrics       ---         VALID (Git: fdc31d5b | Req: fdc31d5b)
infra-role-openkm              ---         VALID (Git: aa4ababf | Req: aa4ababf)
infra-role-jitsi               ---         VALID (Git: e22711d0 | Req: e22711d0)

For my taste only DIRTY is a warning, others are INFO if not DEBUG.

yakimant · 2024-05-22T12:55:28Z

-l seems to not to make an effect:

❯ roles.py --check -l ERROR
[WARNING] [infra-role-bootstrap-linux ] - exists(): state = EXISTS
...

yakimant · 2024-05-22T13:00:21Z

I've ran an install, but it doesn't look good:

❯ roles.py --install
[WARNING] [infra-role-bootstrap-linux ] - exists(): state = EXISTS
[WARNING] [infra-role-wireguard       ] - exists(): state = EXISTS
[WARNING] [infra-role-open-ports      ] - exists(): state = EXISTS
[WARNING] [infra-role-swap-file       ] - exists(): state = EXISTS
[WARNING] [infra-role-consul-service  ] - exists(): state = EXISTS
[WARNING] [infra-role-systemd-timer   ] - exists(): state = EXISTS
[WARNING] [infra-role-nextcloud       ] - exists(): state = EXISTS
[WARNING] [infra-role-coturn          ] - exists(): state = EXISTS
[WARNING] [infra-role-restic-backups  ] - exists(): state = EXISTS
[WARNING] [infra-role-keycloak        ] - exists(): state = EXISTS
[WARNING] [infra-role-swap-file       ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-consul-service  ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-bootstrap-linux ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-wireguard       ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-nextcloud       ] - valid_version(): state = VALID
[WARNING] [infra-role-postgres-ha     ] - exists(): state = EXISTS
[WARNING] [infra-role-restic-backups  ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-open-ports      ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-systemd-timer   ] - valid_version(): state = VALID
[WARNING] [infra-role-nginx           ] - exists(): state = EXISTS
[WARNING] [infra-role-coturn          ] - valid_version(): state = VALID
[WARNING] [infra-role-nginx-metrics   ] - exists(): state = EXISTS
[WARNING] [infra-role-openkm          ] - exists(): state = EXISTS
[WARNING] [infra-role-keycloak        ] - valid_version(): state = VALID
[WARNING] [infra-role-jitsi           ] - exists(): state = EXISTS
[WARNING] [infra-role-swap-file       ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-swap-file       ] - version(): state = NO_VERSION
[WARNING] [infra-role-restic-backups  ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-restic-backups  ] - version(): state = WRONG_VERSION
[WARNING] [infra-role-open-ports      ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-open-ports      ] - version(): state = NO_VERSION
[WARNING] [infra-role-wireguard       ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-wireguard       ] - version(): state = NO_VERSION
[WARNING] [infra-role-nginx           ] - valid_version(): state = VALID
[WARNING] [infra-role-postgres-ha     ] - valid_version(): state = VALID
[WARNING] [infra-role-bootstrap-linux ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-openkm          ] - valid_version(): state = VALID
[WARNING] [infra-role-consul-service  ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-consul-service  ] - version(): state = WRONG_VERSION
[WARNING] [infra-role-nginx-metrics   ] - valid_version(): state = VALID
[WARNING] [infra-role-jitsi           ] - valid_version(): state = VALID
[WARNING] [infra-role-restic-backups  ] - pull(): state = VALID
[WARNING] [infra-role-wireguard       ] - pull(): state = VALID
[WARNING] [infra-role-swap-file       ] - pull(): state = VALID
[WARNING] [infra-role-consul-service  ] - pull(): state = VALID
[WARNING] [infra-role-open-ports      ] - pull(): state = VALID
infra-role-bootstrap-linux     ---         DIRTY (Git: 1022d516 | Req: ANY)
infra-role-wireguard           ---         VALID (Git: 0653ac64 | Req: ANY)
infra-role-open-ports          ---         VALID (Git: ec9b3527 | Req: ANY)
infra-role-swap-file           ---         VALID (Git: ada5240a | Req: ANY)
infra-role-consul-service      ---         VALID (Git: 4d7c9c60 | Req: d6299306)
infra-role-systemd-timer       ---         VALID (Git: 50575b23 | Req: 50575b23)
infra-role-nextcloud           ---         VALID (Git: 4dfadb58 | Req: 4dfadb58)
infra-role-coturn              ---         VALID (Git: 64629f56 | Req: 64629f56)
infra-role-restic-backups      ---         VALID (Git: 789a9c25 | Req: af6b0d7c)
infra-role-keycloak            ---         VALID (Git: 2d2af930 | Req: 2d2af930)
infra-role-postgres-ha         ---         VALID (Git: 9fbe3851 | Req: 9fbe3851)
infra-role-nginx               ---         VALID (Git: 3043c998 | Req: 3043c998)
infra-role-nginx-metrics       ---         VALID (Git: fdc31d5b | Req: fdc31d5b)
infra-role-openkm              ---         VALID (Git: aa4ababf | Req: aa4ababf)
infra-role-jitsi               ---         VALID (Git: e22711d0 | Req: e22711d0)

❯ roles.py --check
[WARNING] [infra-role-bootstrap-linux ] - exists(): state = EXISTS
[WARNING] [infra-role-wireguard       ] - exists(): state = EXISTS
[WARNING] [infra-role-open-ports      ] - exists(): state = EXISTS
[WARNING] [infra-role-swap-file       ] - exists(): state = EXISTS
[WARNING] [infra-role-consul-service  ] - exists(): state = EXISTS
[WARNING] [infra-role-systemd-timer   ] - exists(): state = EXISTS
[WARNING] [infra-role-bootstrap-linux ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-nextcloud       ] - exists(): state = EXISTS
[WARNING] [infra-role-wireguard       ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-open-ports      ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-consul-service  ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-swap-file       ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-coturn          ] - exists(): state = EXISTS
[WARNING] [infra-role-restic-backups  ] - exists(): state = EXISTS
[WARNING] [infra-role-bootstrap-linux ] - is_dirty(): state = DIRTY
[WARNING] [infra-role-keycloak        ] - exists(): state = EXISTS
[WARNING] [infra-role-systemd-timer   ] - valid_version(): state = VALID
[WARNING] [infra-role-postgres-ha     ] - exists(): state = EXISTS
[WARNING] [infra-role-wireguard       ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-wireguard       ] - version(): state = NO_VERSION
[WARNING] [infra-role-nginx           ] - exists(): state = EXISTS
[WARNING] [infra-role-nextcloud       ] - valid_version(): state = VALID
[WARNING] [infra-role-nginx-metrics   ] - exists(): state = EXISTS
[WARNING] [infra-role-open-ports      ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-consul-service  ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-consul-service  ] - version(): state = WRONG_VERSION
[WARNING] [infra-role-open-ports      ] - version(): state = NO_VERSION
[WARNING] [infra-role-openkm          ] - exists(): state = EXISTS
[WARNING] [infra-role-jitsi           ] - exists(): state = EXISTS
[WARNING] [infra-role-keycloak        ] - valid_version(): state = VALID
[WARNING] [infra-role-nginx           ] - valid_version(): state = VALID
[WARNING] [infra-role-swap-file       ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-swap-file       ] - version(): state = NO_VERSION
[WARNING] [infra-role-postgres-ha     ] - valid_version(): state = VALID
[WARNING] [infra-role-coturn          ] - valid_version(): state = VALID
[WARNING] [infra-role-restic-backups  ] - valid_version(): state = WRONG_VERSION
[WARNING] [infra-role-nginx-metrics   ] - valid_version(): state = VALID
[WARNING] [infra-role-jitsi           ] - valid_version(): state = VALID
[WARNING] [infra-role-openkm          ] - valid_version(): state = VALID
[WARNING] [infra-role-restic-backups  ] - is_dirty(): state = WRONG_VERSION
[WARNING] [infra-role-restic-backups  ] - version(): state = WRONG_VERSION
infra-role-bootstrap-linux     ---         DIRTY (Git: 1022d516 | Req: ANY)
infra-role-wireguard           ---    NO_VERSION (Git: 0653ac64 | Req: ANY)
infra-role-open-ports          ---    NO_VERSION (Git: ec9b3527 | Req: ANY)
infra-role-swap-file           ---    NO_VERSION (Git: ada5240a | Req: ANY)
infra-role-consul-service      --- WRONG_VERSION (Git: 4d7c9c60 | Req: d6299306)
infra-role-systemd-timer       ---         VALID (Git: 50575b23 | Req: 50575b23)
infra-role-nextcloud           ---         VALID (Git: 4dfadb58 | Req: 4dfadb58)
infra-role-coturn              ---         VALID (Git: 64629f56 | Req: 64629f56)
infra-role-restic-backups      --- WRONG_VERSION (Git: 789a9c25 | Req: af6b0d7c)
infra-role-keycloak            ---         VALID (Git: 2d2af930 | Req: 2d2af930)
infra-role-postgres-ha         ---         VALID (Git: 9fbe3851 | Req: 9fbe3851)
infra-role-nginx               ---         VALID (Git: 3043c998 | Req: 3043c998)
infra-role-nginx-metrics       ---         VALID (Git: fdc31d5b | Req: fdc31d5b)
infra-role-openkm              ---         VALID (Git: aa4ababf | Req: aa4ababf)
infra-role-jitsi               ---         VALID (Git: e22711d0 | Req: e22711d0)

jakubgs · 2024-05-23T12:23:21Z

@yakimant in what doesn't it doesn't look good? Yes, the warnings are there for debugging, I will switch it to debug level, but other than that, did it work or not?

jakubgs · 2024-05-23T12:24:11Z

Just tested --install on my laptop after not using it for a while and it worked very well:

yakimant · 2024-05-27T10:04:04Z

@yakimant in what doesn't it doesn't look good? Yes, the warnings are there for debugging, I will switch it to debug level, but other than that, did it work or not?

Yes, I mean the log level should be lower, thanks!

yakimant · 2024-05-27T10:08:21Z

I will continue testing it by using insted of make requirements-install.
Will post more findings here.

PS I've put it to ~/bin and my PATH, changing in code:

29c27
< REQUIREMENTS_PATH = path.join(getcwd(), 'ansible/requirements.yml')
---
> REQUIREMENTS_PATH = path.join(SCRIPT_DIR, 'requirements.yml')

yakimant · 2024-05-27T14:43:29Z

Doesn't work with git + yubikey-fido ssh key + touch:

❯ roles.py --install
...
subprocess.CalledProcessError: Command '['git', 'remote', 'update']' returned non-zero exit status 1.

❯ git remote update
Confirm user presence for key ED25519-SK SHA256:uQnnmaZdFHsoVTXqn4gtPwt0pwMObyzzL71s2/fidO8
...

yakimant · 2024-05-27T14:48:36Z

galaxy was stuck here with yubikey blinking:

❯ make requirements-install
ansible-galaxy install --keep-scm-meta --ignore-errors --force -r ansible/requirements.yml
Starting galaxy role install process
- changing role infra-role-bootstrap-linux from  to unspecified

So I had to touch it for every role.

yakimant · 2024-05-27T14:51:39Z

When role is missing - check is crashing:

❯ roles.py --check
...
Traceback (most recent call last):
  File "/Users/status/bin/roles.py", line 337, in <module>
    main()
  File "/Users/status/bin/roles.py", line 323, in main
    BLUE(role.current_commit[:8]),
         ^^^^^^^^^^^^^^^^^^^
  File "/Users/status/bin/roles.py", line 157, in current_commit
    return self._git('rev-parse', 'HEAD')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/status/bin/roles.py", line 134, in _git
    rval = subprocess.run(
           ^^^^^^^^^^^^^^^
  File "/Users/status/.asdf/installs/python/3.11.9/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/status/.asdf/installs/python/3.11.9/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/status/.asdf/installs/python/3.11.9/lib/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/status/.ansible/roles/infra-role-swap-file'

jakubgs · 2024-06-06T17:34:19Z

I've fixed the issue in the last comment and a few others. It seems to be pretty stable, but I'd like to take a stab at fixing the issue with FIDO based auth with YubiKey.

yakimant · 2024-06-12T09:26:34Z

I reproduced the issue (kind of).

❯ roles.py -l DEBUG --install
[ERROR] Clone failed: Cloning into 'infra-role-consul-service'...
notify_start: exec(/opt/homebrew/Cellar/openssh/9.7p1/libexec/ssh-askpass): No such file or directory
sign_and_send_pubkey: signing failed for ED25519-SK "/Users/status/.ssh/id_ed25519_sk_touch": device not found
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/status/.asdf/installs/python/3.11.9/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/status/bin/roles.py", line 243, in handle_role
    role.pull()
  File "/Users/status/bin/roles.py", line 89, in wrapper_decorator
    rval = func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/status/bin/roles.py", line 188, in pull
    self._git('remote', 'update')
  File "/Users/status/bin/roles.py", line 144, in _git
    rval.check_returncode()
  File "/Users/status/.asdf/installs/python/3.11.9/lib/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['git', 'remote', 'update']' returned non-zero exit status 1.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/status/bin/roles.py", line 365, in <module>
    main()
  File "/Users/status/bin/roles.py", line 341, in main
    [r.result() for r in futures.as_completed(these_futures)]
  File "/Users/status/bin/roles.py", line 341, in <listcomp>
    [r.result() for r in futures.as_completed(these_futures)]
     ^^^^^^^^^^
  File "/Users/status/.asdf/installs/python/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/status/.asdf/installs/python/3.11.9/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
subprocess.CalledProcessError: Command '['git', 'remote', 'update']' returned non-zero exit status 1.

infra-role-consul-service was completely missing:

infra-role-consul-service      ---       MISSING (Git: ........ | Req: *)

This is a prompt for failing command:

❯ git remote update
Confirm user presence for key ED25519-SK SHA256:uQnnmaZdFHsoVTXqn4gtPwt0pwMObyzzL71s2/fidO8
<---- waiting for touch here

And that's for clone:

❯ git clone [email protected]:status-im/infra-role-consul-service.git
Cloning into 'infra-role-consul-service'...
Confirm user presence for key ED25519-SK SHA256:uQnnmaZdFHsoVTXqn4gtPwt0pwMObyzzL71s2/fidO8
<---- waiting for touch here
User presence confirmed
remote: Enumerating objects: 232, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (45/45), done.
remote: Total 232 (delta 41), reused 83 (delta 32), pack-reused 137
Receiving objects: 100% (232/232), 51.77 KiB | 563.00 KiB/s, done.
Resolving deltas: 100% (96/96), done.

yakimant · 2024-06-12T09:58:13Z

Workaround for the issues is to use 1 worker for ProcessPoolExecutor, with --workers 1
Then it waits for touch for each role.

yakimant · 2024-06-12T10:00:21Z

BTW, here is a discussion about not asking for touch on GitHub:
https://github.com/orgs/community/discussions/10593

Usage: ``` usage: roles.py [-h] [-f FILTER] [-w WORKERS] [-r REQUIREMENTS] [-s ROLES_SYMLINK] [-l LOG_LEVEL] [-d] [-a] [-i | -c | -u] This tool managed Ansible roles as Git repositories. It is both faster and simpler than Ansible Galaxy. By default ~/.ansible/roles is symlinked to ~/work. Override it using --roles-symlink or ROLES_SYMLINK. Installation behavior: - If no version is specified newest is pulled. - If version is matching nothing is done. - If repo is dirty or detached nothing is done. - If version is newer user is notified. options: -h, --help show this help message and exit -f FILTER, --filter FILTER Filter role repo names. -w WORKERS, --workers WORKERS Max workers to run in parallel. -r REQUIREMENTS, --requirements REQUIREMENTS Location of requirements.yml file. -s ROLES_SYMLINK, --roles-symlink ROLES_SYMLINK Actual location of installed roles. -l LOG_LEVEL, --log-level LOG_LEVEL Logging level. -d, --fail-dirty Fail if repo is dirty. -a, --fail-detached Fail if repo has detached head. -i, --install Clone and update required roles. -c, --check Only check roles, no installing. -u, --update Update requirements with current commits. Examples: ./roles.py --install ./roles.py --check ./roles.py --update ``` Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs · 2024-06-13T15:13:45Z

A few extra fixes I had to apply:

infra-template#750e12d1 - ansible/roles.py: fix formating of role names
infra-template#c17561ec - ansible/roles.py: fix checking git ancestry
infra-template#6515c4b2 - ansible/roles.py: fix checking ancestry during update
infra-template#f31c778d - ansible/roles.py: avoid unnecessary git calls
infra-template#b2e70605 - ansible/roles.py: fix colors when called via ansible

status-im/infra-template#5 Signed-off-by: Jakub Sokołowski <[email protected]>

Introduced in: status-im/infra-template#5 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs requested review from yakimant and apentori March 12, 2024 12:04

jakubgs self-assigned this Mar 12, 2024

jakubgs force-pushed the add-roles-mgm-tool branch 2 times, most recently from d333e6c to f467db2 Compare March 12, 2024 12:22

jakubgs force-pushed the add-roles-mgm-tool branch 4 times, most recently from f7a2140 to 6e7cb7d Compare May 14, 2024 20:41

jakubgs marked this pull request as ready for review May 14, 2024 20:42

jakubgs force-pushed the add-roles-mgm-tool branch 2 times, most recently from 494c404 to cfb8e8c Compare May 15, 2024 08:49

jakubgs force-pushed the add-roles-mgm-tool branch from cfb8e8c to 8732aa8 Compare May 23, 2024 12:37

jakubgs force-pushed the add-roles-mgm-tool branch 4 times, most recently from 00ce0ae to 0877a85 Compare June 6, 2024 12:57

jakubgs force-pushed the add-roles-mgm-tool branch 2 times, most recently from 3e8cc3a to 476743a Compare June 6, 2024 13:17

jakubgs force-pushed the add-roles-mgm-tool branch from 476743a to c750d19 Compare June 11, 2024 06:34

jakubgs force-pushed the add-roles-mgm-tool branch 3 times, most recently from d8c8bf0 to 02e4dd5 Compare June 13, 2024 11:54

jakubgs force-pushed the add-roles-mgm-tool branch from 02e4dd5 to 49df6d5 Compare June 13, 2024 12:07

jakubgs merged commit 49df6d5 into master Jun 13, 2024

jakubgs deleted the add-roles-mgm-tool branch June 13, 2024 12:07

jakubgs added a commit to status-im/infra-status that referenced this pull request Jun 13, 2024

ansible: add roles.py script to manage roles

9fce0e4

status-im/infra-template#5 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit to status-im/infra-waku that referenced this pull request Jun 13, 2024

ansible: add roles.py script to manage roles

2f89b7a

status-im/infra-template#5 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit to status-im/infra-nimbus that referenced this pull request Jun 13, 2024

ansible: add roles.py script to manage roles

eca6fc2

status-im/infra-template#5 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit to status-im/infra-office-legacy that referenced this pull request Jun 13, 2024

ansible: add roles.py script to manage roles

d6ca823

status-im/infra-template#5 Signed-off-by: Jakub Sokołowski <[email protected]>

jakubgs added a commit to status-im/infra-utils that referenced this pull request Jun 13, 2024

ansible/update.py: drop in favor of new roles.py

6fa2f46

Introduced in: status-im/infra-template#5 Signed-off-by: Jakub Sokołowski <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

roles.py: script to replace Ansible Galaxy #5

roles.py: script to replace Ansible Galaxy #5

jakubgs commented Mar 12, 2024 •

edited

Loading

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

jakubgs commented May 23, 2024

jakubgs commented May 23, 2024

yakimant commented May 27, 2024

yakimant commented May 27, 2024 •

edited

Loading

yakimant commented May 27, 2024

yakimant commented May 27, 2024

yakimant commented May 27, 2024

jakubgs commented Jun 6, 2024

yakimant commented Jun 12, 2024 •

edited

Loading

yakimant commented Jun 12, 2024 •

edited

Loading

yakimant commented Jun 12, 2024

jakubgs commented Jun 13, 2024

roles.py: script to replace Ansible Galaxy #5

roles.py: script to replace Ansible Galaxy #5

Conversation

jakubgs commented Mar 12, 2024 • edited Loading

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

yakimant commented May 22, 2024

jakubgs commented May 23, 2024

jakubgs commented May 23, 2024

yakimant commented May 27, 2024

yakimant commented May 27, 2024 • edited Loading

yakimant commented May 27, 2024

yakimant commented May 27, 2024

yakimant commented May 27, 2024

jakubgs commented Jun 6, 2024

yakimant commented Jun 12, 2024 • edited Loading

yakimant commented Jun 12, 2024 • edited Loading

yakimant commented Jun 12, 2024

jakubgs commented Jun 13, 2024

jakubgs commented Mar 12, 2024 •

edited

Loading

yakimant commented May 27, 2024 •

edited

Loading

yakimant commented Jun 12, 2024 •

edited

Loading

yakimant commented Jun 12, 2024 •

edited

Loading