Merge pull request #208 from stackhpc/backport/wallaby-collections

Backport Ansible collection support (Wallaby)

doc/source/contributor/development.rst

@@ -38,25 +38,27 @@ in `etc/kayobe/*.yml
 <https://opendev.org/openstack/kayobe/src/branch/master/etc/kayobe/>`__.
 A number of custom Jinja filters exist in `ansible/filter_plugins/*.py
 <https://opendev.org/openstack/kayobe/src/branch/master/ansible/filter_plugins>`__.
-Kayobe depends on roles hosted on Ansible Galaxy, and these and their version
-requirements are defined in `requirements.yml
+Kayobe depends on roles and collections hosted on Ansible Galaxy, and these and
+their version requirements are defined in `requirements.yml
 <https://opendev.org/openstack/kayobe/src/branch/master/requirements.yml>`__.
 
 Ansible Galaxy
 ==============
 
-Kayobe uses a number of Ansible roles hosted on Ansible Galaxy. The role
-dependencies are tracked in ``requirements.yml``, and specify required
-versions. The process for changing a Galaxy role is as follows:
-
-#. If required, develop changes for the role. This may be done outside of
-   Kayobe, or by modifying the role in place during development. If upstream
-   changes to the role have already been made, this step can be skipped.
-#. Commit changes to the role, typically via a Github pull request.
-#. Request that a tagged release of the role be made, or make one if you have
-   the necessary privileges.
-#. Ensure that automatic imports are configured for the role using e.g. a
-   TravisCI webhook notification, or perform a manual import of the role on
-   Ansible Galaxy.
+Kayobe uses a number of Ansible roles and collections hosted on Ansible Galaxy.
+The role dependencies are tracked in ``requirements.yml``, and specify required
+versions. The process for changing a Galaxy role or collection is as follows:
+
+#. If required, develop changes for the role or collection. This may be done
+   outside of Kayobe, or by modifying the code in place during development. If
+   upstream changes to the code have already been made, this step can be
+   skipped.
+#. Commit changes to the role or collection, typically via a Github pull
+   request.
+#. Request that a tagged release of the role or collection be made, or make one
+   if you have the necessary privileges.
+#. Ensure that automatic imports are configured for the repository using e.g. a
+   webhook notification, or perform a manual import of the role on Ansible
+   Galaxy.
 #. Modify the version in ``requirements.yml`` to match the new release of the
-   role.
+   role or collection.

doc/source/custom-ansible-playbooks.rst

@@ -75,23 +75,25 @@ These symlinks can even be committed to the kayobe-config Git repository.
 Ansible Galaxy
 --------------
 
-Ansible Galaxy provides a means for sharing Ansible roles.  Kayobe
-configuration may provide a Galaxy requirements file that defines roles to be
-installed from Galaxy.  These roles may then be used by custom playbooks.
+Ansible Galaxy provides a means for sharing Ansible roles and collections.
+Kayobe configuration may provide a Galaxy requirements file that defines roles
+and collections to be installed from Galaxy.  These roles and collections may
+then be used by custom playbooks.
 
-Galaxy role dependencies may be defined in
-``$KAYOBE_CONFIG_PATH/ansible/requirements.yml``.  These roles will be
-installed in ``$KAYOBE_CONFIG_PATH/ansible/roles/`` when bootstrapping the
-Ansible control host::
+Galaxy dependencies may be defined in
+``$KAYOBE_CONFIG_PATH/ansible/requirements.yml``.  These roles and collections
+will be installed in ``$KAYOBE_CONFIG_PATH/ansible/roles/`` and
+``$KAYOBE_CONFIG_PATH/ansible/collections`` when bootstrapping the Ansible
+control host::
 
     (kayobe) $ kayobe control host bootstrap
 
 And updated when upgrading the Ansible control host::
 
     (kayobe) $ kayobe control host upgrade
 
-Example
-=======
+Example: roles
+==============
 
 The following example adds a ``foo.yml`` playbook to a set of kayobe
 configuration.  The playbook uses a Galaxy role, ``bar.baz``.
@@ -116,7 +118,8 @@ Here is the playbook, ``ansible/foo.yml``::
 Here is the Galaxy requirements file, ``ansible/requirements.yml``::
 
     ---
-    - bar.baz
+    roles:
+      - bar.baz
 
 We should first install the Galaxy role dependencies, to download the
 ``bar.baz`` role::
@@ -127,6 +130,45 @@ Then, to run the ``foo.yml`` playbook::
 
     (kayobe) $ kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/foo.yml
 
+Example: collections
+====================
+
+The following example adds a ``foo.yml`` playbook to a set of kayobe
+configuration.  The playbook uses a role from a Galaxy collection,
+``bar.baz.qux``.
+
+Here is the kayobe configuration repository structure::
+
+    etc/kayobe/
+        ansible/
+            collections/
+            foo.yml
+            requirements.yml
+        bifrost.yml
+    ...
+
+Here is the playbook, ``ansible/foo.yml``::
+
+    ---
+    - hosts: controllers
+      roles:
+        - name: bar.baz.qux
+
+Here is the Galaxy requirements file, ``ansible/requirements.yml``::
+
+    ---
+    collections:
+      - bar.baz
+
+We should first install the Galaxy dependencies, to download the ``bar.baz``
+collection::
+
+    (kayobe) $ kayobe control host bootstrap
+
+Then, to run the ``foo.yml`` playbook::
+
+    (kayobe) $ kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/foo.yml
+
 .. _custom-playbooks-hooks:
 
 Hooks

kayobe/ansible.py

@@ -291,7 +291,7 @@ def config_dump(parsed_args, host=None, hosts=None, var_name=None,
 def install_galaxy_roles(parsed_args, force=False):
     """Install Ansible Galaxy role dependencies.
 
-    Installs dependencies specified in kayobe, and if present, in kayobe
+    Installs role dependencies specified in kayobe, and if present, in kayobe
     configuration.
 
     :param parsed_args: Parsed command line arguments.
@@ -300,7 +300,7 @@ def install_galaxy_roles(parsed_args, force=False):
     LOG.info("Installing galaxy role dependencies from kayobe")
     requirements = utils.get_data_files_path("requirements.yml")
     roles_destination = utils.get_data_files_path('ansible', 'roles')
-    utils.galaxy_install(requirements, roles_destination, force=force)
+    utils.galaxy_role_install(requirements, roles_destination, force=force)
 
     # Check for requirements in kayobe configuration.
     kc_reqs_path = os.path.join(parsed_args.config_path,
@@ -323,7 +323,49 @@ def install_galaxy_roles(parsed_args, force=False):
                                   (parsed_args.config_path, str(e)))
 
     # Install roles from kayobe-config.
-    utils.galaxy_install(kc_reqs_path, kc_roles_path, force=force)
+    utils.galaxy_role_install(kc_reqs_path, kc_roles_path, force=force)
+
+
+def install_galaxy_collections(parsed_args, force=False):
+    """Install Ansible Galaxy collection dependencies.
+
+    Installs collection dependencies specified in kayobe, and if present, in
+    kayobe configuration.
+
+    :param parsed_args: Parsed command line arguments.
+    :param force: Whether to force reinstallation of roles.
+    """
+    LOG.info("Installing galaxy collection dependencies from kayobe")
+    requirements = utils.get_data_files_path("requirements.yml")
+    collections_destination = utils.get_data_files_path('ansible',
+                                                        'collections')
+    utils.galaxy_collection_install(requirements, collections_destination,
+                                    force=force)
+
+    # Check for requirements in kayobe configuration.
+    kc_reqs_path = os.path.join(parsed_args.config_path,
+                                "ansible", "requirements.yml")
+    if not utils.is_readable_file(kc_reqs_path)["result"]:
+        LOG.info("Not installing galaxy collection dependencies from kayobe "
+                 "config - requirements.yml not present")
+        return
+
+    LOG.info("Installing galaxy collection dependencies from kayobe config")
+    # Ensure a collections directory exists in kayobe-config.
+    kc_collections_path = os.path.join(parsed_args.config_path,
+                                       "ansible", "collections")
+    try:
+        os.makedirs(kc_collections_path)
+    except OSError as e:
+        if e.errno != errno.EEXIST:
+            raise exception.Error("Failed to create directory "
+                                  "ansible/collections/ "
+                                  "in kayobe configuration at %s: %s" %
+                                  (parsed_args.config_path, str(e)))
+
+    # Install collections from kayobe-config.
+    utils.galaxy_collection_install(kc_reqs_path, kc_collections_path,
+                                    force=force)
 
 
 def prune_galaxy_roles(parsed_args):

kayobe/cli/commands.py

@@ -232,6 +232,7 @@ class ControlHostBootstrap(KayobeAnsibleMixin, KollaAnsibleMixin, VaultMixin,
     def take_action(self, parsed_args):
         self.app.LOG.debug("Bootstrapping Kayobe Ansible control host")
         ansible.install_galaxy_roles(parsed_args)
+        ansible.install_galaxy_collections(parsed_args)
         playbooks = _build_playbook_list("bootstrap")
         self.run_kayobe_playbooks(parsed_args, playbooks, ignore_limit=True)
 
@@ -271,8 +272,9 @@ def take_action(self, parsed_args):
         # Remove roles that are no longer used. Do this before installing new
         # ones, just in case a custom role dependency includes any.
         ansible.prune_galaxy_roles(parsed_args)
-        # Use force to upgrade roles.
+        # Use force to upgrade roles and collections.
         ansible.install_galaxy_roles(parsed_args, force=True)
+        ansible.install_galaxy_collections(parsed_args, force=True)
         playbooks = _build_playbook_list("bootstrap")
         self.run_kayobe_playbooks(parsed_args, playbooks, ignore_limit=True)
         playbooks = _build_playbook_list("kolla-ansible")

kayobe/tests/unit/cli/test_commands.py

@@ -35,18 +35,21 @@ def __init__(self):
 class TestCase(unittest.TestCase):
 
     @mock.patch.object(ansible, "install_galaxy_roles", autospec=True)
+    @mock.patch.object(ansible, "install_galaxy_collections", autospec=True)
     @mock.patch.object(ansible, "passwords_yml_exists", autospec=True)
     @mock.patch.object(commands.KayobeAnsibleMixin,
                        "run_kayobe_playbooks")
     def test_control_host_bootstrap(self, mock_run, mock_passwords,
-                                    mock_install):
+                                    mock_install_collections,
+                                    mock_install_roles):
         mock_passwords.return_value = False
         command = commands.ControlHostBootstrap(TestApp(), [])
         parser = command.get_parser("test")
         parsed_args = parser.parse_args([])
         result = command.run(parsed_args)
         self.assertEqual(0, result)
-        mock_install.assert_called_once_with(parsed_args)
+        mock_install_roles.assert_called_once_with(parsed_args)
+        mock_install_collections.assert_called_once_with(parsed_args)
         expected_calls = [
             mock.call(
                 mock.ANY,
@@ -63,20 +66,23 @@ def test_control_host_bootstrap(self, mock_run, mock_passwords,
         self.assertEqual(expected_calls, mock_run.call_args_list)
 
     @mock.patch.object(ansible, "install_galaxy_roles", autospec=True)
+    @mock.patch.object(ansible, "install_galaxy_collections", autospec=True)
     @mock.patch.object(ansible, "passwords_yml_exists", autospec=True)
     @mock.patch.object(commands.KayobeAnsibleMixin,
                        "run_kayobe_playbooks")
     @mock.patch.object(commands.KollaAnsibleMixin,
                        "run_kolla_ansible_overcloud")
     def test_control_host_bootstrap_with_passwords(
-            self, mock_kolla_run, mock_run, mock_passwords, mock_install):
+            self, mock_kolla_run, mock_run, mock_passwords,
+            mock_install_collections, mock_install_roles):
         mock_passwords.return_value = True
         command = commands.ControlHostBootstrap(TestApp(), [])
         parser = command.get_parser("test")
         parsed_args = parser.parse_args([])
         result = command.run(parsed_args)
         self.assertEqual(0, result)
-        mock_install.assert_called_once_with(parsed_args)
+        mock_install_roles.assert_called_once_with(parsed_args)
+        mock_install_collections.assert_called_once_with(parsed_args)
         expected_calls = [
             mock.call(
                 mock.ANY,
@@ -106,16 +112,21 @@ def test_control_host_bootstrap_with_passwords(
         self.assertEqual(expected_calls, mock_kolla_run.call_args_list)
 
     @mock.patch.object(ansible, "install_galaxy_roles", autospec=True)
+    @mock.patch.object(ansible, "install_galaxy_collections", autospec=True)
     @mock.patch.object(ansible, "prune_galaxy_roles", autospec=True)
     @mock.patch.object(commands.KayobeAnsibleMixin,
                        "run_kayobe_playbooks")
-    def test_control_host_upgrade(self, mock_run, mock_prune, mock_install):
+    def test_control_host_upgrade(self, mock_run, mock_prune,
+                                  mock_install_roles,
+                                  mock_install_collections):
         command = commands.ControlHostUpgrade(TestApp(), [])
         parser = command.get_parser("test")
         parsed_args = parser.parse_args([])
         result = command.run(parsed_args)
         self.assertEqual(0, result)
-        mock_install.assert_called_once_with(parsed_args, force=True)
+        mock_install_roles.assert_called_once_with(parsed_args, force=True)
+        mock_install_collections.assert_called_once_with(parsed_args,
+                                                         force=True)
         mock_prune.assert_called_once_with(parsed_args)
         expected_calls = [
             mock.call(