Merge remote-tracking branch 'upstream/master' into HEAD

* upstream/master: (47 commits) simplify rose stem tests (#314) Apply suggestions from code review add developer docs build(deps): bump pypa/gh-action-pypi-publish from 1.8.11 to 1.8.14 (#301) [skip ci] tests: remove defunct cli wrappers (#297) tests/functional: fix scoping issues setup: add missing pytest-asyncio test dependency install/reinstall: adapt to new async interfaces tests: fix flaky text fileinstall: remove asyncio logic Bump codecov/codecov-action from 3 to 4 (#295) use fqdn in rose-stem tests (#294) Update setup.cfg setup: bump rose version fix merge mistake fix flake8 fixed a test Apply suggestions from code review Bump actions/setup-python from 4 to 5 (#280) Update tests/functional/test_utils.py ...
cylc · May 2, 2024 · a18bb82 · a18bb82
2 parents 5073f5b + 69f321c
commit a18bb82
Show file tree

Hide file tree

Showing 38 changed files with 2,573 additions and 1,649 deletions.
diff --git a/.coveragerc b/.coveragerc
@@ -4,3 +4,20 @@ source=./cylc
 
 [report]
 precision=2
+
+exclude_lines =
+    pragma: no cover
+
+    # Don't complain if tests don't hit defensive assertion code:
+    raise NotImplementedError
+    return NotImplemented
+
+    # Ignore type checking code:
+    if (typing\.)?TYPE_CHECKING:
+    @overload( |$)
+
+    # Don't complain about ellipsis (exception classes, typing overloads etc):
+    \.\.\.
+
+    # Ignore abstract methods
+    @(abc\.)?abstractmethod
diff --git a/.github/workflows/1_create_release_pr.yml b/.github/workflows/1_create_release_pr.yml
@@ -26,7 +26,7 @@ jobs:
       uses: cylc/release-actions/stage-1/sanitize-inputs@v1
 
     - name: Checkout repo
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
       with:
         ref: ${{ env.BASE_REF }}
         fetch-depth: 0  # need to fetch all commits to check contributors
@@ -36,10 +36,9 @@ jobs:
       uses: cylc/release-actions/check-shortlog@v1
 
     - name: Setup Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
-        # return to 3.x once cylc-flow is compatible with 3.10+ (pyuv)
-        python-version: '3.9'
+        python-version: '3.x'
 
     - name: Create & checkout PR branch
       uses: cylc/release-actions/stage-1/checkout-pr-branch@v1

diff --git a/.github/workflows/2_auto_publish_release.yml b/.github/workflows/2_auto_publish_release.yml
@@ -21,15 +21,14 @@ jobs:
     steps:
 
     - name: Checkout repo
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
       with:
         ref: ${{ env.MERGE_SHA }}
 
     - name: Setup Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
-        # return to 3.x once cylc-flow is compatible with 3.10+ (pyuv)
-        python-version: '3.9'
+        python-version: '3.x'
 
     - name: Get the version number
       uses: cylc/release-actions/stage-2/get-version-from-pr@v1
@@ -38,7 +37,7 @@ jobs:
       uses: cylc/release-actions/build-python-package@v1
 
     - name: Publish distribution to PyPI
-      uses: pypa/[email protected].8
+      uses: pypa/[email protected].14
       with:
         user: __token__ # uses the API token feature of PyPI - least permissions possible
         password: ${{ secrets.PYPI_TOKEN }}

diff --git a/.github/workflows/shortlog.yml b/.github/workflows/shortlog.yml
@@ -13,7 +13,7 @@ jobs:
     timeout-minutes: 10
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           fetch-depth: 0  # need to fetch all commits to check contributors
 

diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -21,7 +21,7 @@ jobs:
     timeout-minutes: 5
     strategy:
       matrix:
-        python-version: ['3.7', '3.8', '3.9']
+        python-version: ['3.7', '3.8', '3.9', '3.10', '3.11']
       fail-fast: false
     env:
       PYTEST_ADDOPTS: --cov --cov-append --color=yes
@@ -30,7 +30,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Configure Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
@@ -97,7 +97,7 @@ jobs:
           coverage report
 
       - name: Upload coverage artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: coverage_py-${{ matrix.python-version }}
           path: coverage.xml
@@ -111,7 +111,7 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Configure Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: '3.9'
 
@@ -142,10 +142,10 @@ jobs:
         uses: actions/checkout@v4
 
       - name: Download coverage artifacts
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
 
       - name: Codecov upload
-        uses: codecov/codecov-action@v3
+        uses: codecov/codecov-action@v4
         with:
           name: ${{ github.workflow }}
           fail_ci_if_error: true

diff --git a/.github/workflows/update_copyright.yml b/.github/workflows/update_copyright.yml
@@ -11,7 +11,7 @@ jobs:
     timeout-minutes: 10
     steps:
     - name: Checkout repo
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     - name: Configure git
       uses: cylc/release-actions/configure-git@v1

diff --git a/CHANGES.md b/CHANGES.md
@@ -6,13 +6,24 @@ creating a new release entry be sure to copy & paste the span tag with the
 updated. Only the first match gets replaced, so it's fine to leave the old
 ones in. -->
 
+## __cylc-rose-1.4.0 (<span actions:bind='release-date'>Upcoming</span>)__
+
+### Features
+
+[#269](https://github.com/cylc/cylc-rose/pull/269) - Allow environment variables
+set in ``rose-suite.conf`` to be used when parsing ``global.cylc``.
+
+
 ## __cylc-rose-1.3.3 (<span actions:bind='release-date'>Released 2024-04-05</span>)__
 
+### Fixes
+
 [#302](https://github.com/cylc/cylc-rose/pull/302) -
 Fix issues which could cause "fcm_make" and "rose_prune" tasks intermittently
 fail with the message
 "Workflow database is incompatible with Cylc x.y.z, or is corrupted".
 
+
 ## __cylc-rose-1.3.2 (<span actions:bind='release-date'>Released 2024-01-18</span>)__
 
 [#284](https://github.com/cylc/cylc-rose/pull/284) - Allow use of Metomi-Rose 2.2.*.

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -44,6 +44,7 @@ below.
  - Mel Hall
  - Bruno Kinoshita
  - Hilary Oliver
+ - Jonny Williams
  - Mark Dawson
 <!-- end-shortlog -->
 

diff --git a/DEVELOPING.md b/DEVELOPING.md
@@ -0,0 +1,67 @@
+# Cylc Rose Dev Docs
+
+Cylc Rose provides integration between Cylc and Rose.
+
+Its functionality is defined in the
+[cylc-rose proposal](https://github.com/cylc/cylc-admin/blob/master/docs/proposal-cylc-rose-installing-rose-configs.md).
+
+## Cylc Rose Options
+
+### Rose Config Files
+
+Rose suite configurations are Cylc workflows which contain a `rose-suite.conf`
+file at the top level. This file configures:
+
+* Template variables.
+* File installation.
+* Environment variables for workflow configuration.
+
+Rose suite configurations may additionally contain optional configuration files
+in the `opts/` directory. Note that optional configurations may be turned on by
+default via the `opt` configuration in the `rose-suite.conf` file.
+
+### The Options
+
+The Rose suite configuration may be extended or overridden via the Cylc Rose
+CLI options.
+
+Cylc Rose adds three options to Cylc Flow.
+
+* `-S` - template variables
+* `-D` - any arbitrary rose configuration (includes template variables)
+* `-O` - optional configs
+
+These options are currently hardcoded in the cylc-flow source code as we
+do not yet have the ability for Cylc plugins (such as cylc-rose) to inject
+options into cylc-flow commands.
+
+There is also the `ROSE_SUITE_OPT_CONF_KEYS` environment variable which complements
+the `-O` option.
+
+### Storage
+
+Cylc Rose options passed in on the CLI are written to the
+`~/cylc-run/<workflow-id>/opt/rose-suite-cylc-install.conf` file by
+the `post_install` plugin.
+
+This preserves these options so that they are inherited by subsequent commands
+avoiding the need for users to remember which Cylc Rose options they have used
+and specify them with all future commands.
+
+### Reinstallation
+
+On reinstallation, additional Cylc Rose CLI options may be provided.
+
+If specified, these will override those previously specified on the CLI.
+
+### Option Lifecycle
+
+In order to support Cylc's compound commands (single commands that may perform
+multiple individual operations as a transaction), the `post_install` plugin
+deletes the Cylc Rose CLI options (from the object it was passed) after it has
+written them to the filesystem.
+
+This is necessary to avoid the same options being reenacted in later stages of
+compound commands. See https://github.com/cylc/cylc-flow/issues/5968
+
+![lifecycle-diagram](./etc/cylc-rose-opts.svg)
diff --git a/README.md b/README.md
@@ -56,6 +56,10 @@ This plugin provides support for the `rose-suite.conf` file, namely:
   the new Cylc `symlink dirs` functionality.
 * Graphical configuration editors.
 
+### How It Works
+
+For developer documentation, see [DEVELOPING](DEVELOPING.md).
+
 ### Contributing
 
 [![Contributors](https://img.shields.io/github/contributors/cylc/cylc-rose.svg?color=9cf)](https://github.com/cylc/cylc-rose/graphs/contributors)
@@ -81,7 +85,7 @@ pip install -e cylc-rose[all]
 
 [![License](https://img.shields.io/github/license/cylc/cylc-flow.svg?color=lightgrey)](https://github.com/cylc/cylc-flow/blob/master/COPYING)
 
-Copyright (C) 2008-<span actions:bind='current-year'>2023</span> NIWA &
+Copyright (C) 2008-<span actions:bind='current-year'>2024</span> NIWA &
 British Crown (Met Office) & Contributors.
 
 Cylc-rose is free software: you can redistribute it and/or modify it under the terms

diff --git a/cylc/rose/__init__.py b/cylc/rose/__init__.py
@@ -70,11 +70,37 @@
    for ease of porting Cylc 7 workflows.
 
 
+The ``global.cylc`` file
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Cylc Rose Plugin forces the reloading of the ``global.cylc`` file
+to allow environment variables set by Rose to change the global configuration.
+
+For example you could use ``CYLC_SYMLINKS`` as a variable to control
+the behaviour of ``cylc install``:
+
+.. code-block:: cylc
+
+   #!jinja2
+   # part of a global.cylc file
+   [install]
+      [[symlink dirs]]
+         [[[hpc]]]
+   {% if environ["CYLC_SYMLINKS"] | default("x") == "A" %}
+            run = $LOCATION_A
+   {% elif environ["CYLC_SYMLINKS"] | default("x") == "B" %}
+            run = $LOCATION_B
+   {% else %}
+            run = $LOCATION_C
+   {% endif %}
+
+
+
 Special Variables
 -----------------
 
 The Cylc Rose plugin provides two environment/template variables
-to the Cylc scheduler:
+to the Cylc scheduler.
 
 ``ROSE_ORIG_HOST``
    Cylc commands (such as ``cylc install``, ``cylc validate`` and
@@ -111,6 +137,7 @@
       ``CYLC_VERSION`` will be removed from your configuration by the
       Cylc-Rose plugin, as it is now set by Cylc.
 
+
 Additional CLI options
 ----------------------
 You can use command line options to set or override
@@ -205,4 +232,4 @@
 
 """
 
-__version__ = '1.3.4.dev'
+__version__ = '1.4.0.dev'