Create multiple outputs for Cylc Flow package #11

kinow · 2020-09-30T23:14:13Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

Closes #6

First commit adds missing dependencies from cylc-flow master branch (aiofiles, pyuv). Second commit splits the recipe into multiple outputs. Namely:

cylc-flow-base, similar to jupyterhub-base, and matplotlib-base; this -base appears to be the common pattern for base conda recipes
cylc-flow-empty, imports cylc-flow-base, adds empy
cylc-flow-report-timings, imports cylc-flow-base, adds pandas
cylc-flow-main_loop, imports cylc-flow-base, adds pympler and matplotlib-base (this is not pip/SetupTools package, but the Conda matplotlib-base that doesn't include GUI libs like Qt)

The last item is a combination of all the requirements in the SetupTools dependency groups main_loop-log_data_store, main_loop-log_main_loop, and main_loop-log_memory. If others prefer, we can have one output for each dependency group in pip, but I thought it would be easier to maintain the conda recipe this way?

WIP as this would trigger a new release. Created PR because I wanted a break from the UI to reset my brain 🤓 and since this would be necessary anyway for the next release (or at least before the final 8 release)

conda-forge-linter · 2020-09-30T23:14:18Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the 'cylc-flow-empy' output doesn't have any tests.
It looks like the 'cylc-flow-report-timings' output doesn't have any tests.
It looks like the 'cylc-flow-main_loop' output doesn't have any tests.
License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

conda-forge-linter · 2020-09-30T23:15:51Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

conda-forge-linter · 2020-09-30T23:18:18Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

kinow · 2020-09-30T23:19:58Z

Pending:

before the release, bump up the version and build numbers
check if we have any other dependencies missing
thorough tests locally building the package to confirm the generated packages work as expected (both individually in an empty conda env, as well as with cylc-uiserver, cylc-ui, etc)
update installation instructions

kinow · 2021-03-30T01:09:46Z

I had forgotten about this PR 😥 Will try again before the next release.

hjoliver · 2021-03-30T01:17:41Z

So it's not ready to go in for the beta-0 release?

kinow · 2021-03-30T01:20:40Z

So it's not ready to go in for the beta-0 release?

No, I haven't finished testing it. I wrote how I thought it had to look like, but didn't test locally to compare if the produced conda files were correct, nor had time to test with the other packages and confirm it works. Sorry 😥

kinow · 2021-04-07T22:31:25Z

Rebased, and bumped build ID. Let's see what errors we have pending. Will test locally too.

kinow · 2021-04-08T00:08:10Z

Testing with conda build recipe/

Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {"ansimarkup[version='>=1.0.0']", "graphene[version='>=2.1,<3']", "jinja2[version='2.11.0,<2.12']", "rx[version='>=1.6,<2']", "urwid[version='>=2.0,<2.1']", "pyuv[version='>=1.4.0,<1.5.0']", "metomi-isodatetime[version='>=1!2.0.2,<1!2.1.0']", "protobuf[version='>=3.15.0,<3.16.0']"}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {"ansimarkup[version='>=1.0.0']", "graphene[version='>=2.1,<3']", "jinja2[version='2.11.0,<2.12']", "rx[version='>=1.6,<2']", "urwid[version='>=2.0,<2.1']", "pyuv[version='>=1.4.0,<1.5.0']", "metomi-isodatetime[version='>=1!2.0.2,<1!2.1.0']", "protobuf[version='>=3.15.0,<3.16.0']"}
WARNING conda_build.build:test(2859): failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {"ansimarkup[version='>=1.0.0']", "graphene[version='>=2.1,<3']", "jinja2[version='2.11.0,<2.12']", "rx[version='>=1.6,<2']", "urwid[version='>=2.0,<2.1']", "pyuv[version='>=1.4.0,<1.5.0']", "metomi-isodatetime[version='>=1!2.0.2,<1!2.1.0']", "protobuf[version='>=3.15.0,<3.16.0']"}
Tests failed for cylc-flow-base-8.0b0-py39_2.tar.bz2 - moving package to /home/kinow/Development/python/miniconda3/envs/delete/conda-bld/broken
WARNING:conda_build.build:Tests failed for cylc-flow-base-8.0b0-py39_2.tar.bz2 - moving package to /home/kinow/Development/python/miniconda3/envs/delete/conda-bld/broken
WARNING conda_build.build:tests_failed(2955): Tests failed for cylc-flow-base-8.0b0-py39_2.tar.bz2 - moving package to /home/kinow/Development/python/miniconda3/envs/delete/conda-bld/broken
TESTS FAILED: cylc-flow-base-8.0b0-py39_2.tar.bz2

:sighs:

kinow · 2021-04-08T23:08:49Z

Tested again today in the NIWA notebook. Took ages to build with conda build recipe (which gave me some time to read about another workflow manager and compare with cylc 😬 ).

Worked with no issues this time!

kinow · 2021-04-08T23:17:17Z

Except I cannot install any of the packages as I'm using Py 3.9 🤣

Let's create a new env with Py 3.8

kinow · 2021-04-09T02:04:47Z

Installed latest Miniconda, then created an env with py3.7.3. Built successfully.

# To have conda build upload to anaconda.org automatically, use
# conda config --set anaconda_upload yes
anaconda upload \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-base-8.0b0-py37_2.tar.bz2 \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-empy-8.0b0-py37_2.tar.bz2 \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-main_loop-8.0b0-py37_2.tar.bz2 \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-report-timings-8.0b0-py37_2.tar.bz2
anaconda_upload is not set.  Not uploading wheels: []

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "cylc-flow-base-8.0b0-py37_2": {
    "recipe": {}
  },
  "cylc-flow-empy-8.0b0-py37_2": {
    "recipe": {}
  },
  "cylc-flow-main_loop-8.0b0-py37_2": {
    "recipe": {}
  },
  "cylc-flow-report-timings-8.0b0-py37_2": {
    "recipe": {}
  }
}


####################################################################################
Resource usage summary:

Total time: 0:14:33.9
CPU usage: sys=0:00:00.3, user=0:00:01.6
Maximum memory usage observed: 49.2M
Total disk usage observed (not including envs): 2.7K

I only have the old Cylc in my $PATH.

$ cylc --version
7.8.6-7-gf3abe

Then installed cylc-flow conda install -c file:///home/kinow/miniconda3/envs/test/conda-bld/linux-64/ cylc-flow-base. And testing:

(test) kinow@kinow-VirtualBox:~/Development/python/workspace/cylc-flow-feedstock$ cylc --version
8.0b0
(test) kinow@kinow-VirtualBox:~/Development/python/workspace/cylc-flow-feedstock$ cd ~/cylc-suites/five
(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ cylc install
INSTALLED five from /home/kinow/cylc-suites/five -> /home/kinow/cylc-run/five/run6
(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ cylc play --no-detach five/run6
                                                                                    ._.                                                     
                                                                                    | |                                                     
                                                                        ._____._. ._| |_____.                                               
                                                                        | .___| | | | | .___|         The Cylc Workflow Engine [8.0b0]      
                                                                        | !___| !_! | | !___.           Copyright (C) 2008-2021 NIWA        
                                                                        !_____!___. |_!_____!   & British Crown (Met Office) & Contributors.
                                                                              .___! |                                                       
                                                                              !_____!                                                       
2021-04-09T02:02:43Z INFO - Cold Start 20130808T0000Z
2021-04-09T02:02:43Z INFO - Suite server: url=tcp://kinow-VirtualBox:43062/ pid=101707
2021-04-09T02:02:43Z INFO - Suite publisher: url=tcp://kinow-VirtualBox:43038
2021-04-09T02:02:43Z INFO - Run: (re)start=0 log=1
2021-04-09T02:02:43Z INFO - Cylc version: 8.0b0
2021-04-09T02:02:43Z INFO - Run mode: live
2021-04-09T02:02:43Z INFO - Initial point: 20130808T0000Z
2021-04-09T02:02:43Z INFO - Final point: None
2021-04-09T02:02:44Z INFO - [prep.20130808T0000Z] -submit-num=01, host=kinow-VirtualBox
2021-04-09T02:02:44Z INFO - [prep.20130808T0000Z] -triggered off []
^C2021-04-09T02:02:44Z ERROR -
	Traceback (most recent call last):
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/runners.py", line 43, in run
	    return loop.run_until_complete(main)
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
	    self.run_forever()
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
	    self._run_once()
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/base_events.py", line 1739, in _run_once
	    event_list = self._selector.select(timeout)
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/selectors.py", line 468, in select
	    fd_event_list = self._selector.poll(timeout, max_ev)
	KeyboardInterrupt
	During handling of the above exception, another exception occurred:
	Traceback (most recent call last):
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 580, in start_scheduler
	    await self.main_loop()
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 1491, in main_loop
	    await asyncio.sleep(duration)
	  File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/tasks.py", line 568, in sleep
	    return await future
	concurrent.futures._base.CancelledError
2021-04-09T02:02:44Z CRITICAL - Suite shutting down

Also confirmed I cannot import from EmPy.

(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ python -c 'import em'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'em'

Then installed cylc-flow-empy: conda install -c file:///home/kinow/miniconda3/envs/test/conda-bld/linux-64/ cylc-flow-empy

And voilà!

(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ python -c 'import em'
(test) kinow@kinow-VirtualBox:~/cylc-suites/five$

🚀 🎉

kinow · 2021-07-29T10:58:36Z

Just needs to be rebased either now or after next release, and then have @hjoliver or @oliver-sanders to use the generated artefacts and confirm I didn't miss anything. I think the number of outputs hasn't changed, but if we want more we just need to copy an existing output and modify accordingly here 👍

kinow self-assigned this Sep 30, 2020

oliver-sanders mentioned this pull request Apr 1, 2021

proposal: meta package cylc/cylc-flow#4158

Closed

8 tasks

kinow added 4 commits April 8, 2021 10:20

First match dependencies from cylc-flow master branch

edb7d54

Create multiple outputs for the cylc-flow recipe

b3ef9ce

Add test statements for each output

82d8a27

Bump up build number (see if this fixes the build)

40a734d

kinow force-pushed the outputs branch from f06db4b to 40a734d Compare April 7, 2021 22:29

kinow added 5 commits April 8, 2021 10:48

Move build and script to each output step

f98487e

Use $PYTHON (as in matplotlib build_base.sh script)

b00d24d

Include build section in cylc-flow-base

7e2c2d9

Use an external shell script

d9e6039

For empy, we test importing em and not empy

ee728dc

oliver-sanders mentioned this pull request Jul 29, 2021

8.0b2 release #26

Merged

5 tasks

kinow closed this Oct 4, 2021

kinow deleted the outputs branch October 4, 2021 21:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create multiple outputs for Cylc Flow package #11

Create multiple outputs for Cylc Flow package #11

kinow commented Sep 30, 2020 •

edited

Loading

conda-forge-linter commented Sep 30, 2020

conda-forge-linter commented Sep 30, 2020

conda-forge-linter commented Sep 30, 2020

kinow commented Sep 30, 2020 •

edited

Loading

kinow commented Mar 30, 2021 •

edited

Loading

hjoliver commented Mar 30, 2021

kinow commented Mar 30, 2021 •

edited

Loading

kinow commented Apr 7, 2021

kinow commented Apr 8, 2021

kinow commented Apr 8, 2021

kinow commented Apr 8, 2021

kinow commented Apr 9, 2021

kinow commented Jul 29, 2021

Create multiple outputs for Cylc Flow package #11

Create multiple outputs for Cylc Flow package #11

Conversation

kinow commented Sep 30, 2020 • edited Loading

conda-forge-linter commented Sep 30, 2020

conda-forge-linter commented Sep 30, 2020

conda-forge-linter commented Sep 30, 2020

kinow commented Sep 30, 2020 • edited Loading

kinow commented Mar 30, 2021 • edited Loading

hjoliver commented Mar 30, 2021

kinow commented Mar 30, 2021 • edited Loading

kinow commented Apr 7, 2021

kinow commented Apr 8, 2021

kinow commented Apr 8, 2021

kinow commented Apr 8, 2021

kinow commented Apr 9, 2021

kinow commented Jul 29, 2021

kinow commented Sep 30, 2020 •

edited

Loading

kinow commented Sep 30, 2020 •

edited

Loading

kinow commented Mar 30, 2021 •

edited

Loading

kinow commented Mar 30, 2021 •

edited

Loading