Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache CRDS_PATH reffile contents in CI workflows #7333

Merged
merged 5 commits into from
Nov 29, 2022

Conversation

jdavies-st
Copy link
Collaborator

@jdavies-st jdavies-st commented Nov 4, 2022

This PR re-enables caching of the CRDS_PATH for the CI workflows, after having been turned off in #6165. It should make the per-PR CI much more reliable (no failed reffile downloads) and slightly faster overall.

This takes advantage of @zacharyburnett 's excellent tox.ini refactor in PR #7323 (with some minor additions), and uses the new clean structure to distinguish which tox envs are running unit tests which use CRDS and which are not.

Here we use the CRDS_CONTEXT as the cache key. The context will change more often than the contents of CRDS_PATH itself, but that's pretty much the only way to do it that makes sense, as we are trying to cache the snapshot of reffiles that are pulled during the running of the unit tests, and thus need a proxy for the contents of this snapshot before the tests are run. We cannot hash the contents of CRDS_PATH as it doesn't exist until after the tests are run, but we need the hash before the tests are run. Chicken. Egg.

This means that every increment of the default CRDS_CONTEXT will mean that the first CI test run will not have a cached CRDS_PATH, but after it runs, it will create one, and then all subsequent runs will use it until the next CRDS_CONTEXT change.

I've tested this separately in the CI system by running one of the test matrix jobs, and results can be seen here:

https://github.com/jdavies-st/jwst/actions/workflows/ci.yml

particularly the "CI use caching" job numbers 89 through 92, where in 4 stages, respectively, the

  1. tests run without cache, cache is created for the first time upon test completion
  2. cache is successfully used the tests
  3. CRDS_CONTEXT is (artificially) changed, so no cache is restored or used, but a new one is made after the tests finish
  4. new cache is used on tests

I've previously also checked that running matrix works with the cache both when created for the first time and incremented with a new key.

Note:

  • Caching CRDS_PATH theoretically reduces runtime, but not by much. More importantly, it reduces the downloading of files from the crds server, which means the CI should be more reliable. So this PR means slightly reduced runtime for unit tests overall, but much more reliable, i.e. higher likelihood of finishing with out CRDS download errors causing problems.
  • This PR also caches the pip wheel cache for the actions/setup-python@v4 which saves having to download the likely packages from pypi. This works the same here as on one's local machine, which also maintains a pip cache of packages that have been pulled from pypi. The install step still queries pypi.org to get the latest version of a package dependency and will download it if not in the pip cache. This is all default pip behavior. This should speed up installs a bit and has no downsides.
  • We get the default CRDS_CONTEXT without having to separately install crds to do so by using the crds server json api. This speeds things up considerably.

@github-actions github-actions bot added automation Continuous Integration (CI) and testing automation tools testing labels Nov 4, 2022
@codecov
Copy link

codecov bot commented Nov 4, 2022

Codecov Report

Base: 79.63% // Head: 79.64% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (84cdbab) compared to base (fa17e4c).
Patch has no changes to coverable lines.

❗ Current head 84cdbab differs from pull request most recent head 4690040. Consider uploading reports for the commit 4690040 to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #7333   +/-   ##
=======================================
  Coverage   79.63%   79.64%           
=======================================
  Files         412      412           
  Lines       37572    37572           
=======================================
+ Hits        29922    29924    +2     
+ Misses       7650     7648    -2     
Flag Coverage Δ *Carryforward flag
nightly 79.62% <ø> (ø) Carriedforward from c66627f
unit 52.22% <ø> (-0.01%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files Coverage Δ
jwst/transforms/integration.py 92.85% <0.00%> (+7.14%) ⬆️
jwst/datamodels/integration.py 90.90% <0.00%> (+9.09%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@nden
Copy link
Collaborator

nden commented Nov 4, 2022

Zach will review when he's back from vacation.

Copy link
Collaborator

@zacharyburnett zacharyburnett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this LGTM! Are there any instances where we might want to explicitly run tests with a previous CRDS context with workflow_dispatch?

@jdavies-st
Copy link
Collaborator Author

jdavies-st commented Nov 16, 2022

Are there any instances where we might want to explicitly run tests with a previous CRDS context with workflow_dispatch?

Possibly! Maybe a separate PR for that? Here I'm mostly trying to make the CI fast and reliable.

Before this is merged, someone who has admin credentials for this repo will have to change which are the required tests to pass. They've been renamed, and rejiggered here, so for instance merging is blocked because the "Code style check" test hasn't passed yet, but it's there, just labeled "CI / check-style".

Feedback on the CI check names/descriptions would be appreciated. This PR changes them, hopefully in a more structured way. Is this an improvement for the daily developers or just another irritating change? We can go back to the old names.

Copy link
Collaborator

@hbushouse hbushouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea if this code will actually do what you claim it does, but I'm very much in favor of the claim!

python-version: 3.9
toxenv: security

# MacOS job is flaky on Github actions and fails routinely.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are MacOS jobs still flaky? Would be nice to be able to test both platforms.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are slower. Perhaps a token macos test with latest dependencies?

@hbushouse
Copy link
Collaborator

Before this is merged, someone who has admin credentials for this repo will have to change which are the required tests to pass. They've been renamed, and rejiggered here, so for instance merging is blocked because the "Code style check" test hasn't passed yet, but it's there, just labeled "CI / check-style".

Can you indicate which names have been changed, so that we can make sure to cover all of them when modifying the list of required passing tests?

Feedback on the CI check names/descriptions would be appreciated. This PR changes them, hopefully in a more structured way. Is this an improvement for the daily developers or just another irritating change? We can go back to the old names.

The new ones look OK to me.

@jdavies-st
Copy link
Collaborator Author

Can you indicate which names have been changed, so that we can make sure to cover all of them when modifying the list of required passing tests?

All of the following have changed:

Build distribution
Code style check
Latest dependency versions w/coverage
Oldest dependency versions
Python 3.10 Tests
SDP dependencies in requirements-sdp.txt
Security audit
Verify install_requires in setup.py

@hbushouse
Copy link
Collaborator

hbushouse commented Nov 17, 2022

After poking around in the page of required CI test settings, it appears that we may have to merge this PR before the new test names will show up and hence allow me to change the list. So any objections to merging this now? I have the power to merge this PR even though the (old) required tests are not passing. (it's good being king ...)

@zacharyburnett
Copy link
Collaborator

After poking around in the page of required CI test settings, it appears that we may have to merge this PR before the new test names will show up and hence allow me to change the list. So any objections to merging this now? I have the power to merge this PR even though the (old) required tests are not passing. (it's good being king ...)

I'm fine with that, though perhaps we should rebase first to see if the xdist-cov test will pass with new main code

@zacharyburnett
Copy link
Collaborator

zacharyburnett commented Nov 17, 2022

are these test failures important, or ephemeral? They seemed to appear in the test-oldestdeps-xdist-cov environment in the previous run, but now only occur in test-xdist-cov

metacls = <class 'enum.EnumType'>, cls = 'ListCategory'
bases = (<enum 'ListCategory'>,), classdict = {}, boundary = None
_simple = False, kwds = {}, ignore = ['_ignore_'], key = '_ignore_'

    def __new__(metacls, cls, bases, classdict, *, boundary=None, _simple=False, **kwds):
        # an Enum class is final once enumeration items have been defined; it
        # cannot be mixed with other types (int, float, etc.) if it has an
        # inherited __new__ unless a new __new__ is defined (or the resulting
        # class will fail).
        #
        if _simple:
            return super().__new__(metacls, cls, bases, classdict, **kwds)
        #
        # remove any keys listed in _ignore_
        classdict.setdefault('_ignore_', []).append('_ignore_')
        ignore = classdict['_ignore_']
        for key in ignore:
            classdict.pop(key, None)
        #
        # grab member names
>       member_names = classdict._member_names
E       AttributeError: 'dict' object has no attribute '_member_names'

https://github.com/spacetelescope/jwst/actions/runs/3489141823/jobs/5840043311#step:7:362

@hbushouse
Copy link
Collaborator

Probably need @stscieisenhamer to take a look.

@jdavies-st
Copy link
Collaborator Author

jdavies-st commented Nov 18, 2022

Odd error. It was not there before master was merged in. We were also not testing against Python 3.11 either.

It was introduced with #7332.

I don't understand jwst.associations, but the bug is here:

https://github.com/spacetelescope/jwst/actions/runs/3489141823/jobs/5840043311#step:7:338

rule = type(rule_name, (obj,), {})

where a ListCategory of type ListCategory, which subclasses Enum is being dynamically created. But you can't do this with Enums. Here's the minimally reproducible example:

>>> from enum import Enum
>>> type("foo", (Enum,), {})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jdavies/miniconda3/envs/jwst/lib/python3.11/enum.py", line 482, in __new__
    member_names = classdict._member_names
                   ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute '_member_names'

and instead you should do

>>> Enum("foo", {})
<enum 'foo'>

But maybe the bug is that ListCategory should not be there in the first place? All the other rules added are rules and constraints.

@jdavies-st
Copy link
Collaborator Author

I verified that changes introduced in #7332 fail under Python 3.11 but not under Python 3.10.8. The rules to the registry by the test are the same. So there must be some subtle changes in 3.11 vs 3.10, though I found this

https://stackoverflow.com/questions/69328274/enum-raises-attributeerror-dict-object-has-no-attribute-member-names

And that's from a year ago, well before 3.11 was released. 😕

@stscieisenhamer
Copy link
Collaborator

No doubt I am being over-clever, as usual. However, point of information: If installing jwst from master, all tests succeed under Python 3.11. It is only this PR that the errors appears. I can investigate further next week.

@hbushouse
Copy link
Collaborator

No doubt I am being over-clever, as usual. However, point of information: If installing jwst from master, all tests succeed under Python 3.11. It is only this PR that the errors appears. I can investigate further next week.

So ... something in this PR is causing the failures or what?

@jdavies-st
Copy link
Collaborator Author

jdavies-st commented Nov 21, 2022

I see the same bug on master right now under Python 3.11. So nothing related to this branch, other than this branch now tests on Python 3.11.

$ conda create -n jwst python=3.11 -c conda-forge -y
...
$ conda activate jwst
$ pip install -e .[test]
...
$ pytest jwst/associations/tests/test_asn_from_list.py 
============================================== test session starts ===============================================
platform darwin -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0
crds_context: jwst_1017.pmap
rootdir: /Users/jdavies/dev/jwst, configfile: setup.cfg
plugins: jwst-1.8.3.dev40+g5dcff8ac, asdf-2.13.0, requests-mock-1.10.0, doctestplus-0.12.1, cov-4.0.0, ci-watson-0.6.1, openfiles-0.5.0
collected 16 items                                                                                               

jwst/associations/tests/test_asn_from_list.py ...F.......FFF..                                             [100%]

==================================================== FAILURES ====================================================
____________________________________________ test_level2_from_cmdline ____________________________________________

tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-26/test_level2_from_cmdline0')

    def test_level2_from_cmdline(tmpdir):
        """Create a level2 association from the command line"""
        rule = 'DMSLevel2bBase'
        path = tmpdir.join('test_asn.json')
        inlist = ['a', 'b', 'c']
        args = [
            '-o', path.strpath,
            '-r', rule,
        ]
        args = args + inlist
>       Main(args)

jwst/associations/tests/test_asn_from_list.py:76: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
jwst/associations/asn_from_list.py:120: in __init__
    rule = AssociationRegistry(parsed.ruledefs, include_bases=True)[parsed.rule]
jwst/associations/registry.py:109: in __init__
    self.populate(
jwst/associations/registry.py:281: in populate
    self.add_rule(name, obj, global_constraints=global_constraints)
jwst/associations/registry.py:323: in add_rule
    rule = type(rule_name, (obj,), {})
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

metacls = <class 'enum.EnumType'>, cls = 'ListCategory', bases = (<enum 'ListCategory'>,), classdict = {}
boundary = None, _simple = False, kwds = {}, ignore = ['_ignore_'], key = '_ignore_'

    def __new__(metacls, cls, bases, classdict, *, boundary=None, _simple=False, **kwds):
        # an Enum class is final once enumeration items have been defined; it
        # cannot be mixed with other types (int, float, etc.) if it has an
        # inherited __new__ unless a new __new__ is defined (or the resulting
        # class will fail).
        #
        if _simple:
            return super().__new__(metacls, cls, bases, classdict, **kwds)
        #
        # remove any keys listed in _ignore_
        classdict.setdefault('_ignore_', []).append('_ignore_')
        ignore = classdict['_ignore_']
        for key in ignore:
            classdict.pop(key, None)
        #
        # grab member names
>       member_names = classdict._member_names
E       AttributeError: 'dict' object has no attribute '_member_names'

../../miniconda3/envs/jwst/lib/python3.11/enum.py:482: AttributeError
___________________________________________ test_cmdline_success[json] ___________________________________________

format = 'json'
tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-26/test_cmdline_success_json_0')

    @pytest.mark.parametrize(
        "format",
        ['json', 'yaml']
    )
    def test_cmdline_success(format, tmpdir):
        """Create Level3 associations in different formats"""
        path = tmpdir.join('test_asn.json')
        product_name = 'test_product'
        inlist = ['a', 'b', 'c']
        args = [
            '-o', path.strpath,
            '--product-name', product_name,
            '--format', format
        ]
        args = args + inlist
>       Main(args)

jwst/associations/tests/test_asn_from_list.py:209: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
jwst/associations/asn_from_list.py:120: in __init__
    rule = AssociationRegistry(parsed.ruledefs, include_bases=True)[parsed.rule]
jwst/associations/registry.py:109: in __init__
    self.populate(
jwst/associations/registry.py:281: in populate
    self.add_rule(name, obj, global_constraints=global_constraints)
jwst/associations/registry.py:323: in add_rule
    rule = type(rule_name, (obj,), {})
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

metacls = <class 'enum.EnumType'>, cls = 'ListCategory', bases = (<enum 'ListCategory'>,), classdict = {}
boundary = None, _simple = False, kwds = {}, ignore = ['_ignore_'], key = '_ignore_'

    def __new__(metacls, cls, bases, classdict, *, boundary=None, _simple=False, **kwds):
        # an Enum class is final once enumeration items have been defined; it
        # cannot be mixed with other types (int, float, etc.) if it has an
        # inherited __new__ unless a new __new__ is defined (or the resulting
        # class will fail).
        #
        if _simple:
            return super().__new__(metacls, cls, bases, classdict, **kwds)
        #
        # remove any keys listed in _ignore_
        classdict.setdefault('_ignore_', []).append('_ignore_')
        ignore = classdict['_ignore_']
        for key in ignore:
            classdict.pop(key, None)
        #
        # grab member names
>       member_names = classdict._member_names
E       AttributeError: 'dict' object has no attribute '_member_names'

../../miniconda3/envs/jwst/lib/python3.11/enum.py:482: AttributeError
___________________________________________ test_cmdline_success[yaml] ___________________________________________

format = 'yaml'
tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-26/test_cmdline_success_yaml_0')

    @pytest.mark.parametrize(
        "format",
        ['json', 'yaml']
    )
    def test_cmdline_success(format, tmpdir):
        """Create Level3 associations in different formats"""
        path = tmpdir.join('test_asn.json')
        product_name = 'test_product'
        inlist = ['a', 'b', 'c']
        args = [
            '-o', path.strpath,
            '--product-name', product_name,
            '--format', format
        ]
        args = args + inlist
>       Main(args)

jwst/associations/tests/test_asn_from_list.py:209: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
jwst/associations/asn_from_list.py:120: in __init__
    rule = AssociationRegistry(parsed.ruledefs, include_bases=True)[parsed.rule]
jwst/associations/registry.py:109: in __init__
    self.populate(
jwst/associations/registry.py:281: in populate
    self.add_rule(name, obj, global_constraints=global_constraints)
jwst/associations/registry.py:323: in add_rule
    rule = type(rule_name, (obj,), {})
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

metacls = <class 'enum.EnumType'>, cls = 'ListCategory', bases = (<enum 'ListCategory'>,), classdict = {}
boundary = None, _simple = False, kwds = {}, ignore = ['_ignore_'], key = '_ignore_'

    def __new__(metacls, cls, bases, classdict, *, boundary=None, _simple=False, **kwds):
        # an Enum class is final once enumeration items have been defined; it
        # cannot be mixed with other types (int, float, etc.) if it has an
        # inherited __new__ unless a new __new__ is defined (or the resulting
        # class will fail).
        #
        if _simple:
            return super().__new__(metacls, cls, bases, classdict, **kwds)
        #
        # remove any keys listed in _ignore_
        classdict.setdefault('_ignore_', []).append('_ignore_')
        ignore = classdict['_ignore_']
        for key in ignore:
            classdict.pop(key, None)
        #
        # grab member names
>       member_names = classdict._member_names
E       AttributeError: 'dict' object has no attribute '_member_names'

../../miniconda3/envs/jwst/lib/python3.11/enum.py:482: AttributeError
___________________________________________ test_cmdline_change_rules ____________________________________________

tmpdir = local('/private/var/folders/t1/md4315cx5kl3zjv8fkgmtvl586jmhc/T/pytest-of-jdavies/pytest-26/test_cmdline_change_rules0')

    def test_cmdline_change_rules(tmpdir):
        """Command line change the rule"""
        rule = 'Association'
        path = tmpdir.join('test_asn.json')
        inlist = ['a', 'b', 'c']
        args = [
            '-o', path.strpath,
            '-r', rule,
        ]
        args = args + inlist
>       Main(args)

jwst/associations/tests/test_asn_from_list.py:232: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
jwst/associations/asn_from_list.py:120: in __init__
    rule = AssociationRegistry(parsed.ruledefs, include_bases=True)[parsed.rule]
jwst/associations/registry.py:109: in __init__
    self.populate(
jwst/associations/registry.py:281: in populate
    self.add_rule(name, obj, global_constraints=global_constraints)
jwst/associations/registry.py:323: in add_rule
    rule = type(rule_name, (obj,), {})
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

metacls = <class 'enum.EnumType'>, cls = 'ListCategory', bases = (<enum 'ListCategory'>,), classdict = {}
boundary = None, _simple = False, kwds = {}, ignore = ['_ignore_'], key = '_ignore_'

    def __new__(metacls, cls, bases, classdict, *, boundary=None, _simple=False, **kwds):
        # an Enum class is final once enumeration items have been defined; it
        # cannot be mixed with other types (int, float, etc.) if it has an
        # inherited __new__ unless a new __new__ is defined (or the resulting
        # class will fail).
        #
        if _simple:
            return super().__new__(metacls, cls, bases, classdict, **kwds)
        #
        # remove any keys listed in _ignore_
        classdict.setdefault('_ignore_', []).append('_ignore_')
        ignore = classdict['_ignore_']
        for key in ignore:
            classdict.pop(key, None)
        #
        # grab member names
>       member_names = classdict._member_names
E       AttributeError: 'dict' object has no attribute '_member_names'

../../miniconda3/envs/jwst/lib/python3.11/enum.py:482: AttributeError
============================================ short test summary info =============================================
FAILED jwst/associations/tests/test_asn_from_list.py::test_level2_from_cmdline - AttributeError: 'dict' object has no attribute '_member_names'
FAILED jwst/associations/tests/test_asn_from_list.py::test_cmdline_success[json] - AttributeError: 'dict' object has no attribute '_member_names'
FAILED jwst/associations/tests/test_asn_from_list.py::test_cmdline_success[yaml] - AttributeError: 'dict' object has no attribute '_member_names'
FAILED jwst/associations/tests/test_asn_from_list.py::test_cmdline_change_rules - AttributeError: 'dict' object has no attribute '_member_names'
========================================== 4 failed, 12 passed in 3.07s ==========================================

@zacharyburnett
Copy link
Collaborator

Ok, then I think this PR should be merged now, and then another PR opened for fixes for Python 3.11 for those tests.

Alternatively, we could try and solve those issues here.

@stscieisenhamer
Copy link
Collaborator

I'm good with merging this now and investigating the related errors later. @zacharyburnett however you wish to handle the out-of-date issue and then merge is good.

@zacharyburnett zacharyburnett merged commit 478ced4 into spacetelescope:master Nov 29, 2022
@jdavies-st jdavies-st deleted the ci-use-caching branch November 29, 2022 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automation Continuous Integration (CI) and testing automation tools no-changelog-entry-needed testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants