Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add multimodel tests using real data #856

Merged
merged 67 commits into from
Dec 7, 2020
Merged

Add multimodel tests using real data #856

merged 67 commits into from
Dec 7, 2020

Commits on Nov 4, 2020

  1. Fix doc-string

    @stefsmeets
    stefsmeets committed Nov 4, 2020
    Configuration menu
    Copy the full SHA
    7b3800c View commit details
    Browse the repository at this point in the history

  2. Add useful error message

    @stefsmeets
    stefsmeets committed Nov 4, 2020
    Configuration menu
    Copy the full SHA
    e62479f View commit details
    Browse the repository at this point in the history

  3. Add tests using real data using esmvaltool_sample_data 

    https://github.com/ESMValGroup/ESMValTool_sample_data
    @stefsmeets
    stefsmeets committed Nov 4, 2020
    Configuration menu
    Copy the full SHA
    f4938a1 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2020

  1. Fix documentation

    @stefsmeets
    stefsmeets committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    732b5ef View commit details
    Browse the repository at this point in the history

  2. Update tests

    @stefsmeets
    stefsmeets committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    0faa524 View commit details
    Browse the repository at this point in the history

  3. Rename system -> functional tests

    @stefsmeets
    stefsmeets committed Nov 5, 2020
    Configuration menu
    Copy the full SHA
    0541c1e View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2020

  1. Add mark to functional tests using real data 

    i.e. `pytest -m functional`
    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    56303ad View commit details
    Browse the repository at this point in the history

  2. Add regression test

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    6e2cda9 View commit details
    Browse the repository at this point in the history

  3. Add regression data

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    61876f0 View commit details
    Browse the repository at this point in the history

  4. Test different variations of the dataset

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    eadf819 View commit details
    Browse the repository at this point in the history

  5. Add sample data git link to setup.py for develop installs

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    1c8dbb2 View commit details
    Browse the repository at this point in the history

  6. Move sample data dependency to 'test'

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    fec6967 View commit details
    Browse the repository at this point in the history

  7. Avoid Flake8 fail

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    282cb7e View commit details
    Browse the repository at this point in the history

  8. Add documentation

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    37028ef View commit details
    Browse the repository at this point in the history

  9. Add the missing requests dependency

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    578336a View commit details
    Browse the repository at this point in the history

  10. Fix conda build

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    19dbf1f View commit details
    Browse the repository at this point in the history

  11. Merge branch 'master' into multimodel_tests

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    91bb8c4 View commit details
    Browse the repository at this point in the history

  12. Remove pytest step in conda build

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    a95d02d View commit details
    Browse the repository at this point in the history

  13. Ignore functional tests for conda build

    @stefsmeets
    stefsmeets committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    d1ad030 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2020

  1. Skip tests if sample data repo cannot be imported

    @stefsmeets
    stefsmeets committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    7505bfa View commit details
    Browse the repository at this point in the history

  2. Rely on pytest.importskip to ignore functional tests

    @stefsmeets
    stefsmeets committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    0dd7253 View commit details
    Browse the repository at this point in the history

  3. Update developer documentation

    @stefsmeets
    stefsmeets committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    d8bf204 View commit details
    Browse the repository at this point in the history

  4. Update regression data

    @stefsmeets
    stefsmeets committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    5cb0f3d View commit details
    Browse the repository at this point in the history

  5. Fix function names

    @stefsmeets
    stefsmeets committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    a91bb5a View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2020

  1. Rewrite tests

    @stefsmeets
    stefsmeets committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    201ef5f View commit details
    Browse the repository at this point in the history

  2. Add debug comment

    @stefsmeets
    stefsmeets committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    61d2778 View commit details
    Browse the repository at this point in the history

  3. Rename variables

    @stefsmeets
    stefsmeets committed Nov 17, 2020
    Configuration menu
    Copy the full SHA
    03f3b06 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2020

  1. Update multimodel_statistics docstring to reflect current behaviour

    @stefsmeets
    stefsmeets committed Nov 18, 2020
    Configuration menu
    Copy the full SHA
    dd7e4dc View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2020

  1. Group daily data by calendar 

    Multimodel statistics is known to yield incorrect output with
mixed calendars
    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    09b0a53 View commit details
    Browse the repository at this point in the history

  2. Add extract time and caching to speed up tests

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    3309bdb View commit details
    Browse the repository at this point in the history

  3. Implement regression tests for grouped daily data

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    8f268f1 View commit details
    Browse the repository at this point in the history

  4. Rename _amon -> _month

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    6493565 View commit details
    Browse the repository at this point in the history

  5. Update tests with special cases

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    3f01861 View commit details
    Browse the repository at this point in the history

  6. Add special cases for daily tests

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    d2403bb View commit details
    Browse the repository at this point in the history

  7. Remove redundant tests 

    Most issues are already covered by the other tests
    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    71cf40c View commit details
    Browse the repository at this point in the history

  8. Tweak time slices to include entire last month

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    a327af3 View commit details
    Browse the repository at this point in the history

  9. Fix cache for daily data

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    8b67e27 View commit details
    Browse the repository at this point in the history

  10. Compare cube metadata for regression test

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    e9533fd View commit details
    Browse the repository at this point in the history

  11. Fix url sample data repo

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    270f33d View commit details
    Browse the repository at this point in the history

  12. Mark some tests as skip instead of xfail

    @stefsmeets
    stefsmeets committed Nov 27, 2020
    Configuration menu
    Copy the full SHA
    dfd108f View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2020

  1. Use pytest's caching mechanism for loading the data 

    This saves about 30-60 seconds. Cubes are not natively serializable
via json, so we must go via pickle and decode to a string object.
    @stefsmeets
    stefsmeets committed Nov 30, 2020
    Configuration menu
    Copy the full SHA
    472a4e9 View commit details
    Browse the repository at this point in the history

  2. Add regression tests for cube comparison

    @stefsmeets
    stefsmeets committed Nov 30, 2020
    Configuration menu
    Copy the full SHA
    0953eea View commit details
    Browse the repository at this point in the history

  3. Add issues numbers to xfail comments

    @stefsmeets
    stefsmeets committed Nov 30, 2020
    Configuration menu
    Copy the full SHA
    3789d26 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2020

  1. Add missing var_name when overlap == 'full'

    @stefsmeets
    stefsmeets committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    9589115 View commit details
    Browse the repository at this point in the history

  2. Move issue to reason in xfail marks

    @stefsmeets
    stefsmeets committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    c00c30c View commit details
    Browse the repository at this point in the history

  3. Add regression data

    @stefsmeets
    stefsmeets committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    6c60dcf View commit details
    Browse the repository at this point in the history

  4. Add documentation and fix linter errors

    @stefsmeets
    stefsmeets committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    eb69778 View commit details
    Browse the repository at this point in the history

  5. Fix documentation build

    @stefsmeets
    stefsmeets committed Dec 1, 2020
    Configuration menu
    Copy the full SHA
    c0b5443 View commit details
    Browse the repository at this point in the history

Commits on Dec 2, 2020

  1. Merge branch 'master' into multimodel_tests

    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    4ab8815 View commit details
    Browse the repository at this point in the history

  2. Split long repo string

    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    5449d4a View commit details
    Browse the repository at this point in the history

  3. Adjust array comparison 

    Use `almost_equal` to account for possible small errors, and check if
masks are equal first.
    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    9bc3481 View commit details
    Browse the repository at this point in the history

  4. Revert testing unmasked data only

    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    4118290 View commit details
    Browse the repository at this point in the history

  5. Update regression data

    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    7930cf7 View commit details
    Browse the repository at this point in the history

  6. Do not use assert_array_almost_equal for boolean masks

    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    bb8f327 View commit details
    Browse the repository at this point in the history

  7. Cache pytest cache 

    Loading sample data takes ~30 seconds. To save some time, store the
pytest cache for next time.
    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    a27362a View commit details
    Browse the repository at this point in the history

  8. Rename functional -> (use_)sample_data

    @stefsmeets
    stefsmeets committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    c065fb5 View commit details
    Browse the repository at this point in the history

  9. Add cache to test job

    @bouweandela
    bouweandela committed Dec 2, 2020
    Configuration menu
    Copy the full SHA
    20a3db1 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2020

  1. Add regression data

    @stefsmeets
    stefsmeets committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    72ace2f View commit details
    Browse the repository at this point in the history

  2. Update developer documentation

    @stefsmeets
    stefsmeets committed Dec 3, 2020
    Configuration menu
    Copy the full SHA
    4f80146 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2020

  1. Move tests to subdirectory

    @stefsmeets
    stefsmeets committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    9e04e85 View commit details
    Browse the repository at this point in the history

  2. Rename this/other to result/reference cube

    @stefsmeets
    stefsmeets committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    8c8e88b View commit details
    Browse the repository at this point in the history

  3. Use load_cube to load regression data

    @stefsmeets
    stefsmeets committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    66efb21 View commit details
    Browse the repository at this point in the history

  4. Pin dependency to revision

    @stefsmeets
    stefsmeets committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    e58d52d View commit details
    Browse the repository at this point in the history

  5. Update doc/contributing.rst 

    Co-authored-by: Bouwe Andela <[email protected]>
    @stefsmeets @bouweandela
    stefsmeets and bouweandela authored Dec 7, 2020
    Configuration menu
    Copy the full SHA
    6416018 View commit details
    Browse the repository at this point in the history

  6. Re-add regression data

    @stefsmeets
    stefsmeets committed Dec 7, 2020
    Configuration menu
    Copy the full SHA
    c953d8d View commit details
    Browse the repository at this point in the history

  7. Update setup.cfg 

    Co-authored-by: Bouwe Andela <[email protected]>
    @stefsmeets @bouweandela
    stefsmeets and bouweandela authored Dec 7, 2020
    Configuration menu
    Copy the full SHA
    84ebcc7 View commit details
    Browse the repository at this point in the history

  8. Pin to ESMValTool sample data v0.0.1

    @bouweandela
    bouweandela authored Dec 7, 2020
    Configuration menu
    Copy the full SHA
    adab2d2 View commit details
    Browse the repository at this point in the history