dependency upgrades #682

emanuel-schmid · 2023-03-29T10:37:18Z

Changes proposed in this PR:

This PR compiles changes that are not meant to change any functionality. Their only purpose is to make climada ready for the forthcoming upgrade of dependencies.

This PR fixes

PR Author Checklist

PR Reviewer Checklist

… in 1.8 too) see https://shapely.readthedocs.io/en/stable/migration.html

…>= 3.7 possibly >= 3.6, worked fine in 3.5

peanutfun · 2023-03-29T13:08:07Z

The version pins for the respective packages in env_climada.yml then have to be removed?

…A-project/climada_python into feature/conda_environment

Evelyn-M · 2023-03-31T15:38:29Z

What's the best way to go about checking if everything works? I guess installing climada from scratch with the updated env? And then just try out things that aren't covered by unit tests, like some tutorials, etc.? And things in petals, since they would also be affected by this?

I could imagine that especially shapely changes mainly affect our work routine around climada..

Evelyn-M · 2023-04-03T09:26:02Z

I was trying as said, with an updated climada_env.yml file.
However, I keep running into a conda error. Did I make a mistake in updating the requirements file?

Collecting package metadata (repodata.json): done
Solving environment: | 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                                                                                                                                                                                                             

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda_env/cli/main_create.py", line 111, in execute
        result[installer_type] = installer.install(prefix, pkg_specs, args, env)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda_env/installers/conda.py", line 32, in install
        prune=getattr(args, 'prune', False), update_modifier=UpdateModifier.FREEZE_INSTALLED)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 117, in solve_for_transaction
        should_retry_solve)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 158, in solve_for_diff
        force_remove, should_retry_solve)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 281, in solve_final_state
        ssc = self._run_sat(ssc)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/core/solve.py", line 808, in _run_sat
        should_retry_solve=ssc.should_retry_solve
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 1318, in solve
        self.find_conflicts(specs, specs_to_add, history_specs)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 347, in find_conflicts
        bad_deps = self.build_conflict_map(specs, specs_to_add, history_specs)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 507, in build_conflict_map
        root, search_node, dep_graph, num_occurances)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/resolve.py", line 369, in breadth_first_search_for_dep_graph
        last_spec = MatchSpec.union((path[-1], target_paths[-1][-1]))[0]
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 481, in union
        return cls.merge(match_specs, union=True)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 475, in merge
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 475, in <lambda>
        reduce(lambda x, y: x._merge(y, union), group) if len(group) > 1 else group[0]
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 502, in _merge
        final = this_component.union(that_component)
      File "/Users/evelynm/opt/anaconda3/lib/python3.7/site-packages/conda/models/match_spec.py", line 764, in union
        return '|'.join(options)
    TypeError: sequence item 0: expected str instance, Channel found

`$ /Users/evelynm/opt/anaconda3/bin/conda-env create -n climada_upgrade_env -f env_climada.yml`

  environment variables:
                 CIO_TEST=<not set>
  CONDA_AUTO_UPDATE_CONDA=false
        CONDA_DEFAULT_ENV=base
                CONDA_EXE=/Users/evelynm/opt/anaconda3/bin/conda
             CONDA_PREFIX=/Users/evelynm/opt/anaconda3
    CONDA_PROMPT_MODIFIER=(base)
         CONDA_PYTHON_EXE=/Users/evelynm/opt/anaconda3/bin/python
               CONDA_ROOT=/Users/evelynm/opt/anaconda3
              CONDA_SHLVL=1
                     PATH=/Users/evelynm/opt/anaconda3/bin:/Users/evelynm/opt/anaconda3/bin:/Use
                          rs/evelynm/opt/anaconda3/bin:/Users/evelynm/opt/anaconda3/condabin:/us
                          r/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/loc
                          al/munki
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : base
    active env location : /Users/evelynm/opt/anaconda3
            shell level : 1
       user config file : /Users/evelynm/.condarc
 populated config files : /Users/evelynm/.condarc
          conda version : 4.8.2
    conda-build version : 3.18.11
         python version : 3.7.6.final.0
       virtual packages : __osx=10.15.7
       base environment : /Users/evelynm/opt/anaconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/evelynm/opt/anaconda3/pkgs
                          /Users/evelynm/.conda/pkgs
       envs directories : /Users/evelynm/opt/anaconda3/envs
                          /Users/evelynm/.conda/envs
               platform : osx-64
             user-agent : conda/4.8.2 requests/2.24.0 CPython/3.7.6 Darwin/19.6.0 OSX/10.15.7
                UID:GID : 502:20
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

peanutfun · 2023-04-03T09:33:23Z

@Evelyn-M Your conda is still running on Python 3.7 and seems outdated. Before debugging the problem, try updating conda (should be conda update -n base conda), then repeat what you tried to do.

Evelyn-M · 2023-04-03T11:51:00Z

So shapely >= 2.0 and rasterio 1.3 were not installable for me in the same environment:

Could not solve for environment specs
The following packages are incompatible
├─ cartopy >=0.20.0,<0.20.3  is installable with the potential options
│  ├─ cartopy [0.20.0|0.20.1] would require
│  │  └─ geos >=3.9.1,<3.9.2.0a0 , which can be installed;
│  ├─ cartopy 0.20.0 would require
│  │  └─ proj >=8.0.1,<8.0.2.0a0 , which can be installed;
│  ├─ cartopy [0.20.1|0.20.2] would require
│  │  └─ proj >=8.2.0,<8.2.1.0a0 , which can be installed;
│  ├─ cartopy 0.20.1 would require
│  │  ├─ geos >=3.9.1,<3.9.2.0a0 , which can be installed;
│  │  └─ openssl >=1.1.1l,<1.1.2a , which can be installed;
│  ├─ cartopy 0.20.1 would require
│  │  └─ geos >=3.10.0,<3.10.1.0a0 , which can be installed;
│  ├─ cartopy 0.20.2 would require
│  │  └─ geos >=3.10.3,<3.10.4.0a0 , which can be installed;
│  ├─ cartopy 0.20.2 would require
│  │  └─ geos >=3.10.1,<3.10.2.0a0 , which can be installed;
│  └─ cartopy 0.20.2 would require
│     └─ geos >=3.10.2,<3.10.3.0a0 , which can be installed;
├─ rasterio 1.3  is uninstallable because it requires
│  ├─ libgdal >=3.5.0,<3.6.0a0  with the potential options
│  │  ├─ libgdal 3.5.0 would require
│  │  │  └─ geos >=3.10.3,<3.10.4.0a0 , which can be installed;
│  │  ├─ libgdal 3.5.0 would require
│  │  │  └─ geos >=3.10.2,<3.10.3.0a0 , which can be installed;
│  │  ├─ libgdal [3.5.0|3.5.1|3.5.2|3.5.3] would require
│  │  │  └─ geos >=3.11.0,<3.11.1.0a0 , which conflicts with any installable versions previously reported;
│  │  ├─ libgdal 3.5.3 would require
│  │  │  └─ geos >=3.11.1,<3.11.2.0a0 , which conflicts with any installable versions previously reported;
│  │  ├─ libgdal 3.5.3 would require
│  │  │  └─ openssl >=3.0.8,<4.0a0 , which conflicts with any installable versions previously reported;
│  │  └─ libgdal 3.5.3 would require
│  │     └─ openssl >=3.1.0,<4.0a0 , which conflicts with any installable versions previously reported;
│  └─ proj >=9.0.1,<9.0.2.0a0 , which conflicts with any installable versions previously reported;
└─ shapely >=2.0  is uninstallable because there are no viable options
   ├─ shapely [2.0.0|2.0.1] would require
   │  └─ geos >=3.11.1,<3.11.2.0a0 , which conflicts with any installable versions previously reported;
   └─ shapely 2.0.1 would require
      └─ geos >=3.11.2,<3.11.3.0a0 , which conflicts with any installable versions previously reported.

If I understand it correctly, the cartopy <0.20.3 is the issue, as it requires a geos version which is not installable with the other two mentioned packages?

Evelyn-M · 2023-04-03T12:57:22Z

So, for all hdf5 objects which are loaded during unittests, i get this sort of warning that concerns shapely 2.0:

2023-04-03 14:54:04,703 - climada.entity.exposures.base - INFO - Reading /Users/evelynm/climada/demo/data/exp_demo_today.h5
2023-04-03 14:54:04,703 - climada.entity.exposures.base - INFO - Reading /Users/evelynm/climada/demo/data/exp_demo_today.h5
/Users/evelynm/mambaforge/envs/climada_upgradenv/lib/python3.9/pickle.py:1717: UserWarning: Unpickling a shapely <2.0 geometry object. Please save the pickle again; shapely 2.1 will not have this compatibility.

code which uses the .get_map() method of matplotlib receives this deprecation warning now concerning matplotlib 3.7:

/MatplotlibDeprecationWarning: The get_cmap function was deprecated in Matplotlib 3.7 and will be removed two minor releases later. Use ``matplotlib.colormaps[name]`` or ``matplotlib.colormaps.get_cmap(obj)`` instead.

for instance, in

climada_python/climada/entity/measures/measure_set.py:325:   
def_color = plt.cm.get_cmap('Greys', len(meas_dict))

Apart from that, i found no issues so far with the dependency upgrades.

peanutfun · 2023-04-04T08:06:05Z

If I understand it correctly, the cartopy <0.20.3 is the issue, as it requires a geos version which is not installable with the other two mentioned packages?

@Evelyn-M The latest version of cartopy is 0.21. Try removing the cartopy version restriction as I mentioned here: #655 (comment)

Evelyn-M · 2023-04-04T09:18:45Z

@Evelyn-M The latest version of cartopy is 0.21. Try removing the cartopy version restriction as I mentioned here: #655 (comment)

I already did that in the last commit (1f0e6da)

emanuel-schmid · 2023-04-04T12:14:41Z

@Evelyn-M The easiest way to an upgraded environment is probably to create a new environment, install climada from conda-forge, downgrade eccodes to 2.26, then install climada again with pip install -e /path/to/climada. For now at least.

I'll take care of to update the climada_env file, so that the usual installation procedure works again.

peanutfun · 2023-04-05T08:42:28Z

climada/entity/exposures/litpop/litpop.py

@@ -731,7 +731,8 @@ def _get_litpop_single_polygon(polygon, reference_year, res_arcsec, data_dir,
                                                       global_origins=global_origins,
                                                       )
    except ValueError as err:
-        if "height must be > 0" in err.args[0] or "width must be > 0" in err.args[0]:
+        if ("height must be > 0" in str(err) or "width must be > 0" in str(err) # rasterio 1.2


We do not allow rasterio v1.2 anymore, so this case could be removed, right?

peanutfun · 2023-04-05T08:50:46Z

requirements/env_climada.yml

  - cfgrib>=0.9.7,<0.9.10  # 0.9.10 cannot read the icon_grib files from https://opendata.dwd.de
  - contextily>=1.0
  - dask>=2.25
  - gdal!=3.4.1  # has an issue with fiona on Windows (#403)
  - geopandas>=0.8
  - h5py>=2.10
  - haversine>=2.3
-  - matplotlib>=3.2,<3.6  # 3.6 requires cartopy 0.21 (s. https://github.com/SciTools/cartopy/pull/2054)
+  - matplotlib==3.7 


I think pinning a version is dangerous because we might get incompatible environments more quickly. What is the reason to pin to matplotlib v3.7?

I plan to set minimal version numbers for all dependencies according to the currently installed versions (union between Windows, Linux and MacOS) for the develop branch. Hence the env_climada.yml is bound to change!

For the releases (and for the Conda spec) I'm going to to completely fix the version numbers of all dependencies (but not the builds). The idea is to help with reproducibility and in the mean time avoid trouble shooting conda installations.
This will not succeed completely because the indirect dependencies still have some flexibility, but at least it's a start.

In setup.py on the other hand, I don't plan to change anything, because - if someone is able to install climada based on pip alone, they must be mighty savvy and probably have their own agenda. (E.g. Euler)

emanuel-schmid · 2023-04-12T15:29:29Z

Th PR is merged without review for now. Just to see the impact in Jenkins.

emanuel-schmid added 5 commits March 22, 2023 14:56

test_tc_tracks.test_to_geodataframe_line: adapt to shapely 2.0 (works…

fa75338

… in 1.8 too) see https://shapely.readthedocs.io/en/stable/migration.html

Merge branch 'develop' into feature/conda_environment

2966fe0

rasterio throws differrent error classes with different messages now

01128f6

Merge branch 'develop' into feature/conda_environment

b9fa959

all-NaN columns in sensitivity plot lead to failure as of matplotlib …

c057d7e

…>= 3.7 possibly >= 3.6, worked fine in 3.5

emanuel-schmid requested review from tovogt and Evelyn-M March 29, 2023 11:08

emanuel-schmid mentioned this pull request Mar 29, 2023

Make conda install climada the default installation instruction #673

Closed

emanuel-schmid requested review from ThomasRoosli and removed request for tovogt March 29, 2023 13:07

Evelyn-M added 2 commits March 31, 2023 17:34

update dependency versions

715c6ce

Merge branch 'feature/conda_environment' of https://github.com/CLIMAD…

0d15eae

…A-project/climada_python into feature/conda_environment

relax cartopy restrictions for env upgrade

1f0e6da

peanutfun reviewed Apr 5, 2023

View reviewed changes

emanuel-schmid added 2 commits April 12, 2023 13:30

Merge branch 'develop' into feature/conda_environment

d201d43

requirements: full dependency update

b934c76

emanuel-schmid merged commit 3c95127 into develop Apr 12, 2023

emanuel-schmid mentioned this pull request Apr 14, 2023

upgrade cartopy 0.20 -> 0.21 #693

Closed

emanuel-schmid deleted the feature/conda_environment branch April 27, 2023 14:46

This was referenced May 15, 2023

Upgrade shapely to v2.0.1 #655

Closed

upgrade shapely 1.8 -> 2.0 #684

Closed

upgrade matplotlib 3.5 -> 3.7 #685

Closed

upgrade rasterio 1.2 -> 1.3 #683

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dependency upgrades #682

dependency upgrades #682

emanuel-schmid commented Mar 29, 2023 •

edited

Loading

peanutfun commented Mar 29, 2023

Evelyn-M commented Mar 31, 2023 •

edited

Loading

Evelyn-M commented Apr 3, 2023

peanutfun commented Apr 3, 2023

Evelyn-M commented Apr 3, 2023 •

edited

Loading

Evelyn-M commented Apr 3, 2023 •

edited

Loading

peanutfun commented Apr 4, 2023

Evelyn-M commented Apr 4, 2023

emanuel-schmid commented Apr 4, 2023 •

edited

Loading

peanutfun Apr 5, 2023

peanutfun Apr 5, 2023

emanuel-schmid Apr 5, 2023

emanuel-schmid commented Apr 12, 2023

dependency upgrades #682

dependency upgrades #682

Conversation

emanuel-schmid commented Mar 29, 2023 • edited Loading

PR Author Checklist

PR Reviewer Checklist

peanutfun commented Mar 29, 2023

Evelyn-M commented Mar 31, 2023 • edited Loading

Evelyn-M commented Apr 3, 2023

peanutfun commented Apr 3, 2023

Evelyn-M commented Apr 3, 2023 • edited Loading

Evelyn-M commented Apr 3, 2023 • edited Loading

peanutfun commented Apr 4, 2023

Evelyn-M commented Apr 4, 2023

emanuel-schmid commented Apr 4, 2023 • edited Loading

peanutfun Apr 5, 2023

Choose a reason for hiding this comment

peanutfun Apr 5, 2023

Choose a reason for hiding this comment

emanuel-schmid Apr 5, 2023

Choose a reason for hiding this comment

emanuel-schmid commented Apr 12, 2023

emanuel-schmid commented Mar 29, 2023 •

edited

Loading

Evelyn-M commented Mar 31, 2023 •

edited

Loading

Evelyn-M commented Apr 3, 2023 •

edited

Loading

Evelyn-M commented Apr 3, 2023 •

edited

Loading

emanuel-schmid commented Apr 4, 2023 •

edited

Loading