Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

25 tests failing with climada version 5 and conda #946

Open
bguillod opened this issue Sep 11, 2024 · 11 comments
Open

25 tests failing with climada version 5 and conda #946

bguillod opened this issue Sep 11, 2024 · 11 comments

Comments

@bguillod
Copy link
Collaborator

Describe the bug
Following an update of conda, I have installed the climada environment (as a developper, i.e. following the advanced instructions). However, I used conda instead of mambda (however I have run conda config --set solver libmamba so behind the scene mamba is running), and I adjusted the instructions to pick python 3.11 instead of 3.9. The installation seems to work fine, but 25 unit tests fail (!). You will find the summary logs below.

To Reproduce
Steps to reproduce the behavior/error:

  1. conda update -n base -c conda-forge --all
  2. conda create -n climada_env "python=3.11.*"
  3. conda env update -n climada_env -f requirements/env_climada.yml
  4. conda activate climada_env
  5. python -m pip install -e ./
  6. python -m unittest climada.engine.test.test_impact (this already fails one test climada.engine.test.test_impact.TestRPmatrix.test_local_exceedance_imp_pass, exactly as pointed out in Rewrite TestRPmatrix::test_local_exceedance_imp_pass to not rely on loading data #915)
  7. Follow the rest of the installation guide but using conda instead of mambda, including climada_petals installation, etc.
  8. Run pytest . from within the climada_python folder (develop branch). There, a total of 25 unit tests fail.

Expected behavior
All tests should pass.

Climada Version: current develop branch (5.0.1.dev0)

System Information (please complete the following information):

  • Operating system and version: macOS 14.6.1, M1 Pro chip.
  • Python version: 3.11.6
  • Conda version: 24.7.1

Additional context
This does not seem to happen in docker.

Here is the short test summary info output:

================================================================= short test summary info ==================================================================
FAILED climada/engine/test/test_impact.py::TestRPmatrix::test_local_exceedance_imp_pass - AssertionError: 2916964966.388227 != 2916964966.388219 within 5 places (8.106231689453125e-06 difference)
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_BLM150_pass - KeyError: 'GDP_MD'
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_Liechtenstein_15_lit_pass - RuntimeError: Download failed. Please check the network connection and whether filenames are still valid.
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_Liechtenstein_30_pop_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_Monaco150_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_from_nightlight_intensity - FileNotFoundError: The file /Users/bguillod/climada/data/gpw-v4-population-count-rev11_2020_30_sec_tif/gpw_v4_population_count_rev11_2020_30_sec.tif co...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_from_population - FileNotFoundError: The file /Users/bguillod/climada/data/gpw-v4-population-count-rev11_2020_30_sec_tif/gpw_v4_population_count_rev11_2020_30_sec.tif co...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_from_shape_and_countries_zurich_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_from_shape_zurich_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_netherlands150_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_switzerland300_admin1_pc2016_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_switzerland300_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestLitPopExposure::test_switzerland30normPop_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestAdmin1::test_brandenburg - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestAdmin1::test_calc_admin1 - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestAdmin1::test_from_countries_calc_admin1_pass - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_litpop_integr.py::TestGPWPopulation::test_get_gpw_file_path_pass - FileNotFoundError: The file /Users/bguillod/climada/data/gpw-v4-population-count-rev11_2020_30_sec_tif/gpw_v4_population_count_rev11_2020_30_sec.tif co...
FAILED climada/test/test_litpop_integr.py::TestGPWPopulation::test_load_gpw_pop_shape_pass - FileNotFoundError: The file /Users/bguillod/climada/data/gpw-v4-population-count-rev11_2020_30_sec_tif/gpw_v4_population_count_rev11_2020_30_sec.tif co...
FAILED climada/test/test_nightlight.py::TestNightlight::test_check_nl_local_file_exists - AssertionError: 2 not greater than or equal to 3
FAILED climada/test/test_nightlight.py::TestNightlight::test_download_nl_files - RuntimeError: Download failed. Please check the network connection and whether filenames are still valid.
FAILED climada/test/test_nightlight.py::TestNightlight::test_load_nasa_nl_shape_single_tile - rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X of...
FAILED climada/test/test_nightlight.py::TestNightlight::test_read_bm_files - requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(3408213 bytes read, 18480168 more expected)', IncompleteRead(3408213 byte...
FAILED climada/util/test/test_finance.py::TestWBData::test_ne_gdp_aia_2012_pass - KeyError: 'GDP_MD'
FAILED script/jenkins/test_data_api.py::TestDataAvail::test_nasa_nl_pass - RuntimeError: Download failed. Please check the network connection and whether filenames are still valid.
FAILED script/jenkins/test_notebooks.py::NotebookTest::test_notebook - TypeError: chdir: path should be string, bytes, os.PathLike or integer, not NoneType
================================================ 25 failed, 762 passed, 52748 warnings in 722.76s (0:12:02) ================================================
@bguillod bguillod added the bug label Sep 11, 2024
@bguillod
Copy link
Collaborator Author

Running the tests also create the following files within the directory, which are not git-ignored:

  • climada/test/data/LitPop_150arcsec_LUX.hdf5
  • climada/test/data/LitPop_pop_150arcsec_AUT.hdf5
  • climada/test/data/river_flood_150arcsec_rcp26_AUT_2010_2030.hdf5
  • climada/test/data/test_haz.h5

This surprises me, so maybe it hints at some mis-configuration (?) - in any case I thought I'd report this here as well.

@emanuel-schmid
Copy link
Collaborator

@bguillod thanks for the report. It must be something, as we don't encounter this i.g. while running exactly the same installation steps.

To exclude an issue with the repo - what is the outupt of git show and git status ?

@bguillod
Copy link
Collaborator Author

bguillod commented Sep 11, 2024

It must be something, as we don't encounter this i.g. while running exactly the same installation steps.

Thanks @emanuel-schmid for your reply, and I agree on this point. Please note that one of my colleague encounters the exact same issue (also on MacOS with Apple Silicon Processor).

Output of git show:

commit 5f092abbbbdc50556d5ad48bbf7d34d70bf6140a
Author: Samuel Juhel <[email protected]>
Date:   Mon Sep 9 11:50:42 2024 +0200

    Adds subsection on pickle and refers to it when pickle is mentioned (#942)
    
    Co-authored-by: spjuhel <[email protected]>

diff --git a/doc/guide/Guide_Py_Performance.ipynb b/doc/guide/Guide_Py_Performance.ipynb
index 9b9a477e..db700f16 100644
--- a/doc/guide/Guide_Py_Performance.ipynb
+++ b/doc/guide/Guide_Py_Performance.ipynb
@@ -12,16 +12,17 @@
     "\n",
     "This guide covers the following recommendations:\n",
     "\n",
-    "⏲️ **Use profiling tools** to find and assess performance bottlenecks.  \n",
-    "🔁 **Replace for-loops** by built-in functions and efficient external implementations.    \n",
-    "📝 **Consider algorithmic performance**, not only implementation performance.    \n",
-    "🧊 **Get familiar with NumPy:** vectorized functions, slicing, masks and broadcasting.    \n",
-    "⚫ **Miscellaneous:** sparse arrays, Numba, parallelization, huge files (xarray), memory.    \n",
+    "⏲️ **Use profiling tools** to find and assess performance bottlenecks.\n",
+    "🔁 **Replace for-loops** by built-in functions and efficient external implementations.\n",
+    "📝 **Consider algorithmic performance**, not only implementation performance.\n",
+    "🧊 **Get familiar with NumPy:** vectorized functions, slicing, masks and broadcasting.\n",
+    "⚫ **Miscellaneous:** sparse arrays, Numba, parallelization, huge files (xarray), memory, pickle format.\n",
     "⚠️ **Don't over-optimize** at the expense of readability and usability."
    ]
   },
   {
    "cell_type": "markdown",
+   "id": "33f970c4",
    "metadata": {
     "slideshow": {
      "slide_type": "slide"
@@ -1095,6 +1096,23 @@
   },
   {
    "cell_type": "markdown",
+   "id": "0ca35e9e",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "(saving-with-pickle)=\n",
+    "### Using pickle to save python objects\n",
+    "`pickle` is the standard python library [serialization module](https://docs.python.org/3/library/pickle.html).\n",
+    "It has the nice feature of being able to save most of python objects (standard and user defined) using simple methods.\n",
+    "However, pickle is *transient*, i.e. it has limited Portability: Pickle files are specific to the Python environment they were created in. This means that Pickle files may not be compatible across different Python versions or environments, which can make it challenging to share data between systems. As such it should only be used for temporary storage and not for persistent one."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "de8dca2c",
    "metadata": {
     "slideshow": {
      "slide_type": "slide"
diff --git a/doc/tutorial/1_main_climada.ipynb b/doc/tutorial/1_main_climada.ipynb
index e97f2e41..730d5e5e 100644
--- a/doc/tutorial/1_main_climada.ipynb
+++ b/doc/tutorial/1_main_climada.ipynb
@@ -1082,7 +1082,7 @@
     "\n",
     ">**Exercise:** Plot the impacts of Hurricane Maria. To do this you'll need to set `save_mat=True` in the earlier `ImpactCalc.impact()`.\n",
     "\n",
-    "We can save our variables in pickle format using the `save` function and load them with `load`. This will save your results in the folder specified in the configuration file. The default folder is a `results` folder which is created in the current path (see default configuration file `climada/conf/defaults.conf`). However, we recommend to use CLIMADA's writers in `hdf5` or `csv` whenever possible."
+    "We recommend to use CLIMADA's writers in `hdf5` or `csv` whenever possible. It is also possible to save our variables in pickle format using the `save` function and load them with `load`. This will save your results in the folder specified in the configuration file. The default folder is a `results` folder which is created in the current path (see default configuration file `climada/conf/defaults.conf`). The pickle format has a [transient format](saving-with-pickle) and should be avoided when possible."
    ]
   },
   {
diff --git a/doc/tutorial/climada_entity_DiscRates.ipynb b/doc/tutorial/climada_entity_DiscRates.ipynb
index 4d9ff9c0..acb33de0 100644
--- a/doc/tutorial/climada_entity_DiscRates.ipynb
+++ b/doc/tutorial/climada_entity_DiscRates.ipynb
@@ -179,9 +179,10 @@
   },
   {
    "cell_type": "markdown",
+   "id": "37bf8fe8",
    "metadata": {},
    "source": [
-    "Pickle can always be used as well:"
+    "Pickle can always be used as well, but note that pickle has a [transient format](saving-with-pickle) and should be avoided when possible:"
    ]
   },
   {
diff --git a/doc/tutorial/climada_entity_Exposures.ipynb b/doc/tutorial/climada_entity_Exposures.ipynb
index bb032899..b5db1520 100644
--- a/doc/tutorial/climada_entity_Exposures.ipynb
+++ b/doc/tutorial/climada_entity_Exposures.ipynb
@@ -1757,9 +1757,10 @@
   },
   {
    "cell_type": "markdown",
+   "id": "5d078d09",
    "metadata": {},
    "source": [
-    "Finallly, as with any Python object, use climada's save option to save it in pickle format."
+    "Finally, as with any Python object, use climada's save option to save it in pickle format. Note however, that pickle has a transient format and should be avoided when possible."
    ]
   },
   {
diff --git a/doc/tutorial/climada_entity_ImpactFuncSet.ipynb b/doc/tutorial/climada_entity_ImpactFuncSet.ipynb
index 6267b402..2702aa60 100644
--- a/doc/tutorial/climada_entity_ImpactFuncSet.ipynb
+++ b/doc/tutorial/climada_entity_ImpactFuncSet.ipynb
@@ -492,7 +492,7 @@
    "source": [
     "#### Alternative saving format\n",
     "\n",
-    "Alternatively, users may also save the impact functions into [pickle format](https://docs.python.org/3/library/pickle.html), using CLIMADA in-built function `save()`."
+    "Alternatively, users may also save the impact functions into [pickle format](https://docs.python.org/3/library/pickle.html), using CLIMADA in-built function `save()`. Note that pickle has a [transient format](saving-with-pickle) and should be avoided when possible."
    ]
   },
   {
diff --git a/doc/tutorial/climada_entity_MeasureSet.ipynb b/doc/tutorial/climada_entity_MeasureSet.ipynb
index 785a909c..e1b93a10 100644
--- a/doc/tutorial/climada_entity_MeasureSet.ipynb
+++ b/doc/tutorial/climada_entity_MeasureSet.ipynb
@@ -620,9 +620,10 @@
   },
   {
    "cell_type": "markdown",
+   "id": "adb8d606",
    "metadata": {},
    "source": [
-    "Pickle can always be used as well:"
+    "Pickle can be used as well, but note that pickle has a [transient format](saving-with-pickle) and should be avoided when possible:"
    ]
   },
   {
diff --git a/doc/tutorial/climada_hazard_Hazard.ipynb b/doc/tutorial/climada_hazard_Hazard.ipynb
index 8cc95589..94dd517d 100644
--- a/doc/tutorial/climada_hazard_Hazard.ipynb
+++ b/doc/tutorial/climada_hazard_Hazard.ipynb
@@ -864,7 +864,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Pickle will work as well:"
+    "Pickle will work as well, but note that pickle has a [transient format](saving-with-pickle) and should be avoided when possible:"
    ]
   },
   {

Output of git status:

On branch develop
Your branch is up to date with 'origin/develop'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	climada/test/data/

nothing added to commit but untracked files present (use "git add" to track)

@emanuel-schmid
Copy link
Collaborator

Thanks! Now it's clear that the repository can't be blamed.
If the repo is the same as mine and the python version, and the installation procedure too, then maybe it's the operating system. So next, I'll try to reproduce the problem on a Mac...

@bguillod
Copy link
Collaborator Author

We tried out a bit out here and we are 3 people with M1 or M3 Macs experiencing roughly the same tests failing. We are unable to install an environment in which those all pass. I'm curious to see if you get the same issues on your end.

It can't be due to the use of conda instead of mamba, can it?

@emanuel-schmid
Copy link
Collaborator

🤔 Hm. Rather not. I went through the errors and found a reasonable explanation to most of them:

not unusual. floating point discrepancy between os.

  • climada.engine.test.test_impact.RPmatrix.local_exceedance_imp_pass - AssertionError: 2916964966.388227 != 2916964966.388219

to be investigated. missing BlackMarble file. automated download failed. perhaps config?

  • climada.test.test_litpop_integr.LitPopExposure.Liechtenstein_15_lit_pass -
  • climada.test.test_litpop_integr.LitPopExposure.Liechtenstein_30_pop_pass -
  • climada.test.test_litpop_integr.LitPopExposure.Monaco150_pass -
  • climada.test.test_litpop_integr.LitPopExposure.from_shape_and_countries_zurich_pass -
  • climada.test.test_litpop_integr.LitPopExposure.from_shape_zurich_pass -
  • climada.test.test_litpop_integr.LitPopExposure.netherlands150_pass -
  • climada.test.test_litpop_integr.LitPopExposure.switzerland300_admin1_pc2016_pass -
  • climada.test.test_litpop_integr.LitPopExposure.switzerland300_pass -
  • climada.test.test_litpop_integr.LitPopExposure.switzerland30normPop_pass -
  • climada.test.test_litpop_integr.Admin1.brandenburg -
  • climada.test.test_litpop_integr.Admin1.calc_admin1 -
  • climada.test.test_litpop_integr.Admin1.from_countries_calc_admin1_pass -
  • climada.test.test_nightlight.Nightlight.check_nl_local_file_exists -
  • climada.test.test_nightlight.Nightlight.load_nasa_nl_shape_single_tile -

not unusual. broken connection.

  • climada.test.test_nightlight.Nightlight.read_bm_files - requests.exceptions.ChunkedEncodingError

missing gpw_v4_population_count_rev11_2020_30_sec file. no automated download for this file. download and run again.

  • climada.test.test_litpop_integr.LitPopExposure.from_population - gpw_v4_population_count_rev11_2020_30_sec.tif missing.
  • climada.test.test_litpop_integr.LitPopExposure.from_nightlight_intensity - gpw_v4_population_count_rev11_2020_30_sec.tif missing
  • climada.test.test_litpop_integr.GPWPopulation.get_gpw_file_path_pass - gpw_v4_population_count_rev11_2020_30_sec.tif co...
  • climada.test.test_litpop_integr.GPWPopulation.load_gpw_pop_shape_pass - gpw_v4_population_count_rev11_2020_30_sec.tif co...

not unusual. most likely world bank data download failed.

  • climada.test.test_litpop_integr.LitPopExposure.BLM150_pass - KeyError: 'GDP_MD'
  • climada.util.test.test_finance.WBData.ne_gdp_aia_2012_pass - KeyError: 'GDP_MD'

irrelevant. not part of test suit. run make test instead of pytest .

  • script/jenkins/test_data_api.py::TestDataAvail::test_nasa_nl_pass - RuntimeError: Download failed. Please check the network connection and whether filenames are still valid.
  • script/jenkins/test_notebooks.py::NotebookTest::test_notebook - TypeError: chdir: path should be string, bytes, os.PathLike or integer, not NoneType

@emanuel-schmid
Copy link
Collaborator

So all in all it boils down to one interesting finding. the automated blackmarble file download fails.
What is the output for e.g.:

python -m unittest climada.test.test_litpop_integr.LitPopExposure.switzerland300_pass

?

@bguillod
Copy link
Collaborator Author

So all in all it boils down to one interesting finding. the automated blackmarble file download fails. What is the output for e.g.:

python -m unittest climada.test.test_litpop_integr.LitPopExposure.switzerland300_pass

?

The output would be this:

E
======================================================================
ERROR: LitPopExposure (unittest.loader._FailedTest.LitPopExposure)
----------------------------------------------------------------------
AttributeError: module 'climada.test.test_litpop_integr' has no attribute 'LitPopExposure'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

But I assume you actually wanted me to run this instead?
python -m unittest climada.test.test_litpop_integr.TestLitPopExposure.test_switzerland300_pass
If this is indeed what you wanted, here is the output:

E
======================================================================
ERROR: test_switzerland300_pass (climada.test.test_litpop_integr.TestLitPopExposure.test_switzerland300_pass)
Create LitPop entity for Switzerland on 300 arcsec:
----------------------------------------------------------------------
Traceback (most recent call last):
  File "rasterio/_io.pyx", line 938, in rasterio._io.DatasetReaderBase._read
  File "rasterio/_io.pyx", line 176, in rasterio._io.io_multi_band
  File "rasterio/_err.pyx", line 195, in rasterio._err.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X offset 0, Y offset 10127: TIFFReadEncodedStrip() failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bguillod/Documents/work/CT/code/climada_python/climada/test/test_litpop_integr.py", line 64, in test_switzerland300_pass
    ent = lp.LitPop.from_countries(country_name, res_arcsec=resolution, fin_mode=fin_mode,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bguillod/Documents/work/CT/code/climada_python/climada/entity/exposures/litpop/litpop.py", line 164, in from_countries
    [cls._from_country(country,
  File "/Users/bguillod/Documents/work/CT/code/climada_python/climada/entity/exposures/litpop/litpop.py", line 164, in <listcomp>
    [cls._from_country(country,
     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bguillod/Documents/work/CT/code/climada_python/climada/entity/exposures/litpop/litpop.py", line 631, in _from_country
    _get_litpop_single_polygon(polygon, reference_year,
  File "/Users/bguillod/Documents/work/CT/code/climada_python/climada/entity/exposures/litpop/litpop.py", line 726, in _get_litpop_single_polygon
    nlight, meta_nl = nl_util.load_nasa_nl_shape(polygon,
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bguillod/Documents/work/CT/code/climada_python/climada/entity/exposures/litpop/nightlight.py", line 131, in load_nasa_nl_shape
    out_image, meta_tmp = load_nasa_nl_shape_single_tile(geometry,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bguillod/Documents/work/CT/code/climada_python/climada/entity/exposures/litpop/nightlight.py", line 361, in load_nasa_nl_shape_single_tile
    out_image, transform = rasterio.mask.mask(src, [geometry], crop=True)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bguillod/miniconda3/envs/climada_env/lib/python3.11/site-packages/rasterio/mask.py", line 189, in mask
    out_image = dataset.read(
                ^^^^^^^^^^^^^
  File "rasterio/_io.pyx", line 614, in rasterio._io.DatasetReaderBase.read
  File "rasterio/_io.pyx", line 941, in rasterio._io.DatasetReaderBase._read
rasterio.errors.RasterioIOError: Read or write failed. /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif, band 1: IReadBlock failed at X offset 0, Y offset 10127: TIFFReadEncodedStrip() failed.

----------------------------------------------------------------------
Ran 1 test in 0.823s

FAILED (errors=1)

@bguillod
Copy link
Collaborator Author

As for the potentially config-related issue, I actually wonder if there are remnants from quite old climada installations I might have had which I should clean up to start from a cleaner state. However looking at my home (.config and climada/conf) it doesn't seem to be the case. Unless maybe old files are still stored in the Users/bguillod/climada/ folder which cause an issue?

@emanuel-schmid
Copy link
Collaborator

That could indeed be the case! To see whether this is the case you could temporarily rename the Users/bguillod/climada/data folder. This causes climada to download missing files as they are needed.
However, some files may need to be be downloaded manually for full functionality, so don't delete the folder yet.
After renaming the folder, listen to the warnings and react accordingly.

@emanuel-schmid
Copy link
Collaborator

But I assume you actually wanted me to run this instead? python -m unittest climada.test.test_litpop_integr.TestLitPopExposure.test_switzerland300_pass If this is indeed what you wanted, here is the output:

😁 exactly!

Then I'd just remove the file /Users/bguillod/climada/data/BlackMarble_2016_C1_geo_gray.tif and try again to see if it has an effect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants