Porting mct_routing from Lisflood version 4.1 to 4.3.1

.coveragerc

@@ -1,4 +1,3 @@
 [run]
 omit =
-    src/lisflood/hydrological_modules/compile_kinematic_wave_parallel_tools.py
     tests/*

.gitattributes

@@ -0,0 +1,6 @@
+docs/assets/fonts/* linguist-vendored
+docs/assets/js/main.min.js linguist-vendored
+docs/assets/js/lunr/* linguist-vendored
+docs/assets/js/plugins/* linguist-vendored
+docs/assets/js/vendor/* linguist-vendored
+docs/_sass/minimal-mistakes/vendor/* linguist-vendored

.github/workflows/ci_env.yml

@@ -0,0 +1,47 @@
+name: Lisflood OS Unit Tests
+
+on: [push]
+
+jobs:
+  tests:
+    runs-on: ubuntu-20.04
+    strategy:
+      max-parallel: 5
+
+    steps:
+    - uses: actions/checkout@v3
+    - uses: conda-incubator/setup-miniconda@v2
+      with:
+        auto-update-conda: true
+        python-version: 3.7
+    - name: Conda info
+      shell: bash -el {0}
+      run: conda info
+    - name: Install python and gcc
+      shell: bash -el {0}
+      run: |
+        conda install -c conda-forge python=3.7
+        conda install -c conda-forge gcc=12.1.0
+    - name: Install gdal and pcraster
+      shell: bash -el {0}
+      run: |
+        conda install -c conda-forge gdal pcraster
+    - name: Install dependencies
+      shell: bash -el {0}
+      run: |
+        pip install -r requirements.txt
+    - name: Install lisflood-module
+      shell: bash -el {0}
+      run: |
+        pip install .
+    - name: Check installation
+      shell: bash -el {0}
+      run: |
+        gdal-config --version
+        python -c "from osgeo import gdal; print(gdal.__version__)"
+        conda list
+    - name: Test with pytest
+      shell: bash -el {0}
+      run: |
+        pip install pytest pytest-cov
+        pytest

.gitignore

@@ -1,15 +1,27 @@
+*.gem
+*.sublime-project
+*.sublime-workspace
+.bundle
+.DS_Store
+.jekyll-metadata
+.sass-cache
+_asset_bundler_cache
+_site
+codekit-config.json
+example/_site
+Gemfile.lock
+node_modules
+npm-debug.log*
 .idea
 *.py[cod]
 __pycache__/
 build/
 *.so
-kinematic_wave_parallel_tools.html
 lisflood_model.egg-info
 /dist/
 .eggs
 .tox
 /src/dist/
-/src/lisflood/hydrological_modules/kinematic_wave_parallel_tools.c
 .coverage
 *.tox.ini
 .vscode/

Dockerfile

@@ -27,9 +27,6 @@ COPY src/lisfloodSettings_reference.xml /
 COPY LICENSE /
 COPY VERSION /
 
-# Compile kwpt
-RUN cd /lisflood/hydrological_modules && conda run -n lisflood python compile_kinematic_wave_parallel_tools.py build_ext --inplace
-
 # RUN Tests
 COPY tests/. /tests/
 COPY pytest.ini /tests

MANIFEST.in

@@ -1,4 +1,3 @@
-include src/lisflood/hydrological_modules/kinematic_wave_parallel_tools.pyx
 include requirements.txt
 include *.xml
 include VERSION

README.md

@@ -26,7 +26,7 @@ You can use conda environment to easily install dependencies.
 ```bash
 conda create --name lisflood python=3.7 -c conda-forge
 conda activate lisflood
-conda install -c conda-forge pcraster
+conda install -c conda-forge pcraster gdal
 ```
 
 * Install lisflood-model pypi package
@@ -61,30 +61,12 @@ pip install -r requirements.txt
 If you don't use conda but a plain virtualenv, you need to install PCRaster and GDAL by your own and include its python interface in PYTHONPATH environment variable.
 For details, please follow instruction on [official docs](http://pcraster.geo.uu.nl).
 
-
-3. Compile the cython module kinematic_wave_parallel_tool
-
-To compile this Cython module to enable OpenMP multithreading (parallel kinematic wave):
-
-* Delete the files *.so (if any) in directory hydrological-modules  
-
-* Inside the hydrological_modules folder, execute "python compile_kinematic_wave_parallel_tools.py build_ext --inplace"  
-
-Important: the module has to be compiled on the machine where the model is run - the resulting binary is not portable.  
-
-Then in the settings file the option "numberParallelThreadsKinematicWave" may take the following values:
-    - "0"           : auto-detection of the machine/node's number of CPUs (all CPUs are used minus 1) (do not set it if other simulations are running on the same machine/node)
-    - "1"           : serial execution (not parallel)
-    - "2", "3", ... : manual setting of the number of parallel threads.
-                      (if exceeding the number of CPUs, the option is set to "0") -->  
-```xml
-<textvar name="numCPUs_parallelKinematicWave" value="3"/>
-```
-4. Run a cold run for the test catchment
+3. Run a cold run for the test catchment
 
 Now your environment should be set up to run lisflood. Try with a prepared settings file for one of the two test catchments:
 
 ```bash
+mkdir tests/data/LF_ETRS89_UseCase/out
 python src/lisf1.py tests/data/LF_ETRS89_UseCase/settings/cold.xml
 ```
 

VERSION

@@ -1 +1 @@
-4.1.1
+4.3.1

docs/.editorconfig

@@ -0,0 +1,13 @@
+# editorconfig.org
+root = true
+
+[*]
+indent_style = space
+indent_size = 2
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = false
+
+[*.md]
+trim_trailing_whitespace = false

docs/0_disclaimer/index.md

@@ -0,0 +1,3 @@
+# Disclaimer
+
+Both the program code and the LISFLOOD documentation (including the [LISFLOOD Model Documentation](https://ec-jrc.github.io/lisflood-model/), the [LISFLOOD User Guide](https://ec-jrc.github.io/lisflood-code/) and the [LISVAP documentation](https://ec-jrc.github.io/lisflood-lisvap/)) have been carefully inspected before publishing. However, no  warranties, either expressed or implied, are made concerning the accuracy, completeness, reliability, usability, performance, or fitness for any particular purpose of the information contained in this documentation, to the software described in this documentation, and to other material supplied in connection therewith. The material is provided \"as is\". The entire risk as to its quality and performance is with the user.

docs/0_references/index.md

@@ -0,0 +1,101 @@
+[//]: # (References)
+
+Allen, R. G., Pereira, L. S., Raes, D., and Smith, M.: FAO Irrigation and Drainage Paper No. 56: Crop Evapotranspiration (guidelines for computing crop water requirements), 1998. [available online: https://www.researchgate.net/publication/284300773_FAO_Irrigation_and_drainage_paper_No_56, last accessed: 13.05.2021.]
+
+Anderson, 2006Anderson, E., 2006. *Snow Accumulation and Ablation Model -- SNOW-17*. Technical report.
+
+Aston, A.R., 1979. Rainfall interception by eight small trees. Journal of Hydrology 42, 383-396.
+
+Bódis, K., 2009. *Development of a data set for continental hydrologic modelling*. Technical Report EUR 24087 EN JRC Catalogue number: LB-NA-24087-EN-C, Institute for Environment and Sustainability, Joint Research Centre of the European Commission Land Management and Natural Hazards Unit Action FLOOD. Input layers related to topography, channel geometry, land cover and soil characteristics of European and African river basins.
+
+Buchhorn, M., Lesiv, M., Tsendbazar, N.-E., Herold, M., Bertels, L., and Smets, B.: Copernicus Global Land Cover Layers - Collection 2. Remote Sensing 2020, 12Volume 108, 1044. doi:10.3390/rs12061044
+
+Burek, P., Bianchi, A., and Gentile, A.: A pan-European Data Set for hydrological modelling. JRC Technical Reports, 2014 [available online: https://ec-jrc.github.io/lisflood/pdfs/Dataset_hydro.pdf, last accessed: 13.05.2021.].
+
+Burek, P., Van Der Knijff, J., and De Roo, A.: LISFLOOD - Distributed Water Balance and Flood Simulation Model - Revised User Manual. Luxembourg: Publications Office of the European Union, 2013, JRC78917 [available online: https://publications.jrc.ec.europa.eu/repository/handle/JRC78917, last accessed: 02.06.2021.].
+
+Büttner, G., Kosztra, B., Maucha, G., Pataki, R., Kleeschulte, S., Hazeu, G., Vittek, M., and Schröder, C.: CORINE Land Cover Product User Manual – Final draft, Version 1.2. Environment Agency Austria, 2021 [available online: https://land.copernicus.eu/user-corner/technical-library/clc-product-user-manual, last accessed: 13.05.2021.]
+
+Carneiro Freire, S., Macmanus, K., Pesaresi, M., Doxsey-Whitfield, E., and Mills, J.: Development of new open and free multi-temporal global population grids at 250 m resolution. Geospatial Data in a Changing World; Association of Geographic Information Laboratories in Europe (AGILE) (Organiser). AGILE; 2016. JRC100523
+
+Chow, V.T., Maidment, D.R., Mays, L.M., 1988. Applied Hydrology, McGraw-Hill, Singapore, 572 pp.
+
+De Roo, A., Thielen, J., Gouweleeuw, B., 2003. LISFLOOD, a Distributed Water-Balance, Flood Simulation, and Flood Inundation Model, User Manual version 1.2. Internal report, Joint Research Center of the European Communities, Ispra, Italy, 74 pp.
+
+de Sousa, L. M., Poggio, L., Batjes, N. H., Heuvelink, G. B. M., Kempen, B., Riberio, E., and Rossiter, D.: SoilGrids 2.0: producing quality-assessed soil information for the globe, SOIL Discuss. [preprint], https://doi.org/10.5194/soil-2020-65, in review, 2020.
+
+Fröhlich, W., 1996. Wasserstandsvorhersage mit dem Prgramm ELBA. Wasserwirtschaft Wassertechnik, ISSN: 0043-0986, Nr. 7, 1996, 34-37.
+
+Goudriaan, J., 1977. Crop micrometeorology: a simulation study. Simulation Monographs. Pudoc, Wageningen.
+
+Hock, 2003Hock, R., 2003. Temperature index melt modelling in mountain areas. *Journal of Hydrology*, 282(1-4), 104--115.
+
+Laborte, A., Gutierrez, M., Balanza, J. et al. RiceAtlas, a spatial database of global rice calendars and production. Sci Data 4, 170074 (2017). https://doi.org/10.1038/sdata.2017.74
+
+Lehner, B., and Döll, P.: Development and validation of a global database of lakes, reservoirs and wetlands, Journal of Hydrology, Volume 296, Issues 1–4, 20 August 2004, Pages 1-22, http://dx.doi.org/10.1016/j.jhydrol.2004.03.028.
+
+Lindström, G., Johansson, B., Persson, M., Gardelin, M., Bergström, S., Development and test of the distributed HBV-96 hydrological model. Journal of Hydrology 201, 272-288.
+
+Maidment, D.R. (ed.), 1993. Handbook of Hydrology, McGraw-Hill.
+
+Martinec, J., Rango, A., Roberts, R.T., 1998. Snowmelt Runoff Model (SRM) User\'s Manual (Updated Edition 1998, Version 4.0). Geographica Bernensia, Department of Geography - University of Bern, 1999. 84pp.
+
+Merriam, R.A., 1960. A note on the interception loss equation. Journal of Geophysical Research 65, 3850-3851.
+
+Moiret-Guigand, A.: CLC2018 / CLCC1218 VALIDATION REPORT, Issue 1.3, 2021 [available online: https://land.copernicus.eu/user-corner/technical-library/clc-2018-and-clc-change-2012-2018-validation-report/view, last accessed:13.05.2021.].
+
+Molnau, M., Bissell, V.C., 1983. A continuous frozen ground index for flood forecasting. In: Proceedings 51^st^ Annual Meeting Western Snow Conference, 109-119.
+
+Rao, C.X. and Maurer, E.P., 1996. A simplified model for predicting daily transmission losses in a stream channel. Water Resources Bulletin, Vol. 31, No. 6., 1139-1146.
+
+Schiavina, M., Freire, S., and MacManus, K.:  GHS-POP R2019A - GHS population grid multitemporal (1975-1990-2000-2015). European Commission, Joint Research Centre (JRC), 2019. [Dataset] doi:10.2905/0C6B9751-A71F-4062-830B-43C9F432370F PID: http://data.europa.eu/89h/0c6b9751-a71f-4062-830b-43c9f432370f
+
+Smets, B., Verger, A., Camacho, F., Van der Goten, R., and Jacobs, T.: Copernicus Global Land Operations ”Vegetation and Energy”: Product User Manual, Issue 1.33, 2019 [available online: https://land.copernicus.eu/global/sites/cgls.vito.be/files/products/CGLOPS1_PUM_LAI1km-V2_I1.33.pdf, last accessed: 13.05.2021.].
+
+Speers, D.D. , Versteeg, J.D., 1979. Runoff forecasting for reservoir operations -- the past and the future. In: Proceedings 52^nd^ Western Snow Conference, 149-156.
+
+Stroosnijder, L., 1982. Simulation of the soil water balance. In: Penning de Vries, F.W.T., Van Laar, H.H. (eds), Simulation of Plant Growth and Crop Production, Simulation Monographs, Pudoc, Wageningen, pp. 175-193.
+
+Stroosnijder, L., 1987. Soil evaporation: test of a practical approach under semi-arid conditions. Netherlands Journal of Agricultural Science 35, 417-426.
+
+Supit, I., Hoojer, A.A., and Van Diepen, C.A.: System description of the Wofost 6.0 crop simulation model implemented in CGMS. Volume 1: Theory and Algorithms, 1994. [available online: https://www.researchgate.net/publication/282287246_System_description_of_the_Wofost_60_crop_simulation_model_implemented_in_CGMS_Volume_1_Theory_and_Algorithms, last accessed: 13.05.2021.].
+
+Supit, I. , van der Goot, E. (eds.), 2003. Updated System Description of the WOFOST Crop Growth Simulation Model as Implemented in the Crop Growth Monitoring System Applied by the European Commission, Treemail, Heelsum, The Netherlands, 120 pp.
+
+Todini, E., 1996. The ARNO rainfall-runoff model. Journal of Hydrology 175, 339-382.
+
+Tóth, B., Weynants, M., Nemes, A., Makó, A., Bilas, G., and Toth, G.: New generation of hydraulic pedotransfer functions for Europe. EUROPEAN JOURNAL OF SOIL SCIENCE 66 (1); 2015. p. 226-238. JRC91453
+
+Van Der Knijff, J., De Roo, A., 2006. LISFLOOD -- Distributed Water Balance and Flood Simulation Model, User Manual. EUR 22166 EN, Office for Official Publications of the European Communities, Luxembourg, 88
+pp.
+
+Van der Knijff, J., 2008. LISVAP-- Evaporation Pre-Processor for the LISFLOOD Water Balance and Flood Simulation Model, Revised User Manual. EUR 22639 EN/2, Office for Official Publications of the European Communities, Luxembourg, 31 pp.
+
+Van Der Knijff, J., De Roo, A., 2008. LISFLOOD -- Distributed Water Balance and Flood Simulation Model, Revised User Manual. EUR 22166 EN/2, Office for Official Publications of the European Communities, Luxembourg, 109 pp.
+
+Van der Knijff, J. M., Younis, J. and de Roo, A. P. J.: LISFLOOD: A GIS-based distributed model for river basin scale water balance and flood simulation, Int. J. Geogr. Inf. Sci., 24(2), 189--212, 2010.
+
+Van Genuchten, M.Th., 1980. A closed-form equation for predicting the hydraulic conductivity of unsaturated soils. Soil Science Society of America Journal 44, 892-898.
+
+Viviroli et al., 2009Viviroli, D., Zappa, M., Gurtz, J., & Weingartner, R., 2009. An introduction to the hydrological modelling system PREVAH and its pre- and post-processing-tools. *Environmental Modelling & Software*, 24(10), 1209--1222.
+
+Vogt et al., 2007Vogt, J., Soille, P., de Jager, A., Rimaviciute, E., Mehl, W., Foisneau, S., Bodis, K., Dusart, M., Parachini, M., Hasstrup, P.,2007. *A pan-European River and Catchment Database*. JRC Reference Report EUR 22920 EN, Institute for Environment and Sustainability, Joint Research Centre of the European Commission.
+
+Von Hoyningen-Huene, J., 1981. Die Interzeption des Niederschlags in landwirtschaftlichen Pflanzenbeständen (Rainfall interception in agricultural plant stands). In: Arbeitsbericht Deutscher Verband für Wasserwirtschaft und Kulturbau, DVWK, Braunschweig, p.63.
+
+Wesseling, C.G., Karssenberg, D., Burrough, P.A. , Van Deursen, W.P.A., Integrating dynamic environmental models in GIS: The development of a Dynamic Modelling language. Transactions in GIS 1, 40-48.
+
+World Meteorological Organization, 1986. Intercomparison of models of snowmelt runoff. Operational Hydrology Report No. 23.
+
+Yamazaki, D., Ikeshima, D., Sosa, J., Bates, P.D., Allen, G.H., and Pavelsky, T.M.: MERIT Hydro: A high-resolution global hydrography map based on latest topography datasets. Water Resources Research, vol.55, pp.5053-5073, 2019, DOI: 10.1029/2019WR024873
+
+Yamazaki, D., Ikeshima, D., Tawatari, R., Yamaguchi, T., O'Loughlin, F., Neal, J. C., Sampson, C. C., Kanae, S., and Bates, P. D.: A high-accuracy map of global terrain elevations. Geophysical Research Letters, vol.44, pp.5844-5853, 2017, DOI: 10.1002/2017GL072874
+
+Young, G.J. (ed), 1985. Techniques for prediction of runoff from glacierized areas. IAHS Publication 149, Institute of Hydrology, Wallingford.
+
+Yu, Q., You, L., Wood-Sichra, U., Ru, Y., Joglekar, A. K. B., Fritz, S., Xiong, W., Lu, M., Wu, W., and Yang, P.: A cultivated planet in 2010: 2. the global gridded agricultural production maps, Earth Syst. Sci. Data Discuss., https://doi.org/10.5194/essd-2020-11, in review, 2020. doi: 10.5194/essd-2020-11
+
+Zhao, R.J., Liu, X.R., 1995. The Xinanjiang model. In: Singh, V.P. (ed.), Computer Models of Watershed Hydrology, pp. 215-232.
+
+
+

docs/1_introduction_LISFLOOD/index.md

@@ -0,0 +1,27 @@
+## About LISFLOOD
+
+LISFLOOD is a spatially distributed, semi-physical hydrological rainfall-runoff model that has been developed by the Joint Research Centre (JRC) of the European Commission in late 90s. 
+Since then, LISFLOOD has been applied to a wide range of applications such as all kind of water resources assessments looking at e.g. 
+the effects of climate and land-use change as well as river regulation measures. 
+Its most prominent application is probably within the [European Flood Awareness System, EFAS](https://www.efas.eu/en) and the [Global Flood Awareness System, GloFAS](https://www.globalfloods.eu/)
+operated under [Copernicus Emergency Management System, EMS](https://emergency.copernicus.eu/).
+
+Its wide applicability is due to its modular structure as well as its temporal and spatial flexibility. 
+The model can be extended with additional modules when need arises, to satisfy the new target objective. 
+In that sense it can be extended to include anything from a better representation of a particular hydrological flow to the implementation of anthropogenic-influenced processes. 
+At the same time the model has been designed to be applied across a wide range of spatial and temporal scales. 
+LISFLOOD is grid-based, and applications so far have employed grid cells of as little as 100 metres for medium-sized catchments, to 5000 metres for modelling 
+the whole of Europe and up to 0.1° (around 10 km) for modelling globally. Long-term water balance can be simulated (using a daily time step), 
+as well as individual flood events (using hourly time intervals, or even smaller). 
+
+Although LISFLOOD's primary output product is channel discharge, all internal rate and state variables (soil moisture, for example) can be written as output as well.
+ All output can be written as grids, or time series at user-defined points or areas. 
+ The user has complete control over how output is written, thus minimising any waste of disk space or CPU time.
+
+LISFLOOD is implemented in Python and PCRaster Model Framework, wrapped in a Python based interface. 
+
+The Python wrapper of LISFLOOD enables the user to control the model inputs and outputs and the selection of the model modules. 
+LISFLOOD runs on any operating system for which Python and PCRaster are available.
+
+
+[🔝](#top)

docs/1_introduction_usermanual/index.md

@@ -0,0 +1,20 @@
+## About this user guide
+
+
+This guide is hosted on GitHub Pages platform and it's the most up-to-date and complete user instruction of the LISFLOOD model. 
+It contains a detailed step-by-step guide on what you need to do and know through the whole chain from the system requirements to receiving the LISFLOOD output. 
+
+In order to apply this knowledge into practice, we provide two fully implemented test cases: [LISFLOOD User Cases](https://github.com/ec-jrc/lisflood-usecases/). By running the test cases, you can verify that the model has been installed and used by you correctly. Then it will be time for you to move onto your own set-up.
+
+Note, this document is **not a LISFLOOD model documentation**. The [lisflood-model official documentation](https://ec-jrc.github.io/lisflood-model/) contains the most up-to-date and complete technical documentation of the LISFLOOD model. This includes all the concepts and model equations of all the standard LISFLOOD processes, but also all the optional modules. 
+
+Lastly, we also share with you two other tools: 
+
+* LISVAP, our tool to calculate the evapotranspiration and 
+* our calibration tool that we've developed.
+
+LISVAP Source code is on LISVAP [GitHub repository](https://github.com/ec-jrc/lisflood-lisvap) while its documentation can be found at [LISVAP GitHub pages](https://ec-jrc.github.io/lisflood-lisvap/).
+
+Calibration tool source code is on his dedicated [repository](https://github.com/ec-jrc/lisflood-calibration) while documentation is at [Calibration Tool GitHub Pages](https://ec-jrc.github.io/lisflood-calibration/).
+
+[🔝](#top)