This Long Short-Term Memory (LSTM) network was developed for use in the Next Generation Water Resources Modeling Framework (NextGen). LSTMs are able to provide relatively accurate streamflow predictions when compared to other model types. This module is available through a Basic Model Interface (BMI).
- Adaption from NeuralHydrology
- Sample Data
- Configurations
- Trained LSTM Model
- Dependencies
- Running BMI LSTM
- Weights and Biases
- Trained LSTM Model
- Unit Test
This module is dependent on a trained deep learning model. The forward pass of this LSTM model nextgen_cuda_lstm.py is heavily based on NeuralHydrology's CudaLSTM. Other model classes can be applied but bmi_lstm.py would need to load it in. More information about the python package NeuralHydrology can be found here.
All data required for a test run of this model is available in the data/ directory. This includes:
- Forcing data:
usgs-streamflow-nldas_hourly.nc - Observation values: also included in
usgs-streamflow-nldas_hourly.nc - Static attributes: see an example configuration file for a list of these attributes in
./bmi_config_files
for four USGS gauges:
- 02064000 Falling River nr Naruna, VA
- 01547700 Marsh Creek at Blanchard, PA
- 03015500 Brokenstraw Creek at Youngsville, PA
- 01022500 Narraguagus River at Cherryfield, Maine
Note that the data found in this repository are simply examples. The LSTM model can be run on any watershed, provided the necessary static attributes and dynamic forcings. The full list of attributes differs depending on the trained LSTM model chosen. Example files (*.yml) with the required attributes are located in the ./bmi_config_filesdirectory. The attributes required for these configuration files can be found in the camels_attributes_v2.0/ data directory for catchments in the CAMELS dataset or estimated from Addor, N., A.J. Newman, N. Mizukami, and M.P. Clark. 2017. The CAMELS data set: catchment attributes and meteorology for large-sample studies. Hydrol. Earth Syst. Sci. 21: 5293-5313. https://doi.org/10.5194/hess-21-5293-2017.
The LSTM model requires a configuration file for specification of forcings, weights, scalers, run options (like warmup period), runtime period, static basin parameters and model time step. This configuration file needs to be generated for any specific application of the LSTM model.
This LSTM model will run on any basin with the required inputs; however, it was trained on 500+ catchments from the CAMELS dataset across the contiguous United States (CONUS) and is best suited to this CONUS region, for now. The place to set up the run for a specific configuration for a specific basin is in the BMI (*.yml) configuration file. Ideally, the LSTM trained with all forcings and all static attributes will be used, but we've included a few example LSTMs that have limited static attributes and forcings, in the event that the total set of forcings and attributes are not available. For explanations of how the LSTM might perform with limited inputs and on ungauged basins, see Frederik Kratzert et al., Toward Improved Predictions in Ungauged Basins: Exploiting the Power of Machine Learning, Water Resources Research. To set up a specific configuration for a specific basin, change the appropriate BMI configuration file.
Included in this directory are three samples of trained LSTM models:
hourly_all_attributes_and_forcings: This is the model that should be used. It was trained to ingest 8 atmospheric forcings and 26 static attributes, that were chosen from the CAMELS dataset. If you do not have access to all these static attributes, one of the models below are available with limited static attributes, but in general would be best to use all data possible.hourly_slope_mean_precip_temp: This model was trained to ingest only two atmospheric forcings (total precipitation and temperature) and two static attributes (basin mean slope and elevation).hourly_all_forcings_lat_lon_elev: This model was trained to ingest eight atmospheric forcings (total precipitation, longwave radiation, shortwave radiation, pressure, specific humidity, temperature, wind in the X and Y directions) and three static attributes (basin mean elevation, latitude and longitude).
These three models are trained with different inputs, but they all will run with the same BMI and LSTM model.
Running this model requires python and the libraries listed in the environment file. This example uses Anaconda, but it isn’t a requirement. You can opt to set up a python environment without it by using the libraries specified in the environment.yml file. If you have Anaconda, you can easily create an environment (bmi_lstm) with the required libraries using: conda env create -f environment.yml.
Notice that xarray has a specific version defined in the environment file (0.14.0) as the newer versions are incompatible with the current example files. The same goes for llvm-openmp, which we set to version 10.0.0 in the dependencies. On some Mac Anaconda releases, users received an error message stating OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized. If you get this message, please make sure you have - llvm-openmp=10.0.0 set in your environment.yml file. More information on different solutions to resolving this issue can be found here.
If at any point you want to see the full list of the packages and dependencies in your activated bmi_lstm environment, run conda env export > environment_<rename>.yml replacing <rename> with your text of choice to avoid overwriting the original environment.yml file.
This section goes through an example of running the LSTM with the BMI interface. These are only examples. If a user wants to run the LSTM with BMI, then these are a jumping off point. These examples were developed to provide a quick testing ground for running the LSTM with the NextGen framework. See the doc/ folder for more information regarding running this module within NextGen as well as the ngen_files/README.txt found here.
Note that this code assumes the use of the bmi_lstm environment for Anaconda. To load this environment, enter conda activate bmi_lstm. Install the library, pip install lstm and execute python -m lstm. See PACKAGE.md for more information about running lstm as a python library.
Be aware that these scripts are examples and may require changes for your use case. For example, the Python script was developed for the trained LSTM model with limited attributes (hourly_slope_mean_precip_temp) and the for loop will need to be changed if running with the LSTM model that was trained with all attributes (an example of this code can be found in the Jupyter Notebook.
Running these examples of trained LSTM-based hydrological models require these general steps:
- Retrieve atmospheric forcing data that match those included in the trained model
- Retrieve the catchment attributes that match those included in the trained model
- Create a configuration file with the key-value pairs that can be used by the BMI
- Run a script with the Python commands for the BMI model control functions
The Jupyter Notebook and a Python script run_lstm_bmi.py have an example of running the LSTM with BMI model control functions, which can be summarized as follows:
conda activate bmi_lstm- Import required libraries (e.g.,
import torch) - Load in the model from the BMI file:
model = lstm.bmi_LSTM() - Read in the configuration file, and this includes the model weights, etc.:
model.read_cfg_file() - Now start running the BMI functions, starting with initialize:
model.initialize() - The model is now available to run either one timestep at a time:
model.update(), or many timesteps at a time:model.update_until(model.iend), where model.iend is the end of the forcing file, but this can be any value less than or equal to the end of the forcing file. - And finally you should finalize the model instance:
model.finalize()
This repository contains an example file with weather and observed streamflow data for four catchments here. Note that the observed streamflow data isn’t necessary to run the model, but is useful for comparison purposes.
Also contained within this repository are catchment attributes for all CAMELS catchments along with two example configuration files: one for the limited data case and one for the full set of attributes.
To run the LSTM model for another catchment, slight modifications to this code will be needed:
- The configuration file path when setting the
model.initialize(bmi_cfg_file='./path/to/your/config/file.yml')function - Streamflow and weather data path when defining
sample_data. These examples shown here are stored in a NetCDF file, but the user is free to store and read the data for their use case however they please. - Check how the streamflow and weather variables are defined/passed into the model as there could be variations in headers, etc. in your data file – These are defined in a for loop.
The training procedure should produce weights and biases for the LSTM model. These are stored in Pytorch files (*.pt), are kept within the training directories: trained_neuralhydrology_models. Without these the model can still run, but will not make streamflow predictions. These are absolutely necessary for running this model, including coupling, with the NextGen framework. These weights and biases are trained to represent many basins, so they do not change for every basin. The model may be trained regionally, or globally, and the weights and biases need to be consistent across the appropriate basins. In the examples contained within this repository, we trained the models to ingest particular inputs (both static and dynamic), and the weights associated with those models cannot be interchanged.
BMI has functions that are used by a framework, or model driver, that allows interaction with models through consistent commands. The unit tests are designed to test those BMI functions (run in these examples from Python commands), to ensure that a framework, or model driver, will get the expected result when a command is called. BMI includes functions for different parts of the modeling chain, including functions to get information from the models (known as getters), functions to set information in the models (know as setters), functions to setup and run the models, etc. The unit test includes these functions, categorized below:
- Model control functions (4)
- Model information functions (5)
- Variable information functions (6)
- Time functions (5)
- Variable getter and setter functions (5)
- Model grid functions (16)
The test script run_bmi_unit_test.py fully examines the functionality of all applicable definitions.
To run lstm-bmi unit test, from the parent directory, simply call python ./lstm/run_bmi_unit_test.py within the active conda environment bmi_lstm, as outlined in Running BMI LSTM.
Recall that BMI guides interoperability for model-coupling, where model components (i.e. inputs and outputs) are easily shared amongst each other. When testing outside of a true framework, we consider the behavior of BMI function definitions, rather than any expected values they produce.