Add sensitivity nb

[richardarsenault]

Added a notebook to perform Sobol sensitivity analysis. Updated notebook index.

[tlvu]

Do not merge before Ouranosinc/PAVICS-e2e-workflow-tests#127 goes live.

[Zeitsperre]

It's probably safe to merge, but we shouldn't release a new RavenPy until that PR is in prod.

[tlvu]

It's probably safe to merge, but we shouldn't release a new RavenPy until that PR is in prod.

No safe to merge because as soon as it is merge, the new nb will appear in the tutorial-notebooks on the PAVICS Jupyter while the runtime environment is unable to run it yet (missing this new SAlib).

[Zeitsperre]

Ah! Gotcha. My bad.

[tlvu]

@richardarsenault

The Beta env on PAVICS has SAlib installed. I tried a Jenkins run and got a timeout. One cell was timing out after 2000 sec (33 minutes), which is a lot for one cell. Can you try the new notebook in the Beta env see if it does the same excessive run time. Otherwise maybe the notebook is unable run unattended and needs user interaction?

16:49:09  =================================== FAILURES ===================================
16:49:09  _ RavenPy-add_sensitivity_nb/docs/notebooks/Sensitivity_analysis.ipynb::Cell 3 _
16:49:09  Notebook cell execution failed
16:49:09  Cell 3: Timeout of 2000 seconds exceeded while executing cell. Failed to interrupt kernel in 5 seconds, so failing without traceback.
16:49:09  
16:49:09  Input:
16:49:09  # Set the working directory at one place so all files are overwritten at the same place (Avoids creating hundreds or thousands
16:49:09  # of folders with each run's data)
16:49:09  workdir = Path(tempfile.mkdtemp())
16:49:09  
16:49:09  # Pre-define the results matrix based on the number of parameters we will test (and thus how many runs we will need to do). We will test SA with
16:49:09  # two objective functions (NSE and AbsErr). Let's pre-define both vectors now.
16:49:09  Y_NSE = np.zeros([param_values.shape[0]])
16:49:09  Y_ABS = np.zeros([param_values.shape[0]])
16:49:09  
16:49:09  # Define a run name for files
16:49:09  run_name = "SA_Sobol"
16:49:09  
16:49:09  # Now we have a loop that runs the model iteratively, once per parameter set:
16:49:09  for i, X in enumerate(param_values):
16:49:09      # We need to create the desired model with its parameters the same way as in the Notebook 04_Emulating_hydrological_models.
16:49:09      m = GR4JCN(
16:49:09          ObservationData=[rc.ObservationData.from_nc(nc_file, alt_names="qobs")],
16:49:09          Gauge=[
16:49:09              rc.Gauge.from_nc(
16:49:09                  nc_file,
16:49:09                  alt_names=alt_names,
16:49:09                  data_kwds={"ALL": {"elevation": hru["elevation"]}},
16:49:09              )
16:49:09          ],
16:49:09          HRUs=[hru],
16:49:09          StartDate=dt.datetime(1990, 1, 1),
16:49:09          EndDate=dt.datetime(1999, 12, 31),
16:49:09          RunName=run_name,
16:49:09          EvaluationMetrics=eval_metrics,  # We add this code to tell Raven which objective function we want to pass.
16:49:09          SuppressOutput=True,  # This suppresses the writing of files to disk, returning only basic information such as the evaluation metrics values.
16:49:09          params=X.tolist(),  # Here is where we pass the paramter sets to the model, from the loop enumerator X.
16:49:09      )
16:49:09  
16:49:09      # Write the files to disk, and overwrite existing files in the folder (we already got the values we needed from previous runs)
16:49:09      m.write_rv(workdir=workdir, overwrite=True)
16:49:09  
16:49:09      # Run the model and get the path to the outputs folder that can be used in the output reader.
16:49:09      outputs_path = run(modelname=run_name, configdir=workdir)
16:49:09  
16:49:09      # Get the outputs using the Output Reader object.
16:49:09      outputs = OutputReader(run_name=run_name, path=outputs_path)
16:49:09  
16:49:09      # Gather the results for both of the desired objective functions. We will see how the choice of objective function impacts sensitivity.
16:49:09      Y_NSE[i] = outputs.diagnostics["DIAG_NASH_SUTCLIFFE"][0]
16:49:09      Y_ABS[i] = outputs.diagnostics["DIAG_ABSERR"][0]
16:49:09  
16:49:09      # Print the status of the hydrological modelling runs.
16:49:09      print(
16:49:09          str(i + 1)
16:49:09          + " out of "
16:49:09          + str(param_values.shape[0])
16:49:09          + " runs performed. Please wait."
16:49:09      )
16:49:09  
16:49:09  =========================== short test summary info ============================
16:49:09  FAILED RavenPy-add_sensitivity_nb/docs/notebooks/Sensitivity_analysis.ipynb::Cell 3
16:49:09  ============ 1 failed, 258 passed, 2 xfailed in 3486.97s (0:58:06) =============

[richardarsenault]

Hi!

In theory, it should be pretty quick to run (maybe a minute or two MAX). However, while running on my account, I found that it oftentimes killed the kernel, but not always and it is not easily reproducible. It seems to be random. Perhaps this is what is hanging the process?

[tlvu]

Hi!

In theory, it should be pretty quick to run (maybe a minute or two MAX). However, while running on my account, I found that it oftentimes killed the kernel, but not always and it is not easily reproducible. It seems to be random. Perhaps this is what is hanging the process?

Oh, good guess. There was also some error about the kernel "Failed to interrupt kernel in 5 seconds, so failing without traceback.".

Is there a way to modify the notebook to avoid this random kernel killing? If it fails randomly when running interactively, then it's not a very good tutorial notebook.

Does this random failure only happen on PAVICS Jupyter env or also on your local machine. PAVICS Jupyter env is fairly big so maybe there is a package in there that is interfering with the notebook. Hope it is not this case as find that bad package will be complicated.

[richardarsenault]

I did not test in on my system, only on the platform itself. Sometimes it works, sometimes it fails, and I don't know why. I could try to add pauses in case it's a race condition type of thing, but other than that I am stumped!

Some posts on forums talk about ipykernel being the problem but I don't know that it's the same root cause etc.

[richardarsenault]

Here are a few more details:

1- Adding pauses did not help;
2- It fails almost every time (but not always) when I restart the kernel before running;
3- The cell that fails does not use SALib at all. It's just running Ravenpy multiple times in a loop and storing results. So there is no call to SALib in there.
4- Sometimes it fails at iteration 2, 4 or 10... and sometimes it doesn't crash at all.

I know this probably doesn't help, but it's the most info I can provide now. If there are other ideas of tests I could do, please let me know!

[huard]

Ok, I can reproduce the problem. I'm wondering if its related to RAM usage. It's weird that rerunning the same nb, I get different numbers for the Mem value (upper right). From my few experiments, if you stay below 930 MB, it works.

[huard]

Initial memory: 603 MB
import raven: 690 MB
from SALib.analyze import sobol as sobol_analyzer
from SALib.sample import sobol as sobol_sampler: 725 MB
param_values = sobol_sampler.sample(problem, N): 814 MB
rc.ObservationData.from_nc(nc_file, alt_names="qobs") : 913 MB

One option would be to try to reduce the RAM usage of RavenPy and its operations. Maybe a fausse piste, but I have no other ideas at this point.

[tlvu]

FYI 900 MB ram is peanuts for the server. There is also no max ram limit, so taking more ram should not be an issue.

[huard]

Another possibility is that the problem is related to #288

[richardarsenault]

Good catch. This makes sense. However, in this case we have SuppressOutput = True, so is it really calling that function? I would have expected only the Diagnostics would be generated?

[huard]

Maybe it's reading the input files over and over that somehow creates a problem.

[richardarsenault]

Would this also happen during calibration? I don't recall how SpotPy manages Raven etc. I guess it needs to read the files there too. Maybe the time it takes to write the new rvp files is sufficient to let the system read the files after with no issue? Just throwing out ideas here...

[huard]

Try this:

config = dict(ObservationData=[rc.ObservationData.from_nc(nc_file, alt_names="qobs")],
              Gauge=[
                rc.Gauge.from_nc(
                    nc_file,
                    alt_names=alt_names,
                    data_kwds={"ALL": {"elevation": hru["elevation"]}},
                )
                ],
              HRUs=[hru],
              StartDate=dt.datetime(1990, 1, 1),
              EndDate=dt.datetime(1999, 12, 31),
              RunName=run_name,
              EvaluationMetrics=eval_metrics,  # We add this code to tell Raven which objective function we want to pass.
              SuppressOutput=True,  # This suppresses the writing of files to disk, returning only basic information such as the evaluation metrics values.
             )

# Now we have a loop that runs the model iteratively, once per parameter set:
for i, X in enumerate(param_values):
    # We configure the model as we did in Notebook 04_Emulating_hydrological_models.
    m = GR4JCN(
               params=X.tolist(),  # Here is where we pass the parameter sets to the model, from the loop enumerator X.
               **config
               )

On my end it's a lot faster (a lot!) and worked on the first try.

[tlvu]

If the problem might be with the code and not SAlib and we do still want to add SAlib to our Jupyter env, can I go-live with the new Jupyter env that has SAlib?

[richardarsenault]

Hi! @tlvu Please do NOT go live with this just yet. I am giving a workshop to 25 people using PAVICS so ideally we want to change nothing until next monday, is that OK?

thanks!

[tlvu]

@richardarsenault It is an incremental build, meaning no changes to existing packages, so very safe, as describe in Ouranosinc/PAVICS-e2e-workflow-tests#128.

But you are right, there is no rush, let's finish the workshop first.

[richardarsenault]

Thanks, I know nothing about the infrastructure management part but IF were anything to fail, right now would be the worst possible time haha! thanks!

[coveralls]

Pull Request Test Coverage Report for Build 7186977203

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 7 unchanged lines in 3 files lost coverage.
• Overall coverage decreased (-0.2%) to 81.242%

---

Files with Coverage Reduction	New Missed Lines	%
ravenpy/config/parsers.py	1	74.67%
ravenpy/ravenpy.py	3	86.39%
ravenpy/utilities/testdata.py	3	73.51%

Totals
Change from base Build 7186734828:	-0.2%
Covered Lines:	3322
Relevant Lines:	4089

---

💛 - Coveralls

[huard]

Is this good to merge ?

[richardarsenault]

I updated the NB with the more efficient version of the code, and also removed the !pip install SALib code. Probably will fail until we install SALib directly.

[Zeitsperre]

@richardarsenault @tlvu

If you want to add any missing requirements to the environment-rtd.yml and the docs recipe in pyproject.toml, I'd be comfortable merging this. All we would need is salib in PAVICS, right?

[richardarsenault]

Yes, that is correct. They don't seem to have a conda package though, not sure if that is a problem?

https://salib.readthedocs.io/en/latest/user_guide/getting-started.html#installing-salib

[Zeitsperre]

@richardarsenault You sure? https://anaconda.org/conda-forge/salib

[richardarsenault]

nope, not sure anymore! They don't refer to it in their docs, so I assumed it wasn't the same. I stand corrected!

[huard]

Reviewed the nb. Added a progress bar, fixed one typo. Added salib to doc env.
Good to merge once tests pass.

[tlvu]

SAlib added to the stable Jupyter version on PAVICS since last week so no problem here.

But I remember there was a problem with random notebook execution hanging, was that fixed?

[richardarsenault]

I also recall this problem but I can't replicate it. I've just tried running it a whole bunch of times, restarting the kernel and testing again multiple times, with no such error. Perhaps we solved it either with @huard's code optimization or by installing SALib directly on the platform rather than the makeshit import we had before?

In any case, for now I can't replicate it. Maybe we revisit if a nightly run hangs?

[huard]

I believe it's been fixed by avoiding concurrent calls to a netcdf data extraction function.

[tlvu]

Tried a Jenkins, the new Sensitivity_analysis.ipynb passed.

However the HydroShare_integration.ipynb failed with authentication error. This used to work with any bogus token but now looks like they are enforcing proper authentication.

[huard]

I suggest we remove HydroShare_integration.ipynb for the time being.

1. To avoid authentication problems;
2. To avoid Pydantic compatibility issues.

[tlvu]

remove HydroShare_integration.ipynb for the time being.

I assume removing from the automated testing suite (Jenkins) and from being automatically deployed to PAVICS as tutorial-notebooks but still keep it somewhere in the repo.

One way to do this is to remove it from docs/notebooks/ and put it somewhere else, say docs/extra-notebooks/ or another location.

[huard]

Yes. in the pydantic v2 PR, i've removed it from the notebook index and excluded it from the sphinx build.