Python error on cheyenne when using rocoto to run METplus

[briannen]

Hi!

I am currently trying to run METplus on cheyenne using the rocoto workflow manager. I have a shell script that calls the run_metplus.py command that is being run by rocoto. I included user_env_vars.MET_PYTHON_EXE=/glade/p/ral/jntp/MET/METplus/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin/python in my call. The script runs perfectly when I run it manually, but I am getting the following error about the python version when running with rocoto.

• /glade/scratch/briannen/hafs/hafs_tne/scripts/run_gridstat_apcp.ksh
  ERROR: Must be using Python 3.6.3 or higher with the required packages installed. You are using 3.4.10.
  See the METplus documentation for more information.

I've tried adding the correct python version to my path and setting MET_PYTHON_EXE outside of the script, and neither are working.

Any ideas?

[JohnHalleyGotway]

Hi @briannen, hopefully I can help you work through this environment issue. Python is used in METplus in multiple places, and I suspect that's the source of this confusion:

1. METplus runs python wrappers to automate calls to the MET tools.
2. MET can run python embedding scripts to (a) ingest gridded data, (b) ingest point observations, or (c) ingest matched pair data into Stat-Analysis.

The error you're getting refers to the version of python used to run the METplus wrappers in (1). The MET_PYTHON_EXE environment variable relates to MET's python embedding logic in (2). You can read more about MET_PYTHON_EXE in Appendix F if you're curious.

So setting the MET_PYTHON_EXE environment variable won't have any impact on which version of python is used to execute the METplus wrappers. I will say this is a very, very common problem on HPC's. Something runs fine on the login node, but then doesn't run when submitting to the queuing system because of an environment issue. And in this case, the environment issue is the version of python that's being used on the compute node.

So how do we fix this? I'm guessing that your workflow may already have an environment setup step? For example, do you have some script you run to load modules and otherwise setup your environment? If so, that's where you'd address this issue. You'd also activate the correct miniconda environment in the same spot.

Please go take a look and let me know if you're still stuck. If so, I'll ask @georgemccabe to advise on best practices for setting up your METplus runtime environment when submitting jobs on HPC such as Cheyenne.

[briannen]

I do have a script that loads the following:

module use /glade/p/ral/jntp/MET/MET_releases/modulefiles
module load met/11.0.1

module use /glade/p/ral/jntp/MET/METplus/modulefiles
module load metplus/5.0.1

Now that I'm taking a look at it though, I'm not sure if it's even working because the met modules aren't listed when I run a module list afterwards.

[JohnHalleyGotway]

Bri, I believe that @michelleharrold and @KathrynNewman have a lot of experience correctly setting up their runtime environment for job submitted on Cheyenne. If you get stuck, I'm confident they could help unstick you. But it's pretty clear to me that this a runtime environment issue rather than a problem with the MET or METplus software itself.

[georgemccabe]

Hi @briannen, I confirmed that the module load command you sent do work, at least on a login node. Note that the metplus/5.0.1 file loads met/11.0.1, so you do not need to load MET explicitly. Both module files prepend the appropriate conda environment bin directory to the path, so the correct version of Python should be available when running METplus. I do not have experience running rocoto on cheyenne, so I can't speak to any differences in the environment that would result from running this way vs. the login node. You could try adding some commands after the module loads to verify the files that are in your path:

which python3
which run_metplus.py

One option to ensuring the correct version of Python is used is to call the version of Python you need explicitly when calling run_metplus.py. However, this approach isn't ideal because the version of Python that is used to run METplus through the module files may change in a future version and you would be stuck using an old version.

/glade/p/ral/jntp/MET/METplus/miniconda/miniconda3/envs/metplus_v5.0_py3.8/bin/python3 ${METPLUS_PATH}/ush/run_metplus.py \

[briannen]

Thanks @georgemccabe! Calling the run_metplus.py script with the correct version of python in my script that is being submitted by rocoto seems to have worked.

[georgemccabe]

I'm glad that fixed the issue! Keep in mind the version of Python needed for the upcoming METplus 5.1.0 will change so you will have to update the path to Python when you upgrade to that version unless you figure out how to properly get the version of Python from the path set by the module load command.

The new path to Python will be: /glade/p/ral/jntp/MET/METplus/miniconda/miniconda3/envs/metplus_v5.1_py3.10/bin/python3

I don't have experience submitting jobs with rocoto, but If you'd like to share the log output from the script I could see if there is something that suggests why the path isn't being set properly.

[briannen]

Sure! I'm not sure exactly what you need because I'm not sure where it logs any of the environment set up.

The log files from rocoto and the job are in: /glade/scratch/briannen/hafs/hafs_tne/expt_dirs_test/log
The rocoto xml that sets up the job submission is: /glade/scratch/briannen/hafs/hafs_tne/xml/imerg06h/hfsa/hafs_apcp_vx_imerg_hfsa_01l.xml
The script that is running is: /glade/scratch/briannen/hafs/hafs_tne/scripts/run_gridstat_apcp.ksh
My METplus conf files is: /glade/scratch/briannen/hafs/hafs_tne/parm/metplus/APCP_06h_imerg.conf

I'm currently troubleshooting another issue with the imerg HDF files, but hopefully this gives you an idea of how it's being submitted.

[georgemccabe]

It looks like you are no longer running the module commands in the script. They are commented out. Could you change these lines (5-6):

#module use /glade/p/ral/jntp/MET/METplus/modulefiles
#module load metplus/5.0.1

to

module purge
module use /glade/p/ral/jntp/MET/METplus/modulefiles
module load metplus/5.0.1
which run_metplus.py
which python3

and rerun?