[Bug]: File names are not correct when calculating variability indices for certain realizations.

[ShihengDuan]

What happened?

I'm using r*i1p1f1 in the parameter file to calculate variability indices for all realizations but with i1p1f1 (only change r numbers). This seems to work but the output file name does not follow the realization name:

metrics_results : /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF
Converting units by  divide 100.0
 -----  CanESM5  ---------------------
model_path: /p/css03/esgf_publish/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/psl/gn/v20190306/psl_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
 ---  r*i1p1f1  ---
model_lf_path: None
Warning: Data now ends at 2014 instead of 2015
Converting units by  divide 100.0
INFO::2024-07-17 14:40::pcmdi_metrics:: Results saved to a json file: /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF/var_mode_NAO_EOF1_stat_cmip6_historical_mo_atm_CanESM5_r*i1p1f1_1850-2014.json
2024-07-17 14:40:40,049 [INFO]: base.py(write:251) >> Results saved to a json file: /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF/var_mode_NAO_EOF1_stat_cmip6_historical_mo_atm_CanESM5_r*i1p1f1_1850-2014.json
2024-07-17 14:40:40,049 [INFO]: base.py(write:251) >> Results saved to a json file: /p/user_pub/climate_work/duan5/Noise/DuanPMP-New/CanESM5-psl-historical-CBF/var_mode_NAO_EOF1_stat_cmip6_historical_mo_atm_CanESM5_r*i1p1f1_1850-2014.json
model_path: /p/css03/esgf_publish/CMIP6/CMIP/CCCma/CanESM5/historical/r1i1p1f1/Amon/psl/gn/v20190429/psl_Amon_CanESM5_historical_r1i1p1f1_gn_185001-201412.nc
 ---  r*i1p1f1  ---
model_lf_path: None

The model path is correct with r1i1p1f1 but not the file names.

What did you expect to happen? Are there are possible answers you came across?

The file name should follow the model path name.

Minimal Complete Verifiable Example (MVCE)

No response

Relevant log output

No response

Anything else we need to know?

No response

Environment

pcmdi-metrics 3.3.4 pypi_0 pypi

[lee1043]

@ShihengDuan Thank you for reporting this. I am currently on vacation but should be able to work on this in a few weeks after I return. Do you have a workaround during the mean time?

[ShihengDuan]

I'm using * for now and it should be fine for most models. But for CanESM5, there are lots of ensembles with varying physics schemes (which we don't need). So it would be better if we can only get the ensembles with different initial conditions.

[ShihengDuan]

I tried to use xsearch and only select the ensemble members I want to keep. Here is a workaround that is slow but seems to work.

import sys
sys.path.insert(0, '/p/user_pub/xclim/persist/software/xsearch/')
import xsearch as xs
import argparse
import subprocess
import os

def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--model', type=str, default='CanESM5')
    parser.add_argument('--scenario', type=str, default='historical')
    parser.add_argument('--variability', type=str, choices=['PNA', 'NAM', 'SAM', 'NAO'])
    args = vars(parser.parse_args())
    return args

def modify_template_file(template_path, new_file_path, new_lines):
    # Read the content of the template file
    with open(template_path, 'r') as file:
        content = file.readlines()

    # Add new lines to the beginning
    new_content = new_lines + content

    # Save the modified content to a new file
    if os.path.exists(new_file_path):
        os.remove(new_file_path)
    with open(new_file_path, 'w') as file:
        file.writelines(new_content)


args = get_args()
model = args['model']
scenario = args['scenario']
variability = args['variability']
var = 'psl'
print('model: ', model, ' \nscenario: ', scenario)
files = list(xs.findPaths(scenario, var, 'mon', model = model).keys())
print(sorted(files))
print(len(files))
template = 'parameter_template_psl.py'
for file in files:
    member = file.split('/')[-6]
    # print(member)
    if 'i1p1f' in member:
        print(member)
        new_lines = [
            "scenario = "+"'"+scenario+"'\n",
            "gcm_name = "+"'"+model+"'\n",
            "realization = "+"'"+member+"'\n",
            "modpath = "+"'"+file+"'\n",
            "variability_mode = "+"'"+variability+"'\n"
        ]
        modify_template_file(template, variability+'-'+model+'-'+member+'.py', new_lines)

        command = f"variability_modes_driver.py -p "+variability+"-"+model+"-"+member+".py > "+variability+"-"+model+"-"+member+".out"
        subprocess.run(command, shell=True)
       

parameter_template_psl.py only contains parameters that are not relevant to the model and realizations (e.g., CBF, debug options, msyear)