Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle prhmax in cmorization #37

Open
ljoakim opened this issue Jun 11, 2024 · 9 comments · May be fixed by #56
Open

Handle prhmax in cmorization #37

ljoakim opened this issue Jun 11, 2024 · 9 comments · May be fixed by #56

Comments

@ljoakim
Copy link
Collaborator

ljoakim commented Jun 11, 2024

prhmax has a particular type of cell_method which is currently not handled by cmorization in tools.py (in code from here and downwards). This needs to be fixed, see discussion here: WCRP-CORDEX/discuss#7.

@gnikulin
Copy link
Collaborator

gnikulin commented Nov 4, 2024

seems prhmax is our last problematic variable because of cell_method area: mean time: mean within hours time: maximum over hours that should be hardcoded I think, @doblerone ?

It has been decided to delete this variable form the monthly frequency table, only left in the day table.

@doblerone
Copy link
Collaborator

What is the current behaviour for prhmax @ljoakim ?
Does it correctly read the daily cm_type from the table, i.e. area: mean time: mean within hours time: maximum over hours?

elif res=="day":
res_hr=24.
cm_type = params[config.get_config_value('index','INDEX_VAR_CM_DAY')]

(Debug info here)
logger.info("#########################")
logger.log(35," resolution: '%s'" % res)
logger.debug("cell method: '%s' " % (cm_type))
logger.info("#########################")

and write?

f_var.cell_methods = cm_type

And does the prhmax cmorization enter this if statement here?
(and what is cm there?)

# For monthly resolution daily processing is sometimes necessary first
elif res == 'day' or 'within days time' in cm_type:
cmd = "cdo -L -f %s -s day%s -seltimestep,%s %s %s %s" % (config.get_config_value('settings', 'cdo_nctype'),cm,tsteps,cmd_mul,in_file,ftmp_name)

This will not work I guess.

@doblerone
Copy link
Collaborator

It has been decided to delete this variable form the monthly frequency table, only left in the day table.

I have removed the monthly prhmax entry from the table
https://github.com/HCLIMcom/HCLIM2CMOR/blob/master/src/CMORlight/Config/CORDEX6_CMOR_HCLIM_variables_table.csv

@ljoakim
Copy link
Collaborator Author

ljoakim commented Nov 5, 2024

Here's an example log. I only added the last few lines of the stack trace, and the line number might be a little bit off since it was run a few commits ago:

  File "/home/sm_joalo/dev/repos/HCLIM2CMOR/src/CMORlight/tools.py", line 1447, in process_file
    retval = shell(cmd,logger=logger)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sm_joalo/dev/repos/HCLIM2CMOR/src/CMORlight/tools.py", line 44, in shell
    raise Exception('Shell Error: %s for command: %s' % (prc.communicate()[0], cmd) )
Exception: Shell Error: b'\ncdo (Abort): \ndaymean within \x1b[4mhours\x1b[0m time: maximum over hours -seltimestep,1/365 /nobackup/rossby27/proj/rossby/joint_exp/cordex/postprocess/HCLIM2CMOR/data_jl/work/input_CMORlight/EUR11_EUR11_ALADIN43_v1_ERA5_r1i1p1f1_eval/prhmax/prhmax_198501010030-198512312330.nc /nobackup/rossby27/proj/rossby/joint_exp/cordex/postprocess/HCLIM2CMOR/data_jl/temp/1985-a4c92db0-87e0-11ef-8fb8-7cc25546c4cb-prhmax.nc  \n                 ^ Operator cannot be assigned.\n       Reason:\n         No Operators with missing input left.\n\n' for command: cdo -L -f nc4c -s daymean within hours time: maximum over hours -seltimestep,1/365  /nobackup/rossby27/proj/rossby/joint_exp/cordex/postprocess/HCLIM2CMOR/data_jl/work/input_CMORlight/EUR11_EUR11_ALADIN43_v1_ERA5_r1i1p1f1_eval/prhmax/prhmax_198501010030-198512312330.nc /nobackup/rossby27/proj/rossby/joint_exp/cordex/postprocess/HCLIM2CMOR/data_jl/temp/1985-a4c92db0-87e0-11ef-8fb8-7cc25546c4cb-prhmax.nc

So it seems it goes into the elif at line 1440 above, with cm="mean within hours time: maximum over hours", which is part of the total cell method. This causes ncdo to abort. Probably the code a few lines above which extracts cm can not currently handle the complex cell method of prhmax.

@doblerone
Copy link
Collaborator

Thanks!
OK, so (as expected) it goes into the elif at line 1440 above, with cm="mean within hours time: maximum over hours", and causes cdo to abort.
We don't need to run an additional cdo command here because prhmax is already calculated as daymax in mergemon.sh.
So we need to find a clever way to skip this. A quick and dirty solution would be easy to implement (skip in case var=prhmax) but maybe we can come up with a more general one that handles cm_type=area: mean time: mean within hours time: maximum over hours

I'll have a look at it tomorrow.

@doblerone
Copy link
Collaborator

Alternatively, we could make a copy of hourly pr in mergemon.sh for the variable prhmax and then make a daymax in line 1440 to get the daily prhmax (analog tasmax etc.). I.e. we try to extract the "max" from the string time: mean within hours time: maximum over hours

@gnikulin
Copy link
Collaborator

and our final solution ? :-)

@doblerone
Copy link
Collaborator

I didn't have time to implement and test it. It's a nasty variable. I'd think the easiest and most flexible solution is to add another if statement here

# For monthly resolution daily processing is sometimes necessary first
elif res == 'day' or 'within days time' in cm_type:
cmd = "cdo -L -f %s -s day%s -seltimestep,%s %s %s %s" % (config.get_config_value('settings', 'cdo_nctype'),cm,tsteps,cmd_mul,in_file,ftmp_name)

e.g.
if res == 'day' and 'maximum over hours' in cm_type:
cmd = "cdo -L -f %s -s daymax -seltimestep,%s %s %s %s" %(config.get_config_value('settings','cdo_nctype'),tsteps,cmd_mul,in_file,ftmp_name)
else

This should work both if

  1. prhmax is prodced as daymax(pr_1h) in mergemon.sh as it is done now. Then the daymax is simply done twice.
  2. prhmax=daymax(pr_ah) is done in master_cmor.sh. Then the prhmax entry in the table https://github.com/HCLIMcom/HCLIM2CMOR/blob/master/src/CMORlight/Config/CORDEX6_CMOR_HCLIM_variables_table.csv needs to start with pr;pr;prhmax and hurly pr needs to be available in input_CMORlight

@ljoakim can you implement and test this? If you don't have the time, I can give it a try.

@ljoakim
Copy link
Collaborator Author

ljoakim commented Nov 14, 2024

I'll look into it tomorrow and test it.

@ljoakim ljoakim linked a pull request Nov 15, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants