errors in CMORizing ifs output for AerChemMIP

[tommibergman]

I am trying to do IFS cmorizaition for AerChemMIP piControl. Grib filtering goes through fine, but the problems start right after. I have four threads so I get two errors at the same time. I am not sure where to look for to solve the problems. First looks like a problem with missing data on timestep 1, but how to solve it? And second seems to be a problem with the frequency?

1. land_fx

2019-04-04 14:35:23 INFO:ece2cmor3.postproc: Post-processing target land in table fx from file /run/media/bergmant/NTFS/aerchemmip-spinup/ifs/018/ICMGGaspi+186710 with cdo command -setgridtype,regular -seltimestep,1 -selcode,172
^C('caught signal', <cdo.Cdo object at 0x7fb7ef1b74d0>, 2, <frame object at 0x55ada248e910>)
Error in calling operator setgridtype with:
>>> cdo -O -f nc -P 4 setgridtype,regular -seltimestep,1 -selcode,172 /run/media/bergmant/NTFS/aerchemmip-spinup/ifs/018/ICMGGaspi+186710 /nobackup/users/bergmant/tmp/aspi-ifs-1867/land_fx.nc<<<
STDOUT:
STDERR:cdo(2) seltimestep: Process started
cdo(3) selcode: Process started
Warning (cgribexScanTimestep2): Record 73 (id=201.128 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!
cdo(3) selcode: Open failed on >/run/media/bergmant/NTFS/aerchemmip-spinup/ifs/018/ICMGGaspi+186710<
Unsupported file structure

2019-04-04 15:02:12 ERROR:ece2cmor3.cdoapi: (returncode:-2) cdo(2) seltimestep: Process started
cdo(3) selcode: Process started
Warning (cgribexScanTimestep2): Record 73 (id=201.128 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!
cdo(3) selcode: Open failed on >/run/media/bergmant/NTFS/aerchemmip-spinup/ifs/018/ICMGGaspi+186710<
Unsupported file structure

2. ps on Amon table with monC

2019-04-04 20:38:37 ERROR:ece2cmor3.postproc: Unsupported combination of frequency monC with time operators [u'mean within years', u'mean over years'] encountered
2019-04-04 20:38:37 INFO:ece2cmor3.postproc: Post-processing target ps in table Amon from file /nobackup/users/bergmant/tmp/aspi-ifs-1867/134.128.1.3 with cdo command -setgridtype,regular -monmean -selcode,134

[goord]

Hi Tommi, I am working on this one...

[goord]

Hi Tommi, apparently ece2cmor3 is not using your ICMGGaspi+000000 file but instead your output file itself. Can it find the initial state output file?

[tommibergman]

I did not have it before, but now I can tell you that it finds it. Now I am faced with problem with cdo not having openMP support inside, and using --npp 1 still ends up with cdo -P 4. Any suggestions?

[goord]

Shoot, I didn't expose that variable... You could try to change it in the python code?

[tommibergman]

But there is also --ncdo, setting this to four seems to do the trick. Now testing a full year of ifs output.

[tommibergman]

From 132 identified tasks I get following 11 errors:

One before starting anything:

ERROR:ece2cmor3.ifs2cmor: Variable ta has unsupported combination of dimensions [u'latitude', u'plev39', u'time'] and will be skipped.

Nine just before grib_filter:

2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 185.128, level type 1, level 0. Dismissing task cltc in table AERmon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 130.128, level type 109, level -1. Dismissing task wa in table AERmon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 248.128, level type 109, level -1. Dismissing task cl in table Amon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 246.128, level type 109, level -1. Dismissing task clw in table Amon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 247.128, level type 109, level -1. Dismissing task cli in table Amon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 54.128, level type 109, level -1. Dismissing task pfull in table Amon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 130.128, level type 109, level -1. Dismissing task ta in table CFmon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 157.128, level type 109, level -1. Dismissing task hur in table CFmon
2019-05-16 09:46:34 ERROR:ece2cmor3.grib_filter: Field missing in the first day of file: code 133.128, level type 109, level -1. Dismissing task hus in table CFmon

And one during post-processing:

2019-05-16 10:22:49 ERROR:ece2cmor3.postproc: Unsupported combination of frequency monC with time operators [u'mean within years', u'mean over years'] encountered

For now I only use one year of output locally from EC-Earth-AerChem version spin-up, which I am using for testing cmorisation with the master branch.

Just asking are these errors common, so if someone else has reported them also? For me the nine errors there seem to lack the previous year data to extract first time step data. I guess this would be fixed when I have the preceding year downloaded as well.

[treerink]

Your first one for ta is also reported in #435 (comment).

[treerink]

Your last one, see comment in #431.

[jmrgonza]

Hi, I am having a similar using ece2cmor3 version 1.2.0 with output from my scenario run, also related to land fx:

2019-07-03 09:56:47 INFO:ece2cmor3.grib_filter: Filtering grib file /scratch/ms/es/spk/ECEARTH-RUNS/SSP5/output/ifs/167/ICMSHSSP5+201612...
2019-07-03 09:57:28 INFO:ece2cmor3.postproc: Post-processing target land in table fx from file /scratch/ms/es/spk/ECEARTH-RUNS/SSP5/output/ifs/167/ICMGGSSP5+201610 with cdo command -setgridtype,regular -seltimestep,1 -selcode,172
2019-07-03 09:57:28 ERROR:ece2cmor3.cdoapi: (returncode:1) cdo(2) seltimestep: Process started
cdo(3) selcode: Process started
Warning (cgribexScanTimestep2): Record 58 (id=49.128 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!
cdo(3) selcode: Open failed on >/scratch/ms/es/spk/ECEARTH-RUNS/SSP5/output/ifs/167/ICMGGSSP5+201610
<
Unsupported file structure

Should this be fixed by now? Could it be related to location of the ICMGG????+000000 file?
I am getting the same problem with the first and the sedond leg of the scenario run.

[treerink]

Yes looks like the ICMGG????+000000 file issue, so follow #430 (comment)

[jmrgonza]

Thank you for the confirmation Thomas,
I have just seen that I copied the ICM*+000000 files from the historical run, but forgot to rename them to match the new experiment, so that must be the problem.