CORDEX-CMIP6 cmor tables

[larsbuntemeyer]

Hey everybody,
i have some issues since i am working on CMOR tables for downscaling of CMIP6 datasets. I try to understand some of the attribute dependencies and i have some preliminary tables here and a rendered example. For theses tests i have basically adapted the CMIP6_CV table and modified them. This all works fine, however, in CORDEX we don't require any parent_* or sub_* attributes and i tried to remove them completely from the CV and the dataset table. But somehow i can't get it to work, since i always get something like:

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your "sub_experiment_id" key could not be found in
! your Control Vocabulary file.
! 
!
!!!!!!!!!!!!!!!!!!!!!!!!!

So the cmorizer looks for that attribute although it is not defined or required either in the dataset table nor in the CV table. Is there some special attributes for CMOR3 that are required? I suspect so when i have a look at the source code, e.g., here

cmor/Src/cmor_CV.c

Line 821 in c805fe0

int cmor_CV_checkSubExpID(cmor_CV_def_t * CV)

it seems like this is somehow hardcoded and required?

At the same time, i wonder how this works, e.g., with the input4MIps or obs4MIPs tables since they don't define, e.g., the sub_experiment attributes...

Maybe i get something totally wrong, i am not sure...

[larsbuntemeyer]

Ok, i now see that this behaviour is activated by the presence of the _cmip6_option keyword in the dataset table... So i guess this should be absent if one wants to define an arbitrary CV?

[matthew-mizielinski]

If I understand things correctly then I suggest that you are a little cautious when not using the _cmip6_option; when this is unset I think that CMOR will let a number of things through without checking (e.g. activity_id, experiment_id). There is certainly a place for the more relaxed approach, but I would be nervous if this was applied to large scale production as typos could get through.

It might be worth considering an extra if statement around the call to cmor_CV_checkSubExpID here to skip it if no sub_experiment_id field is required.

[larsbuntemeyer]

Thanks @matthew-mizielinski i didn't think of that yet, but it might be worth a try (especially after i now managed to setup a dev environment for cmor :))!

[larsbuntemeyer]

Just one hint for myself here:

If i have derived attributes in the CV, e.g., source depends on source_id, i have to specify both in the input datasets table (because i want both required), however, conflicting values of derived attributes are overwritten silently now. Could there be a warning if derived attributes are overwritten?

Is it correct that i have to specifiy derived required attributes in the dataset table (although theoretically they could be added automatically since they are overwritten anyway...)?

[durack1]

@larsbuntemeyer you're almost there, if you wanted to see some of the CMOR_input_example.json inputs for the input4MIPs project, take a peek at PCMDI/amipbcs/CMOR/drive_input4MIPs_obs.json.

As you noted this does NOT include the _cmip6_option and thanks to the recent CMOR 3.7.0 release, depends on most of the inputs from the registered content (in the input4MIPs case information found in the https://github.com/PCMDI/input4MIPs-cmor-tables (and specifically the info found in the PCMDI/input4MIPs-cmor-tables/Tables/input4MIPs_CV.json) file

[larsbuntemeyer]

Thanks @durack1 for that swift reply! I appreciate it! I guess, there is no way to configure PrePARE to run with arbitrary CV, is there? For example, could you run PrePARE on input4MIPs datasets?

[larsbuntemeyer]

@larsbuntemeyer you're almost there, if you wanted to see some of the CMOR_input_example.json inputs for the input4MIPs project, take a peek at PCMDI/amipbcs/CMOR/drive_input4MIPs_obs.json.

As you noted this does NOT include the _cmip6_option and thanks to the recent CMOR 3.7.0 release, depends on most of the inputs from the registered content (in the input4MIPs case information found in the https://github.com/PCMDI/input4MIPs-cmor-tables (and specifically the info found in the PCMDI/input4MIPs-cmor-tables/Tables/input4MIPs_CV.json) file

Yes, i updated to the latest version, now this makes the input file really slim and convenient, very nice!!

[durack1]

@larsbuntemeyer it makes sense to me that we migrate this "issue" into a "discussion" as there is no obvious issue or bug that this thread identifies, rather it's about CMOR usage (and potential expansion/features)

[larsbuntemeyer]

I agree!

[durack1]

@larsbuntemeyer regarding PrePARE usage, good question. We have been thinking about retooling this for validation across projects (input4MIPs, obs4MIPs, ..) in addition to expanding this to validate some file contents, rather than just metadata. We haven't expanded much past the thought at this stage - are you interested in retooling this for CORDEX?

[larsbuntemeyer]

I think, of course, having PrePARE being useful for different projects would be useful. However, I would be interested in being able to run a QA workflow before submitting datasets to ESGF. There was something like qa-dkrz that was particularly useful also for CORDEX but this is now deprecated in favour of the cfchecker. Anyway, with cmor3, i guess much of the checks are already run during cmorization and running cfchecker on top gives me some assurance.

Having that said, it would be of course quite nice to have some checks that also can validate more than just metadata, e.g., in CORDEX that would be quite useful to ensure consistent grids within regional domains or ensuring consistent units across source_ids. Having a tool that utilizes actual cmor tables for qa would also detect datasets that have been "messing aroung" with the tables. However, i guess, this also depends on which qa tools are run during ESGF publication.

[larsbuntemeyer]

I have one more issue regardin controlled vocabulary. For example, for CORDEX we use a required attribute like CORDEX_domain, in the CV table, it looks, e.g., like this:

{
    "CORDEX_domain": {
        "EUR-11": "Europe high-res.",
        "EUR-44": "Europe"
    }
}

However, it seems that i can still choose any value in my input file for the "CORDEX_domain" attribute while the cmorizer does not complain or warn. There is a different behaviour, e.g., for institution_id or source_id while grid_label seems also to be not checked agains the vocabulary. I see, e.g., that for input4MiPs, the CV also defines an attribute grid_label in the CV, while I suspect one could choose any value for that attribute during cmorization. The question is also, why are those values defined in the CV, if they don't define, e.g., derived attributes for automatic filling? Maybe i get something wrong here?

[larsbuntemeyer]

Ok, it seems to work if i put those allowed values into a list, e.g.,:

       "CORDEX_domain": [    
            "EUR-11",                                                                                                        
            "EUR-44"          
        ],             

Can i still define derived attributes like this? For example, i would like to define "domain_name" that depends on the CORDEX_domain attribute value... I did this before like this:

"CORDEX_domain":{
            "EUR-11": {
                "CORDEX_domain": "EUR-11",
                "domain_name": "Europe"
            }   
        },  

This will automaticall fill derived required attributes, e.g., domain_name and give a warning if i overwrite them, but again, i can put in arbitrary values for CORDEX_domain and domain_name in my input file. This behaviour seems a little bit counterintuitive..

[larsbuntemeyer]

@durack1 I am really curious in your opinion regarding the CORDEX_domain attribute. I am a little on the edge here whether i should use

"CORDEX_domain": [    
            "EUR-11",                                                                                                        
            "EUR-44"          
],       

which avoids people putting unknown values in their input file
or

"CORDEX_domain":{
            "EUR-11": {
                "CORDEX_domain": "EUR-11",
                "domain_name": "Europe"
            }   
        },  

which would automatically add derived attributes. However the latter one does not seem to check the attribute values...

[taylor13]

Regarding the example immediately above, does each CORDEX_domain have a unique domain_name? Or is domain_name meant to identify the rough vicinity covered by CORDEX_domain (which might sometimes apply to more than one CORDEX_domain, e.g., "Europe" might apply to both "EUR-11" and "EUR-44")? Or is the domain_name for EUR-11 "Europe Higher Resolution" and the name for EUR-44 "Europe Lower Resolution"?

[larsbuntemeyer]

Yes, actually i would like the option to have a unique domain_name value depending on the CORDEX_domain value, e.g.,

"CORDEX_domain":{
            "EUR-11": {
                "CORDEX_domain": "EUR-11",
                "domain_name": "Europe high-res."
            } ,
            "EUR-44": {
                "CORDEX_domain": "EUR-44",
                "domain_name": "Europe"
            } ,
            "EUR-22": {
                "CORDEX_domain": "EUR-22",
                "domain_name": "Europe Cordex Core"
            }   
        },  

[taylor13]

(The following was written without realizing that you are already attempting to define such a CV. Sorry about that.)

This, then, is analogous to the pairs of global attributes defined for CMIP6: (experiment_id, experiment), (source_id, source), (institution_id, institution), (sub_experiment_id, sub_experiment). As you probably know, these are defined by reference controlled vocabularies (CV's) stored in json files (e.g., for institution). Then, when the user supplies institution_id to CMOR, I think CMOR has already accessed the institution_id json file and can check whether institution_id is valid and what value of "institution" should be written. (The user doesn't have to provide CMOR with institution; only institution_id.)
I think CORDEX might consider defining a "domain" CV, which would include not only the list of CORDEX_domains, but also (some of?) the information currently found in Tables 1 and 2 of this document (see also this document).
I'm not sure how difficult it would be to generalize CMOR to ingest and make use of such a CV.

[larsbuntemeyer]

Thanks @taylor13 for that detailled explanation and all the support! Yes, i thought i could add some of that domain information to a CV file, i actually wanted to mirror the behaviour of, e.g., source_id. I have, e.g., CORDEX_domain and domain_name in my required_global_attributes list and tried different things, e.g.,

"CORDEX_domain":{
                "EUR-11": {       
                    "CORDEX_domain": "EUR-11",                                                                                                            
                    "domain_name": "Europe"
                },                                                                                                                                        
                "AFR-11": {       
                    "CORDEX_domain": "AFR-11",                                                                                                            
                    "domain_name": "Africa"                                                                                                               
                }                                                                                                                                         
        },
"domain_name": [                                                                                                                                  
          "Europe",
          "Africa"
        ],

which correctly controls the domain_name vocabulary and adds it if i define CORDEX_domain in my input file. However, i can still write in my input file:

 "CORDEX_domain" : "EUR-44",
 "domain_name" : "Africa",

and CMOR3 will neithter complain nor warn about the EUR-44 value although it's not in my CV. This is different from, e.g., the source_id attribute, although i tried to mirror that syntax in my CV. I just thougt this is a behaviour that could be useful for arbitrary required global attributes. I think, what i will do now, is something like this:

"CORDEX_domain": [                                                                                                                                  
          "EUR-11",
          "AFR-11"
        ],
"domain_name": [                                                                                                                                  
          "Europe",
          "Africa"
        ]

which correctly controls the values for CORDEX_domain and domain_name attributes, however, the combination of both can still be mixed up by users in the input file....

[durack1]

Apologies for being slow on response here.

Just to attempt to peel the onion and provide some context, and description, we MANAGE the controlled vocabularies (CVs) for CMIP6 in the WCRP-CMIP/CMIP6_CVs, but importantly, for CMOR, these are reformatted into the PCMDI/cmip6-cmor-tables/Tables/CMIP6_CV.json file which CMOR then uses for validation. I presume this was obvious? We do similar things with input4MIPs_CV.json and obs4MIPs_CV.json.

Regarding defining a CV for CMOR to check, unlike the python json library, CMOR is C and so the format of what is being provided in the *_CV.json file is far less nimble than standard python formatting.

@mauzey1 can you point us to the CMOR code that manages this CV validation? Including the CV read/load etc?

[larsbuntemeyer]

Thanks @durack1 ! I guess, i can manage the CORDEX CV using the approach above. Unfortunately, my C programming is little rusty. Maybe in the future, the behaviour of the source_id attribute could be expanded to more arbitrary attributes that follow the same structure in the CV as source_id...

[mauzey1]

Although CMOR uses the CV JSON files to read which attribute values are necessary for a variable in a MIP, it can only find the global attributes that it was programmed to find. This section of the cmor.h highlights the relevant variables that define the attributes CMOR looks for in a CV. It therefore cannot look for arbitrarily-defined attributes defined in a input file.

[larsbuntemeyer]

Thanks @mauzey1 for that insight! That's really valuable. It works fine now without _cmip6_option and the CORDEX required attributes (similar to the examples like obs4MIPs or input4MIPs provided by @durack1) and we can utilize the source_id attribute for our purposes. We just need to think about some QA without _cmip6_option that can use custom CVs (as PrepARE would have to be adapted as indicated in #685).