Ensemble setting templates within ensemble stat configs

[robdarvell]

Not sure it was concluded which section this would go in yesterday, so I have just placed it within this one, but happy for it to be moved about.

General question about whether there are any alternative ways in which to set up the ensemble forecast template section in order to pick up the correct files.

There is a 'wildcard' option which you can specify and this works (eg: ??), but it seems to start at 1. The ensemble members that I am using start from 0, so for an 18 member ensemble it would go from 00-17. Using the wildcard it tries to go from 1-18 and thus have issues.

Alternative option I have used was to use the begin_end_incr option going from 0 (but then I ran into difficulty in setting the end result to be number of members -1 (I ended just creating a new variable which I just set to the one fewer than the number, but I wouldn't be surprised if you could do something with the original number)).

I did also examine using the custom option which worked only in part as well.

We also run the verification for different member counts, for example for 12 members and 18 members from the same ensemble list. Running for the 12 member list option ran into difficulty if all of the members were stored in the same directory as when behind the scenes it is creating the list it still thinks there are 18 members in the directory (even though the num_members is set to 12) and thus fails by saying I am not asking for enough data. I could get around that by storing each ensemble member within a different directory (which may not be the cleanest solution, I don't know at this point).

Was wondering whether or not there was any merit in say adding a new variable (optional) from which you can set your first ensemble member value to (defaults to 1 if missing) as a possible enhancement.

My current template setting for having things running looks like this:
FCST_ENSEMBLE_STAT_INPUT_TEMPLATE = ncdf_Memberbegin_end_incr(0,{ENSEMBLE_STAT_N_MEMBERS_TEST},1,2)/prods_op_mogreps-g_{init?fmt=%Y%m%d}{init?fmt=%H}??_{lead?fmt=%3H}.nc

Just wondering if people have any thoughts about this, or are we different when we start with member 0 rather than 1 (but I guess others might do in the future).
I won't comment that our members are only 0-17 for 00Z and 12Z for 06Z and 18Z they are 0,18-34 (but we tend to renumber things to create 0-17).

Hope this sort of makes sense to people?

Rob

[j-opatz]

Hi Rob,

Thanks for the inquiry. Based on the info you provided, I think @georgemccabe is best suited for helping find a solution for this. He'll be able to provide attention to this when he returns to the office next week.

[georgemccabe]

Hi Rob,

Could you send me an example directory listing so I can see how the files are named? That would help me determine the best approach to configuring your use cases.

There is a 'wildcard' option which you can specify and this works (eg: ??), but it seems to start at 1. The ensemble members that I am using start from 0, so for an 18 member ensemble it would go from 00-17. Using the wildcard it tries to go from 1-18 and thus have issues.

I have seen other ensembles with a 00 member, so that is not out of the ordinary. I am confused as to why your wildcard expression is skipping the 00 ensemble member. The string template tags are substituted first, then any wildcards are evaluated in the same way as they would if you are using them in an "ls" command. All files that match are found, so if the 00 ensemble is available on disk and matches the same name format as the other ensembles, then it should be found. If you need to be able to run a subset of the available ensemble members, then wildcards may not be the best approach, but I am curious as to why it is not finding the correct files.

Alternative option I have used was to use the begin_end_incr option going from 0 (but then I ran into difficulty in setting the end result to be number of members -1 (I ended just creating a new variable which I just set to the one fewer than the number, but I wouldn't be surprised if you could do something with the original number)).

The begin_end_incr() functionality doesn't support arithmetic expressions in the values, although that does seem like a useful addition. I can create a GitHub issue to implement it.

We also run the verification for different member counts, for example for 12 members and 18 members from the same ensemble list. Running for the 12 member list option ran into difficulty if all of the members were stored in the same directory as when behind the scenes it is creating the list it still thinks there are 18 members in the directory (even though the num_members is set to 12) and thus fails by saying I am not asking for enough data. I could get around that by storing each ensemble member within a different directory (which may not be the cleanest solution, I don't know at this point).

The ENSEMBLE_STAT_N_MEMBERS value is only used for error checking to ensure that it finds the correct number of ensembles. It will error if it finds more than the number. If there are fewer found, it will log a warning and fill in the list with empty files so that the application will still run. The empty files are ignored. You shouldn't have to move the files into different directories to select the correct files with the right configuration.

Was wondering whether or not there was any merit in say adding a new variable (optional) from which you can set your first ensemble member value to (defaults to 1 if missing) as a possible enhancement.

I'm not sure what you mean here. Are you suggesting to have a variable to specifies the first number in the list of available ensemble members?

[robdarvell]

I will keep having a play as well. It might be that my template string isn't operating as I had envisaged.

Further play: It is now working starting with the 00 ensemble member, so I have no idea now what I was doing previously in order for it not to find member 00. I did clear out a directory so that may have had something to do with it.

Thanks for the comment on the use of the ENSEMBLE_STAT_N_MEMBERS aspect, and it only being used for error checking. Does this suggest that I am going to need an alternative setup, rather than using the wildcards in order to process a reduced member list (the output is suggesting this, but I am thinking that the N_MEMBERS information might be an alterative method to use (but have not considered any of the potential pitfalls from this as yet))

[georgemccabe]

Hi Rob,

I'm glad the missing 00 ensemble issue has been resolved.

If you need to run a subset of the files within a directory, you would either have to get very clever with your wildcard expressions or generate a list of the members that you want to process. If using the latter approach, I think that using the N_MEMBERS info could likely be useful in configuring this use case.

Here is the GitHub issue for allowing arithmetic in the begin_end_incr syntax that would help if you use N_MEMBERS to determine the list to generate: #958
Let me know if you need this functionality in the near future and I can discuss with the team about increasing the priority of the task. Ironically the Python range function syntax uses an exclusive end value, which would produce the results you desire without any modifications. I implemented the begin_end_incr syntax to use an inclusive end value because I thought it was more user friendly to define a range of values. A solution that would be quicker to implement would be to add support for a function that mimics the Python range logic so that you could just use 0,{ENSEMBLE_STAT_N_MEMBERS}.

Let me know what you think about tasking or if you run into any other issues.