Error when opening pickup files with open_mdsdataset

[ds4g15]

Hi all,
I'm fairly new at this so this may be more an issue with me than the package, but I'm getting an AssertionError when trying to open the pickup files associated with ECCO v4r4

The files can be found here:
https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/input_init/pickup.0000000001.meta
https://ecco.jpl.nasa.gov/drive/files/Version4/Release4/input_init/pickup.0000000001.data

My hope is to open them, make some modifications and then save somewhere else, using the following lines:

import xmitgcm as xm
xm.open_mdsdataset('~/ECCOv4r4_input/input_init',prefix='pickup',iters=[1],grid_dir='.')   

But run into an error

~/miniconda3/envs/oce/lib/python3.8/site-packages/xmitgcm/utils.py in parse_meta_file(fname)                                                                                                                                                                                     
     61         flds['fldList'] = [re.match("'*(\w+)", g).groups()[0] for g in                                                                                                                                                                                                   
     62                            re.split("'\s+'", flds['fldList'])]                                                                                                                                                                                                           
---> 63         assert flds['nrecords'] == len(flds['fldList'])                                                                                                                                                                                                                  
     64     return flds                                                                                                                                                                                                                                                          
     65                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                 
AssertionError:  

The meta file shows nrecords=403 and nFlds=11, which seems to be what triggers the error:

 nDims = [   2 ];
 dimList = [
    90,    1,   90,
  1170,    1, 1170
 ];
 dataprec = [ 'float64' ];
 nrecords = [   403 ];
 timeStepNumber = [     192840 ];
 timeInterval = [  6.942240000000E+08 ];
 nFlds = [   11 ];
 fldList = {
 'Uvel    ' 'Vvel    ' 'Theta   ' 'Salt    ' 'GuNm1   ' 'GuNm2   ' 'GvNm1   ' 'GvNm2   ' 'EtaN    ' 'dEtaHdt ' 'EtaH    '
 };

Thanks for any help.

[rabernat]

Welcome @ds4g15 and thanks for the useful question! We should definitely be able to support this application, but unfortunately it's a bit tricky, as explained below.

Evidently our assumption in xmitgcm that flds['nrecords'] == len(flds['fldList']) is not correct for pickup files. (Contrast this with a standard mds file that comes out of the diagnostics package, for which the assumption is always true.) The difference is that the pickup file mixes 2D and 3D fields, and rather than have a 3D dimList, it lumps together both different fields and different vertical levels into nrecords. The bottom line is that the pickup .meta file is not sufficiently self-describing to know how to open the file. Which fields are 2D and which are 3D? You need to know some extra information about MITgcm to answer this question.

Fixing this would require generalizing the code to handle this more complex situation, plus hard-coding some information about the pickup variables.

Has anyone else successfully read pickup files with xmitgcm before?

[ds4g15]

Thanks a lot for the reply, that's super helpful insight into what's happening with the pickup files. I'll look out for this functionality in future but for now I've found a way around for my purposes by modifying the ECCO initialisation files. Shall I close the issue?

[rabernat]

No, please leave the issue open, as it remains unresolved. Glad you found a workaround, but we should still try to fix this in xmitgcm eventually.