Shapes not aligned rrror when appending to a FileVariable in Python 3

[zshaheen]

This issue is only present in the Python 3 version of the code, not the Python 2 version.
When doing something like the example below, which concatenates data to a file, we get errors using Python 3.

out_file = cdms2.open(fnm, 'w')
for month_file_nm in monthly_files:
    month_file = cdms2.open(month_file_nm)
    data = month_file['FLNT']
   
    out_file.write(data)

The error is this:

Traceback (most recent call last):
  File "cdms_timeseries.py", line 69, in <module>
    out_file.write(data)
  File "/export/shaheen2/anaconda2/envs/acme_diags_env_dev_py3/lib/python3.6/site-packages/cdms2/dataset.py", line 2068, in write
    v[index:index + len(vec1)] = var.astype(v.dtype)
  File "/export/shaheen2/anaconda2/envs/acme_diags_env_dev_py3/lib/python3.6/site-packages/cdms2/fvariable.py", line 113, in __setitem__
    self._obj_.setitem(*(index, numpy.ma.filled(value)))
ValueError: shapes are not aligned

[dnadeau4]

@zshaheen Do you have a path to your files?

[zshaheen]

The files are located here: /p/user_pub/work/E3SM/1_0/piControl/1deg_atm_60-30km_ocean/atmos/129x256/model-output/mon/ens1/v1. You can access this dir from aims4, acme1, or any other machine that has the correct drive mounted.

[zshaheen]

@dnadeau4 I wonder if using 'a' instead of 'w' would work. Since we are appending to the file, shouldn't we use 'a' instead?

[doutriaux1]

@zshaheen we need w the first time because the file does not exists. If i remember correctly you are not closing the file in between calls to write. Beside it works for 2.0. Seems like a 3.0 issue.

[zshaheen]

Just going off on a small tangent, but when you use 'a', it gives the following error:

CDMS system error: No such file or directory
CDMS I/O error: Opening file /export/shaheen2/e3sm_diags_timeseries/cdat_climo_results/20180129.DECKv1b_piControl.ne30_oEC.edison_11_climo.nc

But does this mean that it still writes to the file?

Because I use 'a' as the default option for when I'm creating the climo using CDAT. I'd hope it's not messing up on something.

[dnadeau4]

@zshaheen this issue is about setitem which change from py2 to py3. It use to call setslice and CDMS now enter in the wrong function of Cdunif. I just have to remember how I fixed it last year for Transient Variables and do the same with File Variables.

[dnadeau4]

8209099

still need a test...

[dnadeau4]

@zshaheen can you send me the files for the test?

[zshaheen]

@dnadeau4 Wouldn't any files work? I would think so.

[dnadeau4]

I think so, but I wanted you to write the test to make sure it works the way you want it.

[dnadeau4]

@zshaheen Can you close this if it works for you?

[dnadeau4]

@zshaheen can you write test for this issue?

[dnadeau4]

@zshaheen can you write test please?

[zshaheen]

@dnadeau4, Apologies, I'll get to this now. We're moving over to Python 3 as our default dev env, so it's more important than ever.

[zshaheen]

@dnadeau4 After testing with both the Python 2 and Python 3 versions of cdms (builds py27h6091dcd_1 and py36h6091dcd_1), all is good. Thanks for fixing this Denis.

Below is the test I used. I'm not sure where to place this, but somewhere in test_cdmsfile_rw.py would probably work.

import cdms2
import vcs

vcs.download_sample_data_files()

files = [vcs.prefix + '/share/uvcdat/sample_data/clt.nc'] * 3

with cdms2.open('temp.nc', 'w') as out_file:
    for f in files:
        with cdms2.open(f) as input_file:
            data = input_file['clt']
            out_file.write(data)

[dnadeau4]

This pull request might fix it. I am waiting for the green light to merge 😄
#314

[dnadeau4]

failed due to circleci being slow for Dask. I reran the workflow.

[dnadeau4]

@zshaheen should be in nightly channel. Can you give it a try?