Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ncdump fails to open a particular file #1151

Closed
WardF opened this issue Sep 24, 2018 · 8 comments · Fixed by #1155
Closed

ncdump fails to open a particular file #1151

WardF opened this issue Sep 24, 2018 · 8 comments · Fixed by #1155
Assignees
@DennisHeimbigner @WardF
Labels
area/DAP4
Milestone
4.6.2

Comments

@WardF
Copy link
Member

WardF commented Sep 24, 2018

As reported by @rsignell-usgs, the following command fails using either netCDF 4.6.2-rc1 or the older 4.4.1.

$ ncdump -h https://stellwagen.er.usgs.gov/thredds/dodsC/TSdata/MVCO_15/10571whV-a.nc

It returns an invalid argument error. Grabbing the file and opening it locally works just fine.
@WardF WardF added this to the 4.6.2 milestone Sep 24, 2018
@rsignell-usgs
Copy link

rsignell-usgs commented Sep 24, 2018 •
edited
Loading

cc @emontgomery-usgs, who reported the problem to me.

@DennisHeimbigner
Copy link
Collaborator

I am looking at it. Looks like a problem inside the DAP2 library. The File looks
ok to a superficial perusal.

@DennisHeimbigner
Copy link
Collaborator

Ok, I think the failure point is the variable "Or", and its "_FillValue" attribute.
The type of "Or" is int and the type of the "_Fillvalue" is float.
I could swear this problem has come up before, but I cannot find the issur or pull request
related to it.

@rsignell-usgs
Copy link

rsignell-usgs commented Sep 25, 2018 •
edited
Loading

@DennisHeimbigner, thanks for the sleuthing and this quick response!
We will change the _FillValue for the variable "Or" to int to match the int variable type

@DennisHeimbigner
Copy link
Collaborator

Ok. I see this occasionally. I do have a question: is it better to do the conversion
or to fail when this mismatch occurs?

@rsignell-usgs
Copy link

How about doing the conversion if the conversion works (e.g. if they converted value has the same numeric value) and failing if it doesn't.

So in this case you would fail be cause there is no way to store 1.e35 as int. But if the value was -9999 instead, it would not fail and be converted.

Does that make sense?

@DennisHeimbigner
Copy link
Collaborator

Doing that check will, of course, only work for atomic types, not user types.
Also, there is the perennial question of what does "equal" mean for floating point
values when, for example, the variable is of type float and the _FillValue is of type
double or UINT64 In that case, converting the double to a float and back to double to compare may cause truncation of least significant bits.
My alternate proposal is to give the user a way to allow or disallow such a mismatch
on a per-url request basis.

DennisHeimbigner added a commit that referenced this issue Oct 1, 2018
@DennisHeimbigner
Modify DAP2 and DAP4 to optionally allow Fillvalue/Variable mismatch
8072d1f 
re: issue #1151

Modify DAP2 and DAP4 code to handle case when _FillValue type is not
same as the parent variable type.

Specifically:
1. Define a parameter [fillmismatch] to allow this mismatch;
   default is to disallow.
2. If allowed, forcibly change the type of the _FillValue to match
   the parent variable.
3. If allowed Convert the values to match new type
4. Generate a log message
5. if not allowed, then fail

Implementing this required some changes to ncdap_test/dapcvt.c
Also added test cases.

Minor Unrelated Changes:
1. There were a number of warnings about e.g.
   assigning a const char* to a char*. Fix these
2. In nccopy.1, replace .NP with .IP "n"
   (re PR #1144)
3. fix minor error in ncdump/ocprint
@DennisHeimbigner DennisHeimbigner mentioned this issue Oct 1, 2018
Merged
@DennisHeimbigner
Copy link
Collaborator

Fixed by pull request #1155

@DennisHeimbigner DennisHeimbigner mentioned this issue Jan 31, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DennisHeimbigner DennisHeimbigner

@WardF WardF

Labels
area/DAP4
Projects
None yet
Milestone
4.6.2
Development

Successfully merging a pull request may close this issue.

Modify DAP2 and DAP4 to optionally allow Fillvalue/Variable mismatch Unidata/netcdf-c
3 participants
@DennisHeimbigner @rsignell-usgs @WardF