Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve OPeNDAP error message #1316

Closed
dopplershift opened this issue Feb 13, 2019 · 12 comments
Closed

Improve OPeNDAP error message #1316

dopplershift opened this issue Feb 13, 2019 · 12 comments

Comments

@dopplershift
Copy link
Member

If we try to access a non-existent URL, a seriously unhelpful error message results:

>ncdump -h https://thredds-test.unidata.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190211/OR_ABI-L1b-RadF-M3C13_G17_s20190421530382_e20190421541160_c20190421541201.nc
ncdump: https://thredds-test.unidata.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190211/OR_ABI-L1b-RadF-M3C13_G17_s20190421530382_e20190421541160_c20190421541201.nc: https://thredds-test.unidata.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190211/OR_ABI-L1b-RadF-M3C13_G17_s20190421530382_e20190421541160_c20190421541201.nc: NetCDF: Not a valid data type or _FillValue type mismatch


>curl https://thredds-test.unidata.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190211/OR_ABI-L1b-RadF-M3C13_G17_s20190421530382_e20190421541160_c20190421541201.nc
Error {
    code = 400;
    message = "Unrecognized request";
};

It's possible this is specific to hitting the 404 page from TDS. Regardless, having the library yield the OPeNDAP equivalent of "File Not Found" or "Invalid File" would be so much more helpful.
@dopplershift
Copy link
Member Author

dopplershift commented Feb 13, 2019 •
edited
Loading

Oh, I see now it's a 400, not a 404. Either way, I'd love a better error message.

@WardF
Copy link
Member

WardF commented Feb 13, 2019

@DennisHeimbigner is there an easy way to check the error code coming back? I could dig into this if so. I'm not sure if DAP is parsing error codes returned or if it's just checking for an error.

@lesserwhirls
Copy link
Contributor

You want to add .dods after the .nc, as well as some constraint. So something like:

curl https://thredds.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc.dods?y[0:1:5423]
Dataset {
    Int16 y[y = 5424];
} satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc;

That said, the issue is bigger than just the error message, as ncdump on the url works in 4.6.1, but not 4.6.2.

So:

netcdf library version 4.6.2 of Dec 17 2018 19:20:47

ncdump -h https://thredds.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc
ncdump: https://thredds.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc: https://thredds.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc: NetCDF: Not a valid data type or _FillValue type mismatch

netcdf library version 4.6.1 of Jan 27 2019 11:38:15

ncdump -h https://thredds.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc | less
sarms@flip:$ ncdump -h https://thredds.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc
netcdf OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201 {
dimensions:
	band = 1 ;
	num_star_looks = 24 ;
	number_of_image_bounds = 2 ;
	number_of_time_bounds = 2 ;
	x = 5424 ;
	y = 5424 ;
<snip>

@WardF
Copy link
Member

WardF commented Feb 13, 2019

So, the _FillType mismatch error is hasn't had coffee yet so recollection is hazy a fix we introduced in 4.6.2 for behavior that was documented but not previously enforced. Let me go back and dig through the PR's/Support emails to refresh my memory.

@WardF
Copy link
Member

WardF commented Feb 13, 2019

Ok, it looks like there was some work done to work around this change when using DAP: #1155

It clearly needs to be highlighted in the documentation.

@WardF
Copy link
Member

WardF commented Feb 13, 2019 •
edited
Loading

I can confirm that, for me, the following command works

$ ./ncdump -h [fillmismatch]https://thredds.ucar.edu/thredds/dodsC/satellite/goes17/GRB17/ABI/FullDisk/Channel13/20190213/OR_ABI-L1b-RadF-M3C13_G17_s20190441600381_e20190441611159_c20190441611201.nc

So perhaps this is a failure of documented DAP behavior, or at least indicates the need to highlight this new behavior. Or, based on conversation in my office with @dopplershift just now, perhaps [fillmismatch] needs to be implicitly true when performing DAP operations.

@DennisHeimbigner
Copy link
Collaborator

Correct. The netcdf spec requires the type of a _FillValue attribute to be
the same as the associated variable. We have started to enforce it.
However, if you suffix a url with '#fillmismatch', an attempt will be made
to forcibly convert the _FillValue value to the type of the variable.

@WardF
Copy link
Member

WardF commented Feb 13, 2019

Playing with the file locally, I do not get a mismatch error. Sitting in a meeting at the moment, but surprised at the different behavior, unless I've already been told and just forgot, in which case ah, yes, I remember.

@DennisHeimbigner
Copy link
Collaborator

The problem attribute is yaw_flip_flag:_FillValue. The type of the yaw_flip_flag variable is byte.
If I download the underlying .nc file using the httpserver protocol, then the type of
the _FillValue is also byte. However if I look at the _FillValue in the opendap .das, its type
is Int16.
I think there might be a consistent problem in the .nc -> DAP converter inside thredds.

@DennisHeimbigner
Copy link
Collaborator

The problem may be that since DAP2 does not support signed byte, the converter
widens to int16 to store the _FillValue.

@DennisHeimbigner
Copy link
Collaborator

Ok, am testing fix now in the netcdf-c library.

DennisHeimbigner added a commit that referenced this issue Feb 13, 2019
@DennisHeimbigner
Add hack to deal with DAP2 signed byte hack.
724128a 
re: issue #1316

The DAP2 data model does not have a signed byte type,
but netcdf-3 does have (only) a signed byte type.
So, when converting a netcdf-3 signed byte typed variable to
a DAP2 unsigned byte, the following actions are taken by thredds:
1. The variable type is marked as DAP2 (unsigned) byte.
2. A special attribute, "_Unsigned=false" is set for the variable
3. The corresponding "_FillValue" attribute, if any, is up-converted
   to the DAP2 Int16 type in order to hold, with sign, any signed byte
   fill value.

On the netcdf-c side, this looks like a fillvalue type mismatch and causes
an error. Instead, the netcdf-c dap2 conversion code needs to recognize
this hack and undo it locally.

So this change looks for the above case, and if found, then it properly
converts the _FillValue type to netcdf-3 signed byte.

Since DAP2 supports both signed and unsigned integers of sizes 16 and 32 bits,
this should be the only hack needed (famous last words).

It may later be desirable for the thredds DAP2 converter to modify its
behavior as well.
@DennisHeimbigner DennisHeimbigner mentioned this issue Feb 13, 2019
Merged
@DennisHeimbigner
Copy link
Collaborator

DennisHeimbigner commented Feb 13, 2019 •
edited
Loading

Fixed as discussedd: see #1317
The relevant branch is called fillhack.dmh

@WardF WardF closed this as completed Feb 13, 2019
@lesserwhirls lesserwhirls mentioned this issue May 30, 2019
Closed
@dopplershift dopplershift mentioned this issue Jan 30, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@lesserwhirls @dopplershift @DennisHeimbigner @WardF