From c5ddd159cd57e6953f8ff0b755a4048919beca99 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Wed, 3 Nov 2021 15:41:17 -0600 Subject: [PATCH 01/12] Add bitgroom support to NCZarr re: PR https://github.com/Unidata/netcdf-c/pull/2088 Primary changes: * Add NCZarr-specific quantize function to the dispatch table. * Copy quantize code from libhdf5 * Add quantize invocation to zvar.c * Add support for _QuantizeBitgroomNumberOfSignificantDigits to ncgen. * Copy quantize test from nc_test4 to nczarr_tests. Remove some parts that are not relevant to NCZarr. Other Changes: * Break zsync.c into zsync.c (writing) and zload.c (reading). * Clean up the fill value handling (many changes) * Disable atexit() under Windows * Move ncjson to libdispatch * Add documentation of differences between netcdf-4 and NCZarr, especially WRT fill value. * Some mingw fixes * Remove some cruft * Cleanup the handling of scalars --- CMakeLists.txt | 7 +- RELEASE_NOTES.md | 2 + docs/nczarr.md | 23 +- libdispatch/dpathmgr.c | 10 +- libnczarr/CMakeLists.txt | 1 + libnczarr/Makefile.am | 1 + libnczarr/zarr.h | 1 + libnczarr/zattr.c | 6 +- libnczarr/zdispatch.c | 4 +- libnczarr/zdispatch.h | 5 +- libnczarr/zfile.c | 6 +- libnczarr/zinternal.c | 21 +- libnczarr/zinternal.h | 5 +- libnczarr/zload.c | 1542 +++++++++++++++++++++++++++++++++ libnczarr/zsync.c | 1557 +-------------------------------- libnczarr/ztracedispatch.h | 2 + libnczarr/zutil.c | 68 ++ libnczarr/zvar.c | 383 +++++++- libnczarr/zwalk.c | 18 +- libnczarr/zxcache.c | 24 +- ncgen/genbin.c | 5 + ncgen/genc.c | 15 + ncgen/ncgen.h | 3 + ncgen/ncgen.l | 3 +- ncgen/ncgen.y | 9 + ncgen/ncgenl.c | 1585 ++++++++++++++++++---------------- ncgen/ncgeny.c | 1241 +++++++++++++------------- ncgen/ncgeny.h | 5 +- nczarr_test/CMakeLists.txt | 3 + nczarr_test/Makefile.am | 6 +- nczarr_test/ref_filtered.cdl | 1 - nczarr_test/ref_purezarr.cdl | 1 - nczarr_test/ref_xarray.cdl | 1 - nczarr_test/run_interop.sh | 6 +- nczarr_test/run_quantize.sh | 26 + nczarr_test/test_quantize.c | 853 ++++++++++++++++++ plugins/Makefile.am | 2 +- 37 files changed, 4441 insertions(+), 3010 deletions(-) create mode 100644 libnczarr/zload.c create mode 100755 nczarr_test/run_quantize.sh create mode 100644 nczarr_test/test_quantize.c diff --git a/CMakeLists.txt b/CMakeLists.txt index 75cc5a752d..c759647e6e 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1715,10 +1715,15 @@ CHECK_FUNCTION_EXISTS(atexit HAVE_ATEXIT) # Control invoking nc_finalize at exit OPTION(ENABLE_ATEXIT_FINALIZE "Invoke nc_finalize at exit." ON) +IF(ENABLE_ATEXIT_FINALIZE) IF(NOT HAVE_ATEXIT) -IF(ENABLE_ATEXIT_FINALIZE AND NOT HAVE_ATEXIT) SET(ENABLE_ATEXIT_FINALIZE OFF CACHE BOOL "Enable ATEXIT" FORCE) MESSAGE(WARNING "ENABLE_ATEXIT_FINALIZE set but atexit() function not defined") +ELSE() + IF(MSVC) + SET(ENABLE_ATEXIT_FINALIZE OFF CACHE BOOL "Enable ATEXIT" FORCE) + MESSAGE(WARNING "ENABLE_ATEXIT_FINALIZE not supported under Windows") + ENDIF() ENDIF() ENDIF() diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 45bfd2ff76..0134a2e83d 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -7,10 +7,12 @@ This file contains a high-level description of this package's evolution. Release ## 4.8.2 - TBD +* [Enhancement] Add bitgroom support to NCZarr. See [Github #2???](https://github.com/Unidata/netcdf-c/pull/2???) * [Enhancement] Support byte-range reading of netcdf-3 files stored in private buckets in S3. See [Github #2134](https://github.com/Unidata/netcdf-c/pull/2134) * [Enhancement] Support Amazon S3 access for NCZarr. Also support use of the existing Amazon SDK credentials system. See [Github #2114](https://github.com/Unidata/netcdf-c/pull/2114) * [Bug Fix] Fix string allocation error in H5FDhttp.c. See [Github #2127](https://github.com/Unidata/netcdf-c/pull/2127). * [Bug Fix] Apply patches for ezxml and for selected oss-fuzz detected errors. See [Github #2125](https://github.com/Unidata/netcdf-c/pull/2125). +* [Enhancement] Support Amazon S3 access for NCZarr. Also support use of the existing Amazon SDK credentials system. See [Github #2114](https://github.com/Unidata/netcdf-c/pull/2114) * [Bug Fix] Ensure that internal Fortran APIs are always defined. See [Github #2098](https://github.com/Unidata/netcdf-c/pull/2098). * [Enhancement] Support filters for NCZarr. See [Github #2101](https://github.com/Unidata/netcdf-c/pull/2101) * [Bug Fix] Make PR 2075 long file name be idempotent. See [Github #2094](https://github.com/Unidata/netcdf-c/pull/2094). diff --git a/docs/nczarr.md b/docs/nczarr.md index af10387c60..1dae4afff3 100644 --- a/docs/nczarr.md +++ b/docs/nczarr.md @@ -293,6 +293,28 @@ Examples of currently unsupported types are as follows: Again, this list should diminish over time. +# NCZarr versus netCDF-4. {#nczarr_netcdf4} + +If ncgen is used to create both a netCDF-4 file and an NCZarr store using +the same .cdl file, then some differences may be observed. + +## _FillValue +The Zarr format stores the fill value as part of the .zarray metadata, +while netcdf-4 stores this in the _FillValue attribute. The .zattr for that +array may also contain the _FillValue attribute as well, so in NCZarr, the +fill value may occur in two places. + +The rule is that if nc_def_var_fill was called or the .cdl file defines the _FillValue attribute, +then that attribute will appear in the .zattr metadata, otherwise not. +However, if the fill_value key is defined, then it is used in place of the _FillValue attribute. + +If a Zarr store is read that was created by some other Zarr implementation, then +the the fill_value key may be set but there will probably not be any _FillValue attribute. +As above, then this value will be used. + +The net result is that NCZarr stores will carry the fill value and use it in subsequent +reads and writes. + # Notes on Debugging NCZarr Access {#nczarr_debug} The NCZarr support has a trace facility. @@ -320,7 +342,6 @@ aws_secret_access_key=YYYY... ``` See Appendix E for additional information. - ## Addressing Style The notion of "addressing style" may need some expansion. diff --git a/libdispatch/dpathmgr.c b/libdispatch/dpathmgr.c index 5335e89961..0d0a9e8d73 100644 --- a/libdispatch/dpathmgr.c +++ b/libdispatch/dpathmgr.c @@ -678,7 +678,7 @@ parsepath(const char* inpath, struct Path* path) && (tmp1[0] == '/') && strchr(windrive,tmp1[1]) != NULL && (tmp1[2] == '/' || tmp1[2] == '\0')) { - /* Assume this is a mingw path */ + /* Assume this is a msys path */ path->drive = tmp1[1]; /* Remainder */ if(tmp1[2] == '\0') @@ -869,11 +869,13 @@ static int getlocalpathkind(void) { int kind = NCPD_UNKNOWN; -#ifdef __CYGWIN__ +#if defined __CYGWIN__ kind = NCPD_CYGWIN; -#elif __MSYS__ +#elif defined __MINGW32__ + kind = NCPD_WIN; /* Do not understand the relationship of MSYS to MINGW */ +#elif defined __MSYS__ kind = NCPD_MSYS; -#elif _MSC_VER /* not _WIN32 */ +#elif defined _MSC_VER /* not _WIN32 */ kind = NCPD_WIN; #else kind = NCPD_NIX; diff --git a/libnczarr/CMakeLists.txt b/libnczarr/CMakeLists.txt index 59a3c0dfe9..2f931e83a8 100644 --- a/libnczarr/CMakeLists.txt +++ b/libnczarr/CMakeLists.txt @@ -28,6 +28,7 @@ zodom.c zopen.c zprov.c zsync.c +zload.c ztype.c zutil.c zvar.c diff --git a/libnczarr/Makefile.am b/libnczarr/Makefile.am index 8803e30dd8..3cf90ed8c8 100644 --- a/libnczarr/Makefile.am +++ b/libnczarr/Makefile.am @@ -46,6 +46,7 @@ zodom.c \ zopen.c \ zprov.c \ zsync.c \ +zload.c \ ztype.c \ zutil.c \ zvar.c \ diff --git a/libnczarr/zarr.h b/libnczarr/zarr.h index 8e531297aa..3a02177dba 100644 --- a/libnczarr/zarr.h +++ b/libnczarr/zarr.h @@ -72,6 +72,7 @@ EXTERNL int NCZ_create_fill_chunk(size64_t chunksize, size_t typesize, const voi EXTERNL int NCZ_s3clear(NCS3INFO* s3map); EXTERNL int NCZ_ischunkname(const char* name,char dimsep); EXTERNL char* NCZ_chunkpath(struct ChunkKey key); +EXTERNL int ncz_rebuild_fill_chunk(NC_VAR_INFO_T* var); /* zwalk.c */ EXTERNL int NCZ_read_chunk(int ncid, int varid, size64_t* zindices, void* chunkdata); diff --git a/libnczarr/zattr.c b/libnczarr/zattr.c index 33643e01d0..ce971bd58e 100644 --- a/libnczarr/zattr.c +++ b/libnczarr/zattr.c @@ -997,9 +997,10 @@ ncz_del_attr(NC_FILE_INFO_T* file, NC_OBJ* container, const char* name) } #endif -/* If we do not have a _FillValue, then go ahead and create it */ +#if 0 +/* If we do not have a _FillValue attribute, then go ahead and create it */ int -ncz_create_fillvalue(NC_VAR_INFO_T* var) +ncz_create_fillvalue_att(NC_VAR_INFO_T* var) { int stat = NC_NOERR; int i; @@ -1025,6 +1026,7 @@ ncz_create_fillvalue(NC_VAR_INFO_T* var) done: return THROW(stat); } +#endif /* Create an attribute; This is an abbreviated form of ncz_put_att above */ diff --git a/libnczarr/zdispatch.c b/libnczarr/zdispatch.c index 3add1431ac..ec94be766d 100644 --- a/libnczarr/zdispatch.c +++ b/libnczarr/zdispatch.c @@ -104,8 +104,8 @@ static const NC_Dispatch NCZ_dispatcher = { NC4_get_var_chunk_cache, NCZ_inq_var_filter_ids, NCZ_inq_var_filter_info, - NC_NOTNC4_def_var_quantize, - NC_NOTNC4_inq_var_quantize, + NCZ_def_var_quantize, + NCZ_inq_var_quantize, }; const NC_Dispatch* NCZ_dispatch_table = NULL; /* moved here from ddispatch.c */ diff --git a/libnczarr/zdispatch.h b/libnczarr/zdispatch.h index fdbd03828a..1f30c7200f 100644 --- a/libnczarr/zdispatch.h +++ b/libnczarr/zdispatch.h @@ -173,9 +173,8 @@ EXTERNL int NCZ_def_var_filter(int ncid, int varid, unsigned int filterid, size_ EXTERNL int NCZ_inq_var_filter_ids(int ncid, int varid, size_t* nfiltersp, unsigned int *filterids); EXTERNL int NCZ_inq_var_filter_info(int ncid, int varid, unsigned int filterid, size_t* nparamsp, unsigned int *params); -EXTERNL int NCZ_def_var_filterx(int ncid, int varid, const char* text); -EXTERNL int NCZ_inq_var_filterx_ids(int ncid, int varid, char** textp); -EXTERNL int NCZ_inq_var_filterx_info(int ncid, int varid, const char* id, char** textp); +EXTERNL int NCZ_def_var_quantize(int ncid, int varid, int quantize_mode, int nsd); +EXTERNL int NCZ_inq_var_quantize(int ncid, int varid, int *quantize_modep, int *nsdp); #if defined(__cplusplus) } diff --git a/libnczarr/zfile.c b/libnczarr/zfile.c index e0a90936fe..1385c2faec 100644 --- a/libnczarr/zfile.c +++ b/libnczarr/zfile.c @@ -115,11 +115,13 @@ NCZ_enddef(int ncid) var = (NC_VAR_INFO_T *)ncindexith(g->vars, j); assert(var); /* set the fill value and _FillValue attribute */ - if((stat = ncz_get_fill_value(h5,var,NULL))) goto done; /* ensure var->fill_value is set */ + if((stat = ncz_ensure_fill_value(var))) goto done; /* ensure var->fill_value is set */ assert(var->fill_value != NULL); var->written_to = NC_TRUE; /* mark it written */ - /* rebuild the fill chunk */ + /* ensure cache is correct */ if((stat = NCZ_adjust_var_cache(var))) goto done; + /* rebuild the fill chunk */ + if((stat = ncz_rebuild_fill_chunk(var))) goto done; /* Build the filter working parameters for any filters */ if((stat = NCZ_filter_setup(var))) goto done; } diff --git a/libnczarr/zinternal.c b/libnczarr/zinternal.c index 0ba9f76287..74457f8d8e 100644 --- a/libnczarr/zinternal.c +++ b/libnczarr/zinternal.c @@ -636,19 +636,19 @@ ncz_find_grp_var_att(int ncid, int varid, const char *name, int attnum, * @internal What fill value should be used for a variable? * Side effects: set as default if necessary and build _FillValue attribute. * - * @param h5 Pointer to file info struct. * @param var Pointer to variable info struct. - * @param fillp Pointer that gets pointer to fill value. + * @param fillp Pointer that gets pointer to fill value; do not free * * @returns NC_NOERR No error. * @returns NC_ENOMEM Out of memory. * @author Ed Hartnett, Dennis Heimbigner */ int -ncz_get_fill_value(NC_FILE_INFO_T *h5, NC_VAR_INFO_T *var, void **fillp) +ncz_ensure_fill_value(NC_VAR_INFO_T *var) { size_t size; int retval = NC_NOERR; + NC_FILE_INFO_T* h5 = NULL; #if 0 /*LOOK*/ /* Find out how much space we need for this type's fill value. */ @@ -659,7 +659,8 @@ ncz_get_fill_value(NC_FILE_INFO_T *h5, NC_VAR_INFO_T *var, void **fillp) else #endif { - if ((retval = nc4_get_typelen_mem(h5, var->type_info->hdr.id, &size))) goto done; + h5 = var->container->nc4_info; + if ((retval = nc4_get_typelen_mem(h5, var->type_info->hdr.id, &size))) goto done; } assert(size); @@ -695,7 +696,6 @@ ncz_get_fill_value(NC_FILE_INFO_T *h5, NC_VAR_INFO_T *var, void **fillp) fv_vlen->len = in_vlen->len; if (!(fv_vlen->p = malloc(basetypesize * in_vlen->len))) { - free(*fillp); *fillp = NULL; return NC_ENOMEM; } @@ -712,17 +712,6 @@ ncz_get_fill_value(NC_FILE_INFO_T *h5, NC_VAR_INFO_T *var, void **fillp) } } #endif /*0*/ - /* Create _FillValue Attribute */ - if((retval = ncz_create_fillvalue(var))) goto done; - if(fillp) { - void* fill = NULL; - /* Allocate the return space. */ - if((fill = calloc(1, size))==NULL) - {retval = NC_ENOMEM; goto done;} - memcpy(fill, var->fill_value, size); - *fillp = fill; - fill = NULL; - } done: return retval; diff --git a/libnczarr/zinternal.h b/libnczarr/zinternal.h index 471bf2f1b0..37f30012c7 100644 --- a/libnczarr/zinternal.h +++ b/libnczarr/zinternal.h @@ -225,7 +225,7 @@ int NCZ_initialize(void); int NCZ_finalize(void); int NCZ_initialize_internal(void); int NCZ_finalize_internal(void); -int ncz_get_fill_value(NC_FILE_INFO_T* file, NC_VAR_INFO_T* var, void **fillp); +int ncz_ensure_fill_value(NC_VAR_INFO_T* var); int ncz_find_grp_var_att(int ncid, int varid, const char *name, int attnum, int use_name, char *norm_name, NC_FILE_INFO_T** file, NC_GRP_INFO_T** grp, NC_VAR_INFO_T** var, @@ -245,12 +245,13 @@ int ncz_close_ncz_file(NC_FILE_INFO_T* file, int abort); /* zattr.c */ int ncz_getattlist(NC_GRP_INFO_T *grp, int varid, NC_VAR_INFO_T **varp, NCindex **attlist); -int ncz_create_fillvalue(NC_VAR_INFO_T* var); +int ncz_create_fillvalue_att(NC_VAR_INFO_T* var); int ncz_makeattr(NC_OBJ*, NCindex* attlist, const char* name, nc_type typid, size_t len, void* values, NC_ATT_INFO_T**); /* zvar.c */ int ncz_gettype(NC_FILE_INFO_T*, NC_GRP_INFO_T*, int xtype, NC_TYPE_INFO_T** typep); int ncz_find_default_chunksizes2(NC_GRP_INFO_T *grp, NC_VAR_INFO_T *var); +int NCZ_ensure_quantizer(int ncid, NC_VAR_INFO_T* var); /* Undefined */ /* Find var, doing lazy var metadata read if needed. */ diff --git a/libnczarr/zload.c b/libnczarr/zload.c new file mode 100644 index 0000000000..b62bfb8975 --- /dev/null +++ b/libnczarr/zload.c @@ -0,0 +1,1542 @@ +/********************************************************************* + * Copyright 1993, UCAR/Unidata + * See netcdf/COPYRIGHT file for copying and redistribution conditions. + *********************************************************************/ + +#include "zincludes.h" +#include "zfilter.h" + +#undef FILLONCLOSE + +/* Forward */ +static int load_jatts(NCZMAP* map, NC_OBJ* container, int nczarrv1, NCjson** jattrsp, NClist** atypes); +static int zconvert(nc_type typeid, size_t typelen, void* dst, NCjson* src); +static int computeattrinfo(const char* name, NClist* atypes, NCjson* values, + nc_type* typeidp, size_t* typelenp, size_t* lenp, void** datap); +static int parse_group_content(NCjson* jcontent, NClist* dimdefs, NClist* varnames, NClist* subgrps); +static int parse_group_content_pure(NCZ_FILE_INFO_T* zinfo, NC_GRP_INFO_T* grp, NClist* varnames, NClist* subgrps); +static int define_grp(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp); +static int define_dims(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* diminfo); +static int define_vars(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* varnames); +static int define_subgrps(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* subgrpnames); +static int searchvars(NCZ_FILE_INFO_T*, NC_GRP_INFO_T*, NClist*); +static int searchsubgrps(NCZ_FILE_INFO_T*, NC_GRP_INFO_T*, NClist*); +static int locategroup(NC_FILE_INFO_T* file, size_t nsegs, NClist* segments, NC_GRP_INFO_T** grpp); +static int createdim(NC_FILE_INFO_T* file, const char* name, size64_t dimlen, NC_DIM_INFO_T** dimp); +static int parsedimrefs(NC_FILE_INFO_T*, NClist* dimnames, size64_t* shape, NC_DIM_INFO_T** dims, int create); +static int decodeints(NCjson* jshape, size64_t* shapes); +static int computeattrdata(nc_type* typeidp, NCjson* values, size_t* typelenp, size_t* lenp, void** datap); +static int inferattrtype(NCjson* values, nc_type* typeidp); +static int mininttype(unsigned long long u64, int negative); +static int computedimrefs(NC_FILE_INFO_T* file, NC_VAR_INFO_T* var, int purezarr, int xarray, int ndims, NClist* dimnames, size64_t* shapes, NC_DIM_INFO_T** dims); + +/**************************************************/ +/**************************************************/ +/* Load the metadata from store into memory */ + +/** + * @internal Read file data from map to memory. + * + * @param file Pointer to file info struct. + * + * @return ::NC_NOERR No error. + * @author Dennis Heimbigner + */ +int +ncz_read_file(NC_FILE_INFO_T* file) +{ + int stat = NC_NOERR; + NCjson* json = NULL; + + LOG((3, "%s: file: %s", __func__, file->controller->path)); + + /* _nczarr should already have been read in ncz_open_dataset */ + + /* Now load the groups starting with root */ + if((stat = define_grp(file,file->root_grp))) + goto done; + +done: + NCJreclaim(json); + return THROW(stat); +} + +/** +@internal Extract attributes from a group or var and return +the corresponding NCjson dict. +@param map - [in] the map object for storage +@param container - [in] the containing object +@param jattrsp - [out] the json for .zattrs +@param jtypesp - [out] the json for .ztypes +@return NC_NOERR +@author Dennis Heimbigner +*/ +static int +load_jatts(NCZMAP* map, NC_OBJ* container, int nczarrv1, NCjson** jattrsp, NClist** atypesp) +{ + int i,stat = NC_NOERR; + char* fullpath = NULL; + char* key = NULL; + NCjson* jnczarr = NULL; + NCjson* jattrs = NULL; + NCjson* jncattr = NULL; + NClist* atypes = NULL; /* envv list */ + + /* alway return (possibly empty) list of types */ + atypes = nclistnew(); + + if(container->sort == NCGRP) { + NC_GRP_INFO_T* grp = (NC_GRP_INFO_T*)container; + /* Get grp's fullpath name */ + if((stat = NCZ_grpkey(grp,&fullpath))) + goto done; + } else { + NC_VAR_INFO_T* var = (NC_VAR_INFO_T*)container; + /* Get var's fullpath name */ + if((stat = NCZ_varkey(var,&fullpath))) + goto done; + } + + /* Construct the path to the .zattrs object */ + if((stat = nczm_concat(fullpath,ZATTRS,&key))) + goto done; + + /* Download the .zattrs object: may not exist */ + switch ((stat=NCZ_downloadjson(map,key,&jattrs))) { + case NC_NOERR: break; + case NC_EEMPTY: stat = NC_NOERR; break; /* did not exist */ + default: goto done; /* failure */ + } + nullfree(key); key = NULL; + + if(jattrs != NULL) { + if(nczarrv1) { + /* Construct the path to the NCZATTRS object */ + if((stat = nczm_concat(fullpath,NCZATTRS,&key))) goto done; + /* Download the NCZATTRS object: may not exist if pure zarr or using deprecated name */ + stat=NCZ_downloadjson(map,key,&jncattr); + if(stat == NC_EEMPTY) { + /* try deprecated name */ + nullfree(key); key = NULL; + if((stat = nczm_concat(fullpath,NCZATTRDEP,&key))) goto done; + stat=NCZ_downloadjson(map,key,&jncattr); + } + } else {/* Get _NCZARR_ATTRS from .zattrs */ + stat = NCJdictget(jattrs,NCZ_V2_ATTR,&jncattr); + } + nullfree(key); key = NULL; + switch (stat) { + case NC_NOERR: break; + case NC_EEMPTY: stat = NC_NOERR; jncattr = NULL; break; + default: goto done; /* failure */ + } + if(jncattr != NULL) { + NCjson* jtypes = NULL; + /* jncattr attribute should be a dict */ + if(NCJsort(jncattr) != NCJ_DICT) {stat = THROW(NC_ENCZARR); goto done;} + /* Extract "types; may not exist if only hidden attributes are defined */ + if((stat = NCJdictget(jncattr,"types",&jtypes))) goto done; + if(jtypes != NULL) { + if(NCJsort(jtypes) != NCJ_DICT) {stat = THROW(NC_ENCZARR); goto done;} + /* Convert to an envv list */ + for(i=0;i= NC_STRING) + {stat = NC_EINTERNAL; goto done;} + if((stat = computeattrdata(&typeid, values, &typelen, &len, &data))) goto done; + + if(typeidp) *typeidp = typeid; + if(lenp) *lenp = len; + if(typelenp) *typelenp = typelen; + if(datap) {*datap = data; data = NULL;} + +done: + nullfree(data); + return THROW(stat); +} + +/* +Extract data for an attribute +*/ +static int +computeattrdata(nc_type* typeidp, NCjson* values, size_t* typelenp, size_t* lenp, void** datap) +{ + int stat = NC_NOERR; + size_t count; + void* data = NULL; + size_t typelen; + nc_type typeid = NC_NAT; + int reclaimvalues = 0; + + /* Get assumed type */ + if(typeidp) typeid = *typeidp; + if(typeid == NC_NAT) if((stat = inferattrtype(values,&typeid))) goto done; + if(typeid == NC_NAT) {stat = NC_EBADTYPE; goto done;} + + /* Collect the length of the attribute; might be a singleton */ + switch (NCJsort(values)) { + case NCJ_DICT: stat = NC_ENCZARR; goto done; + case NCJ_ARRAY: + count = NCJlength(values); + break; + case NCJ_STRING: /* requires special handling as an array of characters; also look out for empty string */ + if(typeid == NC_CHAR) { + count = strlen(NCJstring(values)); + if(count == 0) count = 1; /* Actually a single nul char, probably default fill value ugh!*/ + } else + count = 1; + break; + default: + count = 1; /* singleton */ + break; + } + + if(count > 0) { + /* Allocate data space */ + if((stat = NC4_inq_atomic_type(typeid, NULL, &typelen))) + goto done; + if(typeid == NC_CHAR) + data = malloc(typelen*(count+1)); + else + data = malloc(typelen*count); + if(data == NULL) + {stat = NC_ENOMEM; goto done;} + /* convert to target type */ + if((stat = zconvert(typeid, typelen, data, values))) + goto done; + } + if(lenp) *lenp = count; + if(typelenp) *typelenp = typelen; + if(datap) {*datap = data; data = NULL;} + if(typeidp) *typeidp = typeid; /* return possibly inferred type */ + +done: + if(reclaimvalues) NCJreclaim(values); /* we created it */ + nullfree(data); + return THROW(stat); +} + +static int +inferattrtype(NCjson* value, nc_type* typeidp) +{ + int stat = NC_NOERR; + nc_type typeid; + NCjson* j = NULL; + unsigned long long u64; + long long i64; + int negative = 0; + + if(NCJsort(value) == NCJ_ARRAY && NCJlength(value) == 0) + {typeid = NC_NAT; goto done;} + + if(NCJsort(value) == NCJ_NULL) + {typeid = NC_NAT; goto done;} + + if(value->sort == NCJ_ARRAY) { + j=NCJith(value,0); + return inferattrtype(j,typeidp); + } + + switch (NCJsort(value)) { + case NCJ_NULL: + typeid = NC_CHAR; + return NC_NOERR; + case NCJ_DICT: /* fall thru */ + case NCJ_UNDEF: + return NC_EINVAL; + default: /* atomic */ + break; + } + if(NCJstring(value) != NULL) + negative = (NCJstring(value)[0] == '-'); + switch (value->sort) { + case NCJ_INT: + if(negative) { + sscanf(NCJstring(value),"%lld",&i64); + u64 = (unsigned long long)i64; + } else + sscanf(NCJstring(value),"%llu",&u64); + typeid = mininttype(u64,negative); + break; + case NCJ_DOUBLE: + typeid = NC_DOUBLE; + break; + case NCJ_BOOLEAN: + typeid = NC_UBYTE; + break; + case NCJ_STRING: /* requires special handling as an array of characters */ + typeid = NC_CHAR; + break; + default: + stat = NC_ENCZARR; + } +done: + if(typeidp) *typeidp = typeid; + return stat; +} + +static int +mininttype(unsigned long long u64, int negative) +{ + long long i64 = (long long)u64; /* keep bit pattern */ + if(!negative && u64 >= NC_MAX_INT64) return NC_UINT64; + if(i64 < 0) { + if(i64 >= NC_MIN_BYTE) return NC_BYTE; + if(i64 >= NC_MIN_SHORT) return NC_SHORT; + if(i64 >= NC_MIN_INT) return NC_INT; + return NC_INT64; + } + if(i64 <= NC_MAX_BYTE) return NC_BYTE; + if(i64 <= NC_MAX_UBYTE) return NC_UBYTE; + if(i64 <= NC_MAX_SHORT) return NC_SHORT; + if(i64 <= NC_MAX_USHORT) return NC_USHORT; + if(i64 <= NC_MAX_INT) return NC_INT; + if(i64 <= NC_MAX_UINT) return NC_UINT; + return NC_INT64; +} + +/** + * @internal Read group data from map to memory + * + * @param file Pointer to file struct + * @param grp Pointer to grp struct + * + * @return ::NC_NOERR No error. + * @author Dennis Heimbigner + */ +static int +define_grp(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp) +{ + int stat = NC_NOERR; + NCZ_FILE_INFO_T* zinfo = NULL; + NCZMAP* map = NULL; + char* fullpath = NULL; + char* key = NULL; + NCjson* json = NULL; + NCjson* jgroup = NULL; + NCjson* jdict = NULL; + NClist* dimdefs = nclistnew(); + NClist* varnames = nclistnew(); + NClist* subgrps = nclistnew(); + int purezarr = 0; + int v1 = 0; + + LOG((3, "%s: dims: %s", __func__, key)); + + zinfo = file->format_file_info; + map = zinfo->map; + + /* Construct grp path */ + if((stat = NCZ_grpkey(grp,&fullpath))) + goto done; + + if(zinfo->controls.flags & FLAG_PUREZARR) { + if((stat = parse_group_content_pure(zinfo,grp,varnames,subgrps))) + goto done; + purezarr = 1; + } else { /*!purezarr*/ + if(zinfo->controls.flags & FLAG_NCZARR_V1) { + /* build NCZGROUP path */ + if((stat = nczm_concat(fullpath,NCZGROUP,&key))) + goto done; + /* Read */ + jdict = NULL; + stat=NCZ_downloadjson(map,key,&jdict); + v1 = 1; + } else { + /* build ZGROUP path */ + if((stat = nczm_concat(fullpath,ZGROUP,&key))) + goto done; + /* Read */ + switch (stat=NCZ_downloadjson(map,key,&jgroup)) { + case NC_NOERR: /* we read it */ + /* Extract the NCZ_V2_GROUP dict */ + if((stat = NCJdictget(jgroup,NCZ_V2_GROUP,&jdict))) goto done; + break; + case NC_EEMPTY: /* does not exist, use search */ + if((stat = parse_group_content_pure(zinfo,grp,varnames,subgrps))) + goto done; + purezarr = 1; + break; + default: goto done; + } + } + nullfree(key); key = NULL; + if(jdict) { + /* Pull out lists about group content */ + if((stat = parse_group_content(jdict,dimdefs,varnames,subgrps))) + goto done; + } + } + + if(!purezarr) { + /* Define dimensions */ + if((stat = define_dims(file,grp,dimdefs))) goto done; + } + + /* Define vars taking xarray into account */ + if((stat = define_vars(file,grp,varnames))) goto done; + + /* Define sub-groups */ + if((stat = define_subgrps(file,grp,subgrps))) goto done; + +done: + if(v1) NCJreclaim(jdict); + NCJreclaim(json); + NCJreclaim(jgroup); + nclistfreeall(dimdefs); + nclistfreeall(varnames); + nclistfreeall(subgrps); + nullfree(fullpath); + nullfree(key); + return THROW(stat); +} + + +/** +@internal Read attributes from a group or var and create a list +of annotated NC_ATT_INFO_T* objects. This will process +_NCProperties attribute specially. +@param zfile - [in] the containing file (annotation) +@param container - [in] the containing object +@return NC_NOERR +@author Dennis Heimbigner +*/ +int +ncz_read_atts(NC_FILE_INFO_T* file, NC_OBJ* container) +{ + int stat = NC_NOERR; + int i; + char* fullpath = NULL; + char* key = NULL; + NCZ_FILE_INFO_T* zinfo = NULL; + NCZMAP* map = NULL; + NC_ATT_INFO_T* att = NULL; + NCindex* attlist = NULL; + NCjson* jattrs = NULL; + NClist* atypes = NULL; + nc_type typeid; + size_t len, typelen; + void* data = NULL; + NC_ATT_INFO_T* fillvalueatt = NULL; + + zinfo = file->format_file_info; + map = zinfo->map; + + if(container->sort == NCGRP) + attlist = ((NC_GRP_INFO_T*)container)->att; + else + attlist = ((NC_VAR_INFO_T*)container)->att; + + switch ((stat = load_jatts(map, container, (zinfo->controls.flags & FLAG_NCZARR_V1), &jattrs, &atypes))) { + case NC_NOERR: break; + case NC_EEMPTY: /* container has no attributes */ + stat = NC_NOERR; + break; + default: goto done; /* true error */ + } + + if(jattrs != NULL) { + /* Iterate over the attributes to create the in-memory attributes */ + /* Watch for special cases: _FillValue, _ARRAY_DIMENSIONS (xarray) */ + for(i=0;isort == NCGRP + && file->root_grp == (NC_GRP_INFO_T*)container) { + /* Setup provenance */ + if(NCJsort(value) != NCJ_STRING) + {stat = THROW(NC_ENCZARR); goto done;} /*malformed*/ + if((stat = NCZ_read_provenance(file,NCJstring(key),NCJstring(value)))) + goto done; + } + /* case 2: name = _ARRAY_DIMENSIONS, sort==NCVAR, flags & HIDDENATTRFLAG */ + if(strcmp(NCJstring(key),NC_XARRAY_DIMS)==0 + && container->sort == NCVAR + && (ra->flags & HIDDENATTRFLAG)) { + /* store for later */ + NCZ_VAR_INFO_T* zvar = (NCZ_VAR_INFO_T*)((NC_VAR_INFO_T*)container)->format_var_info; + int i; + assert(NCJsort(value) == NCJ_ARRAY); + if((zvar->xarray = nclistnew())==NULL) + {stat = NC_ENOMEM; goto done;} + for(i=0;ixarray,strdup(NCJstring(k))); + } + } + /* else ignore */ + continue; + } + /* Create the attribute */ + /* Collect the attribute's type and value */ + if((stat = computeattrinfo(NCJstring(key),atypes,value, + &typeid,&typelen,&len,&data))) + goto done; + if((stat = ncz_makeattr(container,attlist,NCJstring(key),typeid,len,data,&att))) + goto done; + nullfree(data); data = NULL; /* passed to the attribute */ + /* Is this _FillValue ? */ + if(strcmp(att->hdr.name,_FillValue)==0) fillvalueatt = att; + } + } + /* If we have not read a _FillValue, then go ahead and create it */ + if(fillvalueatt == NULL && container->sort == NCVAR) { + if((stat = ncz_ensure_fill_value((NC_VAR_INFO_T*)container))) goto done; + } + + /* Remember that we have read the atts for this var or group. */ + if(container->sort == NCVAR) + ((NC_VAR_INFO_T*)container)->atts_read = 1; + else + ((NC_GRP_INFO_T*)container)->atts_read = 1; + +done: + NCJreclaim(jattrs); + nclistfreeall(atypes); + nullfree(fullpath); + nullfree(data); + nullfree(key); + return THROW(stat); +} + +/** + * @internal Materialize dimensions into memory + * + * @param file Pointer to file info struct. + * @param grp Pointer to grp info struct. + * @param diminfo List of (name,length) pairs + * + * @return ::NC_NOERR No error. + * @author Dennis Heimbigner + */ +static int +define_dims(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* diminfo) +{ + int i,stat = NC_NOERR; + + /* Reify each dim in turn */ + for(i = 0; i < nclistlength(diminfo); i+=2) { + NC_DIM_INFO_T* dim = NULL; + size64_t len = 0; + const char* name = nclistget(diminfo,i); + const char* value = nclistget(diminfo,i+1); + + /* Create the NC_DIM_INFO_T object */ + sscanf(value,"%lld",&len); /* Get length */ + if(len <= 0) + {stat = NC_EDIMSIZE; goto done;} + if((stat = nc4_dim_list_add(grp, name, (size_t)len, -1, &dim))) + goto done; + if((dim->format_dim_info = calloc(1,sizeof(NCZ_DIM_INFO_T))) == NULL) + {stat = NC_ENOMEM; goto done;} + ((NCZ_DIM_INFO_T*)dim->format_dim_info)->common.file = file; + } + +done: + return THROW(stat); +} + +/** + * @internal Materialize vars into memory; + * Take xarray and purezarr into account. + * + * @param file Pointer to file info struct. + * @param grp Pointer to grp info struct. + * @param varnames List of names of variables in this group + * + * @return ::NC_NOERR No error. + * @author Dennis Heimbigner + */ +static int +define_vars(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* varnames) +{ + int stat = NC_NOERR; + int i,j; + char* varpath = NULL; + char* key = NULL; + NCZ_FILE_INFO_T* zinfo = NULL; + NCZ_VAR_INFO_T* zvar = NULL; + NCZMAP* map = NULL; + NCjson* jvar = NULL; + NCjson* jncvar = NULL; + NCjson* jdimrefs = NULL; + NCjson* jvalue = NULL; + NCjson* jfilter = NULL; + int purezarr = 0; + int xarray = 0; + int formatv1 = 0; + nc_type typeid; + size64_t* shapes = NULL; + int rank = 0; + NClist* dimnames = nclistnew(); + + zinfo = file->format_file_info; + map = zinfo->map; + + if(zinfo->controls.flags & FLAG_PUREZARR) purezarr = 1; + if(zinfo->controls.flags & FLAG_NCZARR_V1) formatv1 = 1; + if(zinfo->controls.flags & FLAG_XARRAYDIMS) {purezarr = 1; xarray = 1;} + + /* Load each var in turn */ + for(i = 0; i < nclistlength(varnames); i++) { + NC_VAR_INFO_T* var; + const char* varname = nclistget(varnames,i); + + /* Create the NC_VAR_INFO_T object */ + if((stat = nc4_var_list_add2(grp, varname, &var))) + goto done; + + /* And its annotation */ + if((zvar = calloc(1,sizeof(NCZ_VAR_INFO_T)))==NULL) + {stat = NC_ENOMEM; goto done;} + var->format_var_info = zvar; + zvar->common.file = file; + + /* Indicate we do not have quantizer yet */ + var->quantize_mode = -1; + + /* Set filter list */ + assert(var->filters == NULL); + var->filters = (void*)nclistnew(); + + /* Construct var path */ + if((stat = NCZ_varkey(var,&varpath))) + goto done; + + /* Construct the path to the zarray object */ + if((stat = nczm_concat(varpath,ZARRAY,&key))) + goto done; + /* Download the zarray object */ + if((stat=NCZ_readdict(map,key,&jvar))) + goto done; + nullfree(key); key = NULL; + assert(NCJsort(jvar) == NCJ_DICT); + + /* Extract the .zarray info from jvar */ + + /* Verify the format */ + { + int version; + if((stat = NCJdictget(jvar,"zarr_format",&jvalue))) goto done; + sscanf(NCJstring(jvalue),"%d",&version); + if(version != zinfo->zarr.zarr_version) + {stat = THROW(NC_ENCZARR); goto done;} + } + /* Set the type and endianness of the variable */ + { + nc_type vtype; + int endianness; + if((stat = NCJdictget(jvar,"dtype",&jvalue))) goto done; + /* Convert dtype to nc_type + endianness */ + if((stat = ncz_dtype2typeinfo(NCJstring(jvalue),&vtype,&endianness))) + goto done; + if(vtype > NC_NAT && vtype < NC_STRING) { + /* Locate the NC_TYPE_INFO_T object */ + if((stat = ncz_gettype(file,grp,vtype,&var->type_info))) + goto done; + } else {stat = NC_EBADTYPE; goto done;} + if(endianness == NC_ENDIAN_NATIVE) + endianness = zinfo->native_endianness; + if(endianness == NC_ENDIAN_LITTLE || endianness == NC_ENDIAN_BIG) { + var->endianness = endianness; + } else {stat = NC_EBADTYPE; goto done;} + var->type_info->endianness = var->endianness; /* Propagate */ + } + /* shape */ + { + if((stat = NCJdictget(jvar,"shape",&jvalue))) goto done; + if(NCJsort(jvalue) != NCJ_ARRAY) {stat = THROW(NC_ENCZARR); goto done;} + /* Verify the rank */ + if(zvar->scalar) rank = 0; else rank = NCJlength(jvalue); + /* Set the rank of the variable */ + if((stat = nc4_var_set_ndims(var, rank))) goto done; + /* extract the shapes */ + if((shapes = (size64_t*)malloc(sizeof(size64_t)*rank)) == NULL) + {stat = THROW(NC_ENOMEM); goto done;} + if((stat = decodeints(jvalue, shapes))) goto done; + } + /* Capture dimension_separator (must precede chunk cache creation) */ + { + NCRCglobalstate* ngs = ncrc_getglobalstate(); + assert(ngs != NULL); + zvar->dimension_separator = 0; + if((stat = NCJdictget(jvar,"dimension_separator",&jvalue))) goto done; + if(jvalue != NULL) { + /* Verify its value */ + if(NCJsort(jvalue) == NCJ_STRING && NCJstring(jvalue) != NULL && strlen(NCJstring(jvalue)) == 1) + zvar->dimension_separator = NCJstring(jvalue)[0]; + } + /* If value is invalid, then use global default */ + if(!islegaldimsep(zvar->dimension_separator)) + zvar->dimension_separator = ngs->zarr.dimension_separator; /* use global value */ + assert(islegaldimsep(zvar->dimension_separator)); /* we are hosed */ + } + /* fill_value; must precede calls to adjust cache */ + { + if((stat = NCJdictget(jvar,"fill_value",&jvalue))) goto done; + if(jvalue == NULL || NCJsort(jvalue) == NCJ_NULL) + var->no_fill = NC_TRUE; + else { + size_t fvlen; + typeid = var->type_info->hdr.id; + var->no_fill = NC_FALSE; + if((stat = computeattrdata(&typeid, jvalue, NULL, &fvlen, &var->fill_value))) + goto done; + assert(typeid == var->type_info->hdr.id); + /* Note that we do not create the _FillValue + attribute here to avoid having to read all + the attributes and thus foiling lazy read.*/ + } + } + /* chunks */ + { + int rank; + size64_t chunks[NC_MAX_VAR_DIMS]; + if((stat = NCJdictget(jvar,"chunks",&jvalue))) goto done; + if(jvalue != NULL && NCJsort(jvalue) != NCJ_ARRAY) + {stat = THROW(NC_ENCZARR); goto done;} + { + /* Verify the rank */ + rank = NCJlength(jvalue); + var->storage = NC_CHUNKED; + if(var->ndims != rank) + {stat = THROW(NC_ENCZARR); goto done;} + if((var->chunksizes = malloc(sizeof(size_t)*(rank+zvar->scalar))) == NULL) + {stat = NC_ENOMEM; goto done;} + if(zvar->scalar) { + chunks[0] = 1; + } else { + if((stat = decodeints(jvalue, chunks))) goto done; + } + /* validate the chunk sizes */ + zvar->chunkproduct = 1; + for(j=0;jscalar;j++) { + if(chunks[j] == 0 || chunks[j] > shapes[j]) + {stat = THROW(NC_ENCZARR); goto done;} + var->chunksizes[j] = (size_t)chunks[j]; + zvar->chunkproduct *= chunks[j]; + } + zvar->chunksize = zvar->chunkproduct * var->type_info->size; + /* Create the cache */ + if((stat = NCZ_create_chunk_cache(var,var->type_info->size*zvar->chunkproduct,zvar->dimension_separator,&zvar->cache))) + goto done; + if((stat = NCZ_adjust_var_cache(var))) goto done; + } + } + /* Capture row vs column major; currently, column major not used*/ + { + if((stat = NCJdictget(jvar,"order",&jvalue))) goto done; + if(strcmp(NCJstring(jvalue),"C")==1) + ((NCZ_VAR_INFO_T*)var->format_var_info)->order = 1; + else ((NCZ_VAR_INFO_T*)var->format_var_info)->order = 0; + } + /* filters key */ + /* From V2 Spec: A list of JSON objects providing codec configurations, + or null if no filters are to be applied. Each codec configuration + object MUST contain a "id" key identifying the codec to be used. */ + /* Do filters key before compressor key so final filter chain is in correct order */ + { + int k; + if(var->filters == NULL) var->filters = (void*)nclistnew(); + if((stat = NCZ_filter_initialize())) goto done; + if((stat = NCJdictget(jvar,"filters",&jvalue))) goto done; + if(jvalue != NULL && NCJsort(jvalue) != NCJ_NULL) { + if(NCJsort(jvalue) != NCJ_ARRAY) {stat = NC_EFILTER; goto done;} + for(k=0;;k++) { + jfilter = NULL; + jfilter = NCJith(jvalue,k); + if(jfilter == NULL) break; /* done */ + if(NCJsort(jfilter) != NCJ_DICT) {stat = NC_EFILTER; goto done;} + if((stat = NCZ_filter_build(file,var,jfilter))) goto done; + } + } + } + + /* compressor key */ + /* From V2 Spec: A JSON object identifying the primary compression codec and providing + configuration parameters, or ``null`` if no compressor is to be used. */ + { + if(var->filters == NULL) var->filters = (void*)nclistnew(); + if((stat = NCZ_filter_initialize())) goto done; + if((stat = NCJdictget(jvar,"compressor",&jfilter))) goto done; + if(jfilter != NULL && NCJsort(jfilter) != NCJ_NULL) { + if(NCJsort(jfilter) != NCJ_DICT) {stat = NC_EFILTER; goto done;} + if((stat = NCZ_filter_build(file,var,jfilter))) goto done; + } + } + if(!purezarr) { + if(formatv1) { + /* Construct the path to the zarray object */ + if((stat = nczm_concat(varpath,NCZARRAY,&key))) + goto done; + /* Download the nczarray object */ + if((stat=NCZ_readdict(map,key,&jncvar))) + goto done; + nullfree(key); key = NULL; + } else {/* format v2 */ + /* Extract the NCZ_V2_ARRAY dict */ + if((stat = NCJdictget(jvar,NCZ_V2_ARRAY,&jncvar))) goto done; + } + if(jncvar == NULL) {stat = NC_ENCZARR; goto done;} + assert((NCJsort(jncvar) == NCJ_DICT)); + /* Extract storage flag */ + if((stat = NCJdictget(jncvar,"storage",&jvalue))) + goto done; + if(jvalue != NULL) { + if(strcmp(NCJstring(jvalue),"chunked") == 0) { + var->storage = NC_CHUNKED; + } else if(strcmp(NCJstring(jvalue),"compact") == 0) { + var->storage = NC_COMPACT; + } else if(strcmp(NCJstring(jvalue),"scalar") == 0) { + var->storage = NC_CONTIGUOUS; + zvar->scalar = 1; + } else { /*storage = NC_CONTIGUOUS;*/ + var->storage = NC_CONTIGUOUS; + } + } + /* Extract dimnames list */ + switch ((stat = NCJdictget(jncvar,"dimrefs",&jdimrefs))) { + case NC_NOERR: /* Extract the dimref names */ + assert((NCJsort(jdimrefs) == NCJ_ARRAY)); + assert(NCJlength(jdimrefs) == rank); + for(j=0;jdim))) + goto done; + + /* Extract the dimids */ + for(j=0;jdimids[j] = var->dim[j]->hdr.id; + + /* At this point, we can finalize the filters */ + if((stat = NCZ_filter_setup(var))) goto done; + + /* Clean up from last cycle */ + nclistfreeall(dimnames); dimnames = nclistnew(); + nullfree(varpath); varpath = NULL; + nullfree(shapes); shapes = NULL; + if(formatv1) {NCJreclaim(jncvar); jncvar = NULL;} + NCJreclaim(jvar); jvar = NULL; + } + +done: + nullfree(shapes); + nullfree(varpath); + nullfree(key); + nclistfreeall(dimnames); + NCJreclaim(jvar); + if(formatv1) NCJreclaim(jncvar); + return THROW(stat); +} + +/** + * @internal Materialize subgroups into memory + * + * @param file Pointer to file info struct. + * @param grp Pointer to grp info struct. + * @param subgrpnames List of names of subgroups in this group + * + * @return ::NC_NOERR No error. + * @author Dennis Heimbigner + */ +static int +define_subgrps(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* subgrpnames) +{ + int i,stat = NC_NOERR; + + /* Load each subgroup name in turn */ + for(i = 0; i < nclistlength(subgrpnames); i++) { + NC_GRP_INFO_T* g = NULL; + const char* gname = nclistget(subgrpnames,i); + char norm_name[NC_MAX_NAME]; + /* Check and normalize the name. */ + if((stat = nc4_check_name(gname, norm_name))) + goto done; + if((stat = nc4_grp_list_add(file, grp, norm_name, &g))) + goto done; + if(!(g->format_grp_info = calloc(1, sizeof(NCZ_GRP_INFO_T)))) + {stat = NC_ENOMEM; goto done;} + ((NCZ_GRP_INFO_T*)g->format_grp_info)->common.file = file; + } + + /* Recurse to fill in subgroups */ + for(i=0;ichildren);i++) { + NC_GRP_INFO_T* g = (NC_GRP_INFO_T*)ncindexith(grp->children,i); + if((stat = define_grp(file,g))) + goto done; + } + +done: + return THROW(stat); +} + +int +ncz_read_superblock(NC_FILE_INFO_T* file, char** nczarrvp, char** zarrfp) +{ + int stat = NC_NOERR; + NCjson* jnczgroup = NULL; + NCjson* jzgroup = NULL; + NCjson* jsuper = NULL; + NCjson* jtmp = NULL; + char* nczarr_version = NULL; + char* zarr_format = NULL; + NCZ_FILE_INFO_T* zinfo = (NCZ_FILE_INFO_T*)file->format_file_info; + + /* See if the V1 META-Root is being used */ + switch(stat = NCZ_downloadjson(zinfo->map, NCZMETAROOT, &jnczgroup)) { + case NC_EEMPTY: /* not there */ + stat = NC_NOERR; + break; + case NC_NOERR: + if((stat = NCJdictget(jnczgroup,"nczarr_version",&jtmp))) goto done; + nczarr_version = strdup(NCJstring(jtmp)); + break; + default: goto done; + } + /* Also gett Zarr Root Group */ + switch(stat = NCZ_downloadjson(zinfo->map, ZMETAROOT, &jzgroup)) { + case NC_NOERR: + break; + case NC_EEMPTY: /* not there */ + stat = NC_NOERR; + assert(jzgroup == NULL); + break; + default: goto done; + } + if(jzgroup != NULL) { + /* See if this NCZarr V2 */ + if((stat = NCJdictget(jzgroup,NCZ_V2_SUPERBLOCK,&jsuper))) goto done; + if(jsuper != NULL) { + /* Extract the equivalent attribute */ + if(jsuper->sort != NCJ_DICT) + {stat = NC_ENCZARR; goto done;} + if((stat = NCJdictget(jsuper,"version",&jtmp))) goto done; + nczarr_version = nulldup(NCJstring(jtmp)); + } + /* In any case, extract the zarr format */ + if((stat = NCJdictget(jzgroup,"zarr_format",&jtmp))) goto done; + zarr_format = nulldup(NCJstring(jtmp)); + } + /* Set the controls */ + if(jnczgroup == NULL && jsuper == NULL) { + zinfo->controls.flags |= FLAG_PUREZARR; + } else if(jnczgroup != NULL) { + zinfo->controls.flags |= FLAG_NCZARR_V1; + /* Also means file is read only */ + file->no_write = 1; + } else if(jsuper != NULL) { + /* ! FLAG_NCZARR_V1 && ! FLAG_PUREZARR */ + } + if(nczarrvp) {*nczarrvp = nczarr_version; nczarr_version = NULL;} + if(zarrfp) {*zarrfp = zarr_format; zarr_format = NULL;} +done: + nullfree(zarr_format); + nullfree(nczarr_version); + NCJreclaim(jzgroup); + NCJreclaim(jnczgroup); + return THROW(stat); +} + +/**************************************************/ +/* Utilities */ + +static int +parse_group_content(NCjson* jcontent, NClist* dimdefs, NClist* varnames, NClist* subgrps) +{ + int i,stat = NC_NOERR; + NCjson* jvalue = NULL; + + if((stat=NCJdictget(jcontent,"dims",&jvalue))) goto done; + if(jvalue != NULL) { + if(NCJsort(jvalue) != NCJ_DICT) {stat = THROW(NC_ENCZARR); goto done;} + /* Extract the dimensions defined in this group */ + for(i=0;imap,zakey,&jvar))) + goto done; + assert((NCJsort(jvar) == NCJ_DICT)); + nullfree(varkey); varkey = NULL; + nullfree(zakey); zakey = NULL; + /* Extract the shape */ + if((stat=NCJdictget(jvar,"shape",&jvalue))) goto done; + if((stat = decodeints(jvalue, shapes))) goto done; + +done: + NCJreclaim(jvar); + NCJreclaim(jvalue); + nullfree(varkey); varkey = NULL; + nullfree(zakey); zakey = NULL; + return THROW(stat); +} +#endif + +static int +searchvars(NCZ_FILE_INFO_T* zfile, NC_GRP_INFO_T* grp, NClist* varnames) +{ + int i,stat = NC_NOERR; + char* grpkey = NULL; + char* varkey = NULL; + char* zarray = NULL; + NClist* matches = nclistnew(); + + /* Compute the key for the grp */ + if((stat = NCZ_grpkey(grp,&grpkey))) goto done; + /* Get the map and search group */ + if((stat = nczmap_search(zfile->map,grpkey,matches))) goto done; + for(i=0;imap,zarray)) == NC_NOERR) + nclistpush(varnames,strdup(name)); + stat = NC_NOERR; + nullfree(varkey); varkey = NULL; + nullfree(zarray); zarray = NULL; + } + +done: + nullfree(grpkey); + nullfree(varkey); + nullfree(zarray); + nclistfreeall(matches); + return stat; +} + +static int +searchsubgrps(NCZ_FILE_INFO_T* zfile, NC_GRP_INFO_T* grp, NClist* subgrpnames) +{ + int i,stat = NC_NOERR; + char* grpkey = NULL; + char* subkey = NULL; + char* zgroup = NULL; + NClist* matches = nclistnew(); + + /* Compute the key for the grp */ + if((stat = NCZ_grpkey(grp,&grpkey))) goto done; + /* Get the map and search group */ + if((stat = nczmap_search(zfile->map,grpkey,matches))) goto done; + for(i=0;imap,zgroup)) == NC_NOERR) + nclistpush(subgrpnames,strdup(name)); + stat = NC_NOERR; + nullfree(subkey); subkey = NULL; + nullfree(zgroup); zgroup = NULL; + } + +done: + nullfree(grpkey); + nullfree(subkey); + nullfree(zgroup); + nclistfreeall(matches); + return stat; +} + +/* Convert a list of integer strings to 64 bit integers */ +static int +decodeints(NCjson* jshape, size64_t* shapes) +{ + int i, stat = NC_NOERR; + + for(i=0;iroot_grp; + NC_DIM_INFO_T* thed = NULL; + if((stat = nc4_dim_list_add(root, name, (size_t)dimlen, -1, &thed))) + goto done; + assert(thed != NULL); + /* Create struct for NCZ-specific dim info. */ + if (!(thed->format_dim_info = calloc(1, sizeof(NCZ_DIM_INFO_T)))) + {stat = NC_ENOMEM; goto done;} + ((NCZ_DIM_INFO_T*)thed->format_dim_info)->common.file = file; + *dimp = thed; thed = NULL; +done: + return stat; +} + + +/* +Given a list of segments, find corresponding group. +*/ +static int +locategroup(NC_FILE_INFO_T* file, size_t nsegs, NClist* segments, NC_GRP_INFO_T** grpp) +{ + int i, j, found, stat = NC_NOERR; + NC_GRP_INFO_T* grp = NULL; + + grp = file->root_grp; + for(i=0;ichildren);j++) { + NC_GRP_INFO_T* subgrp = (NC_GRP_INFO_T*)ncindexith(grp->children,j); + if(strcmp(subgrp->hdr.name,norm_name)==0) { + grp = subgrp; + found = 1; + break; + } + } + if(!found) {stat = NC_ENOGRP; goto done;} + } + /* grp should be group of interest */ + if(grpp) *grpp = grp; + +done: + return THROW(stat); +} + +static int +parsedimrefs(NC_FILE_INFO_T* file, NClist* dimnames, size64_t* shape, NC_DIM_INFO_T** dims, int create) +{ + int i, stat = NC_NOERR; + NClist* segments = NULL; + + for(i=0;idim);j++) { + d = (NC_DIM_INFO_T*)ncindexith(g->dim,j); + if(strcmp(d->hdr.name,dimname)==0) { + dims[i] = d; + break; + } + } + if(dims[i] == NULL && create) { + /* If not found and create then create it */ + if((stat = createdim(file, dimname, shape[i], &dims[i]))) + goto done; + } + assert(dims[i] != NULL); + assert(dims[i]->len == shape[i]); + } +done: + nclistfreeall(segments); + return THROW(stat); +} + +/** + * @internal Get the metadata for a variable. + * + * @param var Pointer to var info struct. + * + * @return ::NC_NOERR No error. + * @return ::NC_EBADID Bad ncid. + * @return ::NC_ENOMEM Out of memory. + * @return ::NC_EHDFERR HDF5 returned error. + * @return ::NC_EVARMETA Error with var metadata. + * @author Ed Hartnett + */ +int +ncz_get_var_meta(NC_FILE_INFO_T* file, NC_VAR_INFO_T* var) +{ + int retval = NC_NOERR; + + assert(file && var && var->format_var_info); + LOG((3, "%s: var %s", __func__, var->hdr.name)); + + /* Have we already read the var metadata? */ + if (var->meta_read) + return NC_NOERR; + +#ifdef LOOK + /* Get the current chunk cache settings. */ + if ((access_pid = H5Dget_access_plist(hdf5_var->hdf_datasetid)) < 0) + BAIL(NC_EVARMETA); + + /* Learn about current chunk cache settings. */ + if ((H5Pget_chunk_cache(access_pid, &(var->chunk_cache_nelems), + &(var->chunk_cache_size), &rdcc_w0)) < 0) + BAIL(NC_EHDFERR); + var->chunk_cache_preemption = rdcc_w0; + + /* Get the dataset creation properties. */ + if ((propid = H5Dget_create_plist(hdf5_var->hdf_datasetid)) < 0) + BAIL(NC_EHDFERR); + + /* Get var chunking info. */ + if ((retval = get_chunking_info(propid, var))) + BAIL(retval); + + /* Get filter info for a var. */ + if ((retval = get_filter_info(propid, var))) + BAIL(retval); + + /* Get fill value, if defined. */ + if ((retval = get_fill_info(propid, var))) + BAIL(retval); + + /* Is this a deflated variable with a chunksize greater than the + * current cache size? */ + if ((retval = nc4_adjust_var_cache(var))) + BAIL(retval); + + /* Is there an attribute which means quantization was used? */ + if ((retval = get_quantize_info(var))) + BAIL(retval); + + if (var->coords_read && !var->dimscale) + if ((retval = get_attached_info(var, hdf5_var, var->ndims, hdf5_var->hdf_datasetid))) + return retval; +#endif + + /* Remember that we have read the metadata for this var. */ + var->meta_read = NC_TRUE; + + return retval; +} + +#if 0 +int +ncz_create_superblock(NCZ_FILE_INFO_T* zinfo) +{ + int stat = NC_NOERR; + NCjson* json = NULL; + NCZMAP* map = NULL; + char version[1024]; + + ZTRACE(4,"zinfo=%s",zinfo->common.file->controller->path); + + /* If V2, then do not create a superblock per-se */ + if(!(zinfo->controls.flags & FLAG_NCZARR_V1)) goto done; + + map = zinfo->map; + + /* create superblock json */ + if((stat = NCJnew(NCJ_DICT,&json))) + goto done; + + /* fill */ + snprintf(version,sizeof(version),"%d",zinfo->zarr.zarr_version); + if((stat = NCJaddstring(json,NCJ_STRING,"zarr_format"))) goto done; + if((stat = NCJaddstring(json,NCJ_INT,version))) goto done; + if((stat = NCJaddstring(json,NCJ_STRING,NCZ_V2_VERSION))) goto done; + { + char ver[1024]; + snprintf(ver,sizeof(ver),"%lu.%lu.%lu", + zinfo->zarr.nczarr_version.major, + zinfo->zarr.nczarr_version.minor, + zinfo->zarr.nczarr_version.release); + if((stat = NCJaddstring(json,NCJ_STRING,ver))) goto done; + } + /* Write back to map */ + if((stat=NCZ_uploadjson(map,NCZMETAROOT,json))) + goto done; +done: + NCJreclaim(json); + return ZUNTRACE(stat); +} +#endif + +/* Compute the set of dim refs for this variable, taking purezarr and xarray into account */ +static int +computedimrefs(NC_FILE_INFO_T* file, NC_VAR_INFO_T* var, int purezarr, int xarray, int ndims, NClist* dimnames, size64_t* shapes, NC_DIM_INFO_T** dims) +{ + int stat = NC_NOERR; + int i; + int createdims = 0; /* 1 => we need to create the dims in root if they do not already exist */ + NCZ_FILE_INFO_T* zfile = (NCZ_FILE_INFO_T*)file->format_file_info; + NCZ_VAR_INFO_T* zvar = (NCZ_VAR_INFO_T*)(var->format_var_info); + NCjson* jatts = NULL; + + assert(zfile && zvar); + + /* xarray => purezarr */ + assert(!xarray || purezarr); + + if(xarray) {/* Read in the attributes to get xarray dimdef attribute; Note that it might not exist */ + char zdimname[4096]; + if(zvar->xarray == NULL) { + assert(nclistlength(dimnames) == 0); + if((stat = ncz_read_atts(file,(NC_OBJ*)var))) goto done; + } + if(zvar->xarray != NULL) { + /* convert xarray to the dimnames */ + for(i=0;ixarray);i++) { + snprintf(zdimname,sizeof(zdimname),"/%s",(const char*)nclistget(zvar->xarray,i)); + nclistpush(dimnames,strdup(zdimname)); + } + } + createdims = 1; /* may need to create them */ + } + + /* If pure zarr and we have no dimref names, then fake it */ + if(purezarr && nclistlength(dimnames) == 0) { + createdims = 1; + for(i=0;ino_fill*/ int atomictype = var->type_info->hdr.id; + if((stat = ncz_ensure_fill_value(var))) goto done; assert(var->fill_value != NULL); /* Convert var->fill_value to a string */ if((stat = NCZ_stringconvert(atomictype,1,var->fill_value,&jfill))) goto done; @@ -499,7 +473,7 @@ ncz_sync_var(NC_FILE_INFO_T* file, NC_VAR_INFO_T* var, int isclose) int stat = NC_NOERR; NCZ_VAR_INFO_T* zvar = var->format_var_info; - if(!isclose) { + if(isclose) { if((stat = ncz_sync_var_meta(file,var,isclose))) goto done; } @@ -596,12 +570,14 @@ int ncz_sync_atts(NC_FILE_INFO_T* file, NC_OBJ* container, NCindex* attlist, int isclose) { int i,stat = NC_NOERR; + NC_VAR_INFO_T* var = NULL; NCZ_FILE_INFO_T* zinfo = NULL; NCjson* jatts = NULL; NCjson* jtypes = NULL; NCjson* jtype = NULL; NCjson* jdimrefs = NULL; NCjson* jdict = NULL; + NCjson* jint = NULL; NCZMAP* map = NULL; char* fullpath = NULL; char* key = NULL; @@ -618,7 +594,7 @@ ncz_sync_atts(NC_FILE_INFO_T* file, NC_OBJ* container, NCindex* attlist, int isc if(zinfo->controls.flags & FLAG_XARRAYDIMS) isxarray = 1; if(container->sort == NCVAR) { - NC_VAR_INFO_T* var = (NC_VAR_INFO_T*)container; + var = (NC_VAR_INFO_T*)container; if(var->container && var->container->parent == NULL) isrootgroup = 1; } @@ -660,7 +636,7 @@ ncz_sync_atts(NC_FILE_INFO_T* file, NC_OBJ* container, NCindex* attlist, int isc if((stat = ncz_jsonize_atts(attlist,&jatts))) goto done; - if(container->sort == NCVAR) { + if(var) { if(isrootgroup && isxarray) { NC_VAR_INFO_T* var = (NC_VAR_INFO_T*)container; /* Insert the XARRAY _ARRAY_ATTRIBUTE attribute */ @@ -689,6 +665,22 @@ ncz_sync_atts(NC_FILE_INFO_T* file, NC_OBJ* container, NCindex* attlist, int isc jdict = NULL; } + /* Add Quantize Attribute */ + if(var && var->quantize_mode > 0) { + char mode[64]; + snprintf(mode,sizeof(mode),"%d",var->nsd); + if((stat = NCJnewstring(NCJ_INT,mode,&jint))) + goto done; + /* Insert the quantize attribute */ + switch (var->quantize_mode) { + case NC_QUANTIZE_BITGROOM: + if((stat = NCJinsert(jatts,NC_QUANTIZE_ATT_NAME,jint))) goto done; + jint = NULL; + break; + default: break; + } + } + /* write .zattrs path */ if((stat = nczm_concat(fullpath,ZATTRS,&key))) goto done; @@ -707,12 +699,10 @@ ncz_sync_atts(NC_FILE_INFO_T* file, NC_OBJ* container, NCindex* attlist, int isc NCJreclaim(jtype); NCJreclaim(jdimrefs); NCJreclaim(jdict); + NCJreclaim(jint); return THROW(stat); } - -/**************************************************/ - /** @internal Convert a list of attributes to corresponding json. Note that this does not push to the file. @@ -753,1502 +743,3 @@ ncz_jsonize_atts(NCindex* attlist, NCjson** jattrsp) NCJreclaim(jdata); return THROW(stat); } - -/** -@internal Extract attributes from a group or var and return -the corresponding NCjson dict. -@param map - [in] the map object for storage -@param container - [in] the containing object -@param jattrsp - [out] the json for .zattrs -@param jtypesp - [out] the json for .ztypes -@return NC_NOERR -@author Dennis Heimbigner -*/ -static int -load_jatts(NCZMAP* map, NC_OBJ* container, int nczarrv1, NCjson** jattrsp, NClist** atypesp) -{ - int i,stat = NC_NOERR; - char* fullpath = NULL; - char* key = NULL; - NCjson* jnczarr = NULL; - NCjson* jattrs = NULL; - NCjson* jncattr = NULL; - NClist* atypes = NULL; /* envv list */ - - /* alway return (possibly empty) list of types */ - atypes = nclistnew(); - - if(container->sort == NCGRP) { - NC_GRP_INFO_T* grp = (NC_GRP_INFO_T*)container; - /* Get grp's fullpath name */ - if((stat = NCZ_grpkey(grp,&fullpath))) - goto done; - } else { - NC_VAR_INFO_T* var = (NC_VAR_INFO_T*)container; - /* Get var's fullpath name */ - if((stat = NCZ_varkey(var,&fullpath))) - goto done; - } - - /* Construct the path to the .zattrs object */ - if((stat = nczm_concat(fullpath,ZATTRS,&key))) - goto done; - - /* Download the .zattrs object: may not exist */ - switch ((stat=NCZ_downloadjson(map,key,&jattrs))) { - case NC_NOERR: break; - case NC_EEMPTY: stat = NC_NOERR; break; /* did not exist */ - default: goto done; /* failure */ - } - nullfree(key); key = NULL; - - if(jattrs != NULL) { - if(nczarrv1) { - /* Construct the path to the NCZATTRS object */ - if((stat = nczm_concat(fullpath,NCZATTRS,&key))) goto done; - /* Download the NCZATTRS object: may not exist if pure zarr or using deprecated name */ - stat=NCZ_downloadjson(map,key,&jncattr); - if(stat == NC_EEMPTY) { - /* try deprecated name */ - nullfree(key); key = NULL; - if((stat = nczm_concat(fullpath,NCZATTRDEP,&key))) goto done; - stat=NCZ_downloadjson(map,key,&jncattr); - } - } else {/* Get _NCZARR_ATTRS from .zattrs */ - stat = NCJdictget(jattrs,NCZ_V2_ATTR,&jncattr); - } - nullfree(key); key = NULL; - switch (stat) { - case NC_NOERR: break; - case NC_EEMPTY: stat = NC_NOERR; jncattr = NULL; break; - default: goto done; /* failure */ - } - if(jncattr != NULL) { - NCjson* jtypes = NULL; - /* jncattr attribute should be a dict */ - if(NCJsort(jncattr) != NCJ_DICT) {stat = THROW(NC_ENCZARR); goto done;} - /* Extract "types; may not exist if only hidden attributes are defined */ - if((stat = NCJdictget(jncattr,"types",&jtypes))) goto done; - if(jtypes != NULL) { - if(NCJsort(jtypes) != NCJ_DICT) {stat = THROW(NC_ENCZARR); goto done;} - /* Convert to an envv list */ - for(i=0;i= NC_STRING) - {stat = NC_EINTERNAL; goto done;} - if((stat = computeattrdata(&typeid, values, &typelen, &len, &data))) goto done; - - if(typeidp) *typeidp = typeid; - if(lenp) *lenp = len; - if(typelenp) *typelenp = typelen; - if(datap) {*datap = data; data = NULL;} - -done: - nullfree(data); - return THROW(stat); -} - -/* -Extract data for an attribute -*/ -static int -computeattrdata(nc_type* typeidp, NCjson* values, size_t* typelenp, size_t* lenp, void** datap) -{ - int stat = NC_NOERR; - size_t count; - void* data = NULL; - size_t typelen; - nc_type typeid = NC_NAT; - int reclaimvalues = 0; - - /* Get assumed type */ - if(typeidp) typeid = *typeidp; - if(typeid == NC_NAT) if((stat = inferattrtype(values,&typeid))) goto done; - if(typeid == NC_NAT) {stat = NC_EBADTYPE; goto done;} - - /* Collect the length of the attribute; might be a singleton */ - switch (NCJsort(values)) { - case NCJ_DICT: stat = NC_ENCZARR; goto done; - case NCJ_ARRAY: - count = NCJlength(values); - break; - case NCJ_STRING: /* requires special handling as an array of characters; also look out for empty string */ - if(typeid == NC_CHAR) { - count = strlen(NCJstring(values)); - if(count == 0) count = 1; /* Actually a single nul char, probably default fill value ugh!*/ - } else - count = 1; - break; - default: - count = 1; /* singleton */ - break; - } - - if(count > 0) { - /* Allocate data space */ - if((stat = NC4_inq_atomic_type(typeid, NULL, &typelen))) - goto done; - if(typeid == NC_CHAR) - data = malloc(typelen*(count+1)); - else - data = malloc(typelen*count); - if(data == NULL) - {stat = NC_ENOMEM; goto done;} - /* convert to target type */ - if((stat = zconvert(typeid, typelen, data, values))) - goto done; - } - if(lenp) *lenp = count; - if(typelenp) *typelenp = typelen; - if(datap) {*datap = data; data = NULL;} - if(typeidp) *typeidp = typeid; /* return possibly inferred type */ - -done: - if(reclaimvalues) NCJreclaim(values); /* we created it */ - nullfree(data); - return THROW(stat); -} - -static int -inferattrtype(NCjson* value, nc_type* typeidp) -{ - int stat = NC_NOERR; - nc_type typeid; - NCjson* j = NULL; - unsigned long long u64; - long long i64; - int negative = 0; - - if(NCJsort(value) == NCJ_ARRAY && NCJlength(value) == 0) - {typeid = NC_NAT; goto done;} - - if(NCJsort(value) == NCJ_NULL) - {typeid = NC_NAT; goto done;} - - if(value->sort == NCJ_ARRAY) { - j=NCJith(value,0); - return inferattrtype(j,typeidp); - } - - switch (NCJsort(value)) { - case NCJ_NULL: - typeid = NC_CHAR; - return NC_NOERR; - case NCJ_DICT: /* fall thru */ - case NCJ_UNDEF: - return NC_EINVAL; - default: /* atomic */ - break; - } - if(NCJstring(value) != NULL) - negative = (NCJstring(value)[0] == '-'); - switch (value->sort) { - case NCJ_INT: - if(negative) { - sscanf(NCJstring(value),"%lld",&i64); - u64 = (unsigned long long)i64; - } else - sscanf(NCJstring(value),"%llu",&u64); - typeid = mininttype(u64,negative); - break; - case NCJ_DOUBLE: - typeid = NC_DOUBLE; - break; - case NCJ_BOOLEAN: - typeid = NC_UBYTE; - break; - case NCJ_STRING: /* requires special handling as an array of characters */ - typeid = NC_CHAR; - break; - default: - stat = NC_ENCZARR; - } -done: - if(typeidp) *typeidp = typeid; - return stat; -} - -static int -mininttype(unsigned long long u64, int negative) -{ - long long i64 = (long long)u64; /* keep bit pattern */ - if(!negative && u64 >= NC_MAX_INT64) return NC_UINT64; - if(i64 < 0) { - if(i64 >= NC_MIN_BYTE) return NC_BYTE; - if(i64 >= NC_MIN_SHORT) return NC_SHORT; - if(i64 >= NC_MIN_INT) return NC_INT; - return NC_INT64; - } - if(i64 <= NC_MAX_BYTE) return NC_BYTE; - if(i64 <= NC_MAX_UBYTE) return NC_UBYTE; - if(i64 <= NC_MAX_SHORT) return NC_SHORT; - if(i64 <= NC_MAX_USHORT) return NC_USHORT; - if(i64 <= NC_MAX_INT) return NC_INT; - if(i64 <= NC_MAX_UINT) return NC_UINT; - return NC_INT64; -} - - -/** - * @internal Read file data from map to memory. - * - * @param file Pointer to file info struct. - * - * @return ::NC_NOERR No error. - * @author Dennis Heimbigner - */ -int -ncz_read_file(NC_FILE_INFO_T* file) -{ - int stat = NC_NOERR; - NCjson* json = NULL; - - LOG((3, "%s: file: %s", __func__, file->controller->path)); - - /* _nczarr should already have been read in ncz_open_dataset */ - - /* Now load the groups starting with root */ - if((stat = define_grp(file,file->root_grp))) - goto done; - -done: - NCJreclaim(json); - return THROW(stat); -} - -/** - * @internal Read group data from map to memory - * - * @param file Pointer to file struct - * @param grp Pointer to grp struct - * - * @return ::NC_NOERR No error. - * @author Dennis Heimbigner - */ -static int -define_grp(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp) -{ - int stat = NC_NOERR; - NCZ_FILE_INFO_T* zinfo = NULL; - NCZMAP* map = NULL; - char* fullpath = NULL; - char* key = NULL; - NCjson* json = NULL; - NCjson* jgroup = NULL; - NCjson* jdict = NULL; - NClist* dimdefs = nclistnew(); - NClist* varnames = nclistnew(); - NClist* subgrps = nclistnew(); - int purezarr = 0; - int v1 = 0; - - LOG((3, "%s: dims: %s", __func__, key)); - - zinfo = file->format_file_info; - map = zinfo->map; - - /* Construct grp path */ - if((stat = NCZ_grpkey(grp,&fullpath))) - goto done; - - if(zinfo->controls.flags & FLAG_PUREZARR) { - if((stat = parse_group_content_pure(zinfo,grp,varnames,subgrps))) - goto done; - purezarr = 1; - } else { /*!purezarr*/ - if(zinfo->controls.flags & FLAG_NCZARR_V1) { - /* build NCZGROUP path */ - if((stat = nczm_concat(fullpath,NCZGROUP,&key))) - goto done; - /* Read */ - jdict = NULL; - stat=NCZ_downloadjson(map,key,&jdict); - v1 = 1; - } else { - /* build ZGROUP path */ - if((stat = nczm_concat(fullpath,ZGROUP,&key))) - goto done; - /* Read */ - switch (stat=NCZ_downloadjson(map,key,&jgroup)) { - case NC_NOERR: /* we read it */ - /* Extract the NCZ_V2_GROUP dict */ - if((stat = NCJdictget(jgroup,NCZ_V2_GROUP,&jdict))) goto done; - break; - case NC_EEMPTY: /* does not exist, use search */ - if((stat = parse_group_content_pure(zinfo,grp,varnames,subgrps))) - goto done; - purezarr = 1; - break; - default: goto done; - } - } - nullfree(key); key = NULL; - if(jdict) { - /* Pull out lists about group content */ - if((stat = parse_group_content(jdict,dimdefs,varnames,subgrps))) - goto done; - } - } - - if(!purezarr) { - /* Define dimensions */ - if((stat = define_dims(file,grp,dimdefs))) goto done; - } - - /* Define vars taking xarray into account */ - if((stat = define_vars(file,grp,varnames))) goto done; - - /* Define sub-groups */ - if((stat = define_subgrps(file,grp,subgrps))) goto done; - -done: - if(v1) NCJreclaim(jdict); - NCJreclaim(json); - NCJreclaim(jgroup); - nclistfreeall(dimdefs); - nclistfreeall(varnames); - nclistfreeall(subgrps); - nullfree(fullpath); - nullfree(key); - return THROW(stat); -} - - -/** -@internal Read attributes from a group or var and create a list -of annotated NC_ATT_INFO_T* objects. This will process -_NCProperties attribute specially. -@param zfile - [in] the containing file (annotation) -@param container - [in] the containing object -@return NC_NOERR -@author Dennis Heimbigner -*/ -int -ncz_read_atts(NC_FILE_INFO_T* file, NC_OBJ* container) -{ - int stat = NC_NOERR; - int i; - char* fullpath = NULL; - char* key = NULL; - NCZ_FILE_INFO_T* zinfo = NULL; - NCZMAP* map = NULL; - NC_ATT_INFO_T* att = NULL; - NCindex* attlist = NULL; - NCjson* jattrs = NULL; - NClist* atypes = NULL; - nc_type typeid; - size_t len, typelen; - void* data = NULL; - NC_ATT_INFO_T* fillvalueatt = NULL; - - zinfo = file->format_file_info; - map = zinfo->map; - - if(container->sort == NCGRP) - attlist = ((NC_GRP_INFO_T*)container)->att; - else - attlist = ((NC_VAR_INFO_T*)container)->att; - - switch ((stat = load_jatts(map, container, (zinfo->controls.flags & FLAG_NCZARR_V1), &jattrs, &atypes))) { - case NC_NOERR: break; - case NC_EEMPTY: /* container has no attributes */ - stat = NC_NOERR; - break; - default: goto done; /* true error */ - } - - if(jattrs != NULL) { - /* Iterate over the attributes to create the in-memory attributes */ - /* Watch for special cases: _FillValue and _ARRAY_DIMENSIONS (xarray) */ - for(i=0;isort == NCGRP - && file->root_grp == (NC_GRP_INFO_T*)container) { - /* Setup provenance */ - if(NCJsort(value) != NCJ_STRING) - {stat = THROW(NC_ENCZARR); goto done;} /*malformed*/ - if((stat = NCZ_read_provenance(file,NCJstring(key),NCJstring(value)))) - goto done; - } - /* case 2: name = _ARRAY_DIMENSIONS, sort==NCVAR, flags & HIDDENATTRFLAG */ - if(strcmp(NCJstring(key),NC_XARRAY_DIMS)==0 - && container->sort == NCVAR - && (ra->flags & HIDDENATTRFLAG)) { - /* store for later */ - NCZ_VAR_INFO_T* zvar = (NCZ_VAR_INFO_T*)((NC_VAR_INFO_T*)container)->format_var_info; - int i; - assert(NCJsort(value) == NCJ_ARRAY); - if((zvar->xarray = nclistnew())==NULL) - {stat = NC_ENOMEM; goto done;} - for(i=0;ixarray,strdup(NCJstring(k))); - } - } - /* else ignore */ - continue; - } - /* Create the attribute */ - /* Collect the attribute's type and value */ - if((stat = computeattrinfo(NCJstring(key),atypes,value, - &typeid,&typelen,&len,&data))) - goto done; - if((stat = ncz_makeattr(container,attlist,NCJstring(key),typeid,len,data,&att))) - goto done; - nullfree(data); data = NULL; /* passed to the attribute */ - /* Is this _FillValue ? */ - if(strcmp(att->hdr.name,_FillValue)==0) fillvalueatt = att; - } - } - /* If we have not read a _FillValue, then go ahead and create it */ - if(fillvalueatt == NULL && container->sort == NCVAR) { - if((stat = ncz_create_fillvalue((NC_VAR_INFO_T*)container))) - goto done; - } - - /* Remember that we have read the atts for this var or group. */ - if(container->sort == NCVAR) - ((NC_VAR_INFO_T*)container)->atts_read = 1; - else - ((NC_GRP_INFO_T*)container)->atts_read = 1; - -done: - NCJreclaim(jattrs); - nclistfreeall(atypes); - nullfree(fullpath); - nullfree(data); - nullfree(key); - return THROW(stat); -} - -/** - * @internal Materialize dimensions into memory - * - * @param file Pointer to file info struct. - * @param grp Pointer to grp info struct. - * @param diminfo List of (name,length) pairs - * - * @return ::NC_NOERR No error. - * @author Dennis Heimbigner - */ -static int -define_dims(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* diminfo) -{ - int i,stat = NC_NOERR; - - /* Reify each dim in turn */ - for(i = 0; i < nclistlength(diminfo); i+=2) { - NC_DIM_INFO_T* dim = NULL; - size64_t len = 0; - const char* name = nclistget(diminfo,i); - const char* value = nclistget(diminfo,i+1); - - /* Create the NC_DIM_INFO_T object */ - sscanf(value,"%lld",&len); /* Get length */ - if(len <= 0) - {stat = NC_EDIMSIZE; goto done;} - if((stat = nc4_dim_list_add(grp, name, (size_t)len, -1, &dim))) - goto done; - if((dim->format_dim_info = calloc(1,sizeof(NCZ_DIM_INFO_T))) == NULL) - {stat = NC_ENOMEM; goto done;} - ((NCZ_DIM_INFO_T*)dim->format_dim_info)->common.file = file; - } - -done: - return THROW(stat); -} - -/** - * @internal Materialize vars into memory; - * Take xarray and purezarr into account. - * - * @param file Pointer to file info struct. - * @param grp Pointer to grp info struct. - * @param varnames List of names of variables in this group - * - * @return ::NC_NOERR No error. - * @author Dennis Heimbigner - */ -static int -define_vars(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* varnames) -{ - int stat = NC_NOERR; - int i,j; - char* varpath = NULL; - char* key = NULL; - NCZ_FILE_INFO_T* zinfo = NULL; - NCZ_VAR_INFO_T* zvar = NULL; - NCZMAP* map = NULL; - NCjson* jvar = NULL; - NCjson* jncvar = NULL; - NCjson* jdimrefs = NULL; - NCjson* jvalue = NULL; - NCjson* jfilter = NULL; - int purezarr = 0; - int xarray = 0; - int formatv1 = 0; - nc_type typeid; - size64_t* shapes = NULL; - int rank = 0; - NClist* dimnames = nclistnew(); - - zinfo = file->format_file_info; - map = zinfo->map; - - if(zinfo->controls.flags & FLAG_PUREZARR) purezarr = 1; - if(zinfo->controls.flags & FLAG_NCZARR_V1) formatv1 = 1; - if(zinfo->controls.flags & FLAG_XARRAYDIMS) {purezarr = 1; xarray = 1;} - - /* Load each var in turn */ - for(i = 0; i < nclistlength(varnames); i++) { - NC_VAR_INFO_T* var; - const char* varname = nclistget(varnames,i); - - /* Create the NC_VAR_INFO_T object */ - if((stat = nc4_var_list_add2(grp, varname, &var))) - goto done; - - /* And its annotation */ - if((zvar = calloc(1,sizeof(NCZ_VAR_INFO_T)))==NULL) - {stat = NC_ENOMEM; goto done;} - var->format_var_info = zvar; - zvar->common.file = file; - - /* Set filter list */ - assert(var->filters == NULL); - var->filters = (void*)nclistnew(); - - /* Construct var path */ - if((stat = NCZ_varkey(var,&varpath))) - goto done; - - /* Construct the path to the zarray object */ - if((stat = nczm_concat(varpath,ZARRAY,&key))) - goto done; - /* Download the zarray object */ - if((stat=NCZ_readdict(map,key,&jvar))) - goto done; - nullfree(key); key = NULL; - assert(NCJsort(jvar) == NCJ_DICT); - - /* Extract the .zarray info from jvar */ - - /* Verify the format */ - { - int version; - if((stat = NCJdictget(jvar,"zarr_format",&jvalue))) goto done; - sscanf(NCJstring(jvalue),"%d",&version); - if(version != zinfo->zarr.zarr_version) - {stat = THROW(NC_ENCZARR); goto done;} - } - /* Set the type and endianness of the variable */ - { - nc_type vtype; - int endianness; - if((stat = NCJdictget(jvar,"dtype",&jvalue))) goto done; - /* Convert dtype to nc_type + endianness */ - if((stat = ncz_dtype2typeinfo(NCJstring(jvalue),&vtype,&endianness))) - goto done; - if(vtype > NC_NAT && vtype < NC_STRING) { - /* Locate the NC_TYPE_INFO_T object */ - if((stat = ncz_gettype(file,grp,vtype,&var->type_info))) - goto done; - } else {stat = NC_EBADTYPE; goto done;} - if(endianness == NC_ENDIAN_NATIVE) - endianness = zinfo->native_endianness; - if(endianness == NC_ENDIAN_LITTLE || endianness == NC_ENDIAN_BIG) { - var->endianness = endianness; - } else {stat = NC_EBADTYPE; goto done;} - var->type_info->endianness = var->endianness; /* Propagate */ - } - /* shape */ - { - if((stat = NCJdictget(jvar,"shape",&jvalue))) goto done; - if(NCJsort(jvalue) != NCJ_ARRAY) {stat = THROW(NC_ENCZARR); goto done;} - /* Verify the rank */ - if(zvar->scalar) rank = 0; else rank = NCJlength(jvalue); - /* Set the rank of the variable */ - if((stat = nc4_var_set_ndims(var, rank))) goto done; - /* extract the shapes */ - if((shapes = (size64_t*)malloc(sizeof(size64_t)*rank)) == NULL) - {stat = THROW(NC_ENOMEM); goto done;} - if((stat = decodeints(jvalue, shapes))) goto done; - } - /* Capture dimension_separator (must precede chunk cache creation) */ - { - NCRCglobalstate* ngs = ncrc_getglobalstate(); - assert(ngs != NULL); - zvar->dimension_separator = 0; - if((stat = NCJdictget(jvar,"dimension_separator",&jvalue))) goto done; - if(jvalue != NULL) { - /* Verify its value */ - if(NCJsort(jvalue) == NCJ_STRING && NCJstring(jvalue) != NULL && strlen(NCJstring(jvalue)) == 1) - zvar->dimension_separator = NCJstring(jvalue)[0]; - } - /* If value is invalid, then use global default */ - if(!islegaldimsep(zvar->dimension_separator)) - zvar->dimension_separator = ngs->zarr.dimension_separator; /* use global value */ - assert(islegaldimsep(zvar->dimension_separator)); /* we are hosed */ - } - /* fill_value; must precede calls to adjust cache */ - { - if((stat = NCJdictget(jvar,"fill_value",&jvalue))) goto done; - if(jvalue == NULL || NCJsort(jvalue) == NCJ_NULL) - var->no_fill = 1; - else { - size_t fvlen; - typeid = var->type_info->hdr.id; - var->no_fill = 0; - if((stat = computeattrdata(&typeid, jvalue, NULL, &fvlen, &var->fill_value))) - goto done; - assert(typeid == var->type_info->hdr.id); - /* Note that we do not create the _FillValue - attribute here to avoid having to read all - the attributes and thus foiling lazy read.*/ - } - } - /* chunks */ - { - int rank; - size64_t chunks[NC_MAX_VAR_DIMS]; - if((stat = NCJdictget(jvar,"chunks",&jvalue))) goto done; - if(jvalue != NULL && NCJsort(jvalue) != NCJ_ARRAY) - {stat = THROW(NC_ENCZARR); goto done;} - /* Verify the rank */ - rank = NCJlength(jvalue); - if(rank > 0) { - var->storage = NC_CHUNKED; - if(var->ndims+zvar->scalar != rank) - {stat = THROW(NC_ENCZARR); goto done;} - if((var->chunksizes = malloc(sizeof(size_t)*rank)) == NULL) - {stat = NC_ENOMEM; goto done;} - if((stat = decodeints(jvalue, chunks))) goto done; - /* validate the chunk sizes */ - zvar->chunkproduct = 1; - for(j=0;j shapes[j]) - {stat = THROW(NC_ENCZARR); goto done;} - var->chunksizes[j] = (size_t)chunks[j]; - zvar->chunkproduct *= chunks[j]; - } - zvar->chunksize = zvar->chunkproduct * var->type_info->size; - /* Create the cache */ - if((stat = NCZ_create_chunk_cache(var,var->type_info->size*zvar->chunkproduct,zvar->dimension_separator,&zvar->cache))) - goto done; - if((stat = NCZ_adjust_var_cache(var))) goto done; - } - } - /* Capture row vs column major; currently, column major not used*/ - { - if((stat = NCJdictget(jvar,"order",&jvalue))) goto done; - if(strcmp(NCJstring(jvalue),"C")==1) - ((NCZ_VAR_INFO_T*)var->format_var_info)->order = 1; - else ((NCZ_VAR_INFO_T*)var->format_var_info)->order = 0; - } - /* filters key */ - /* From V2 Spec: A list of JSON objects providing codec configurations, - or null if no filters are to be applied. Each codec configuration - object MUST contain a "id" key identifying the codec to be used. */ - /* Do filters key before compressor key so final filter chain is in correct order */ - { - int k; - if(var->filters == NULL) var->filters = (void*)nclistnew(); - if((stat = NCZ_filter_initialize())) goto done; - if((stat = NCJdictget(jvar,"filters",&jvalue))) goto done; - if(jvalue != NULL && NCJsort(jvalue) != NCJ_NULL) { - if(NCJsort(jvalue) != NCJ_ARRAY) {stat = NC_EFILTER; goto done;} - for(k=0;;k++) { - jfilter = NULL; - jfilter = NCJith(jvalue,k); - if(jfilter == NULL) break; /* done */ - if(NCJsort(jfilter) != NCJ_DICT) {stat = NC_EFILTER; goto done;} - if((stat = NCZ_filter_build(file,var,jfilter))) goto done; - } - } - } - - /* compressor key */ - /* From V2 Spec: A JSON object identifying the primary compression codec and providing - configuration parameters, or ``null`` if no compressor is to be used. */ - { - if(var->filters == NULL) var->filters = (void*)nclistnew(); - if((stat = NCZ_filter_initialize())) goto done; - if((stat = NCJdictget(jvar,"compressor",&jfilter))) goto done; - if(jfilter != NULL && NCJsort(jfilter) != NCJ_NULL) { - if(NCJsort(jfilter) != NCJ_DICT) {stat = NC_EFILTER; goto done;} - if((stat = NCZ_filter_build(file,var,jfilter))) goto done; - } - } - - if(!purezarr) { - if(formatv1) { - /* Construct the path to the zarray object */ - if((stat = nczm_concat(varpath,NCZARRAY,&key))) - goto done; - /* Download the nczarray object */ - if((stat=NCZ_readdict(map,key,&jncvar))) - goto done; - nullfree(key); key = NULL; - } else {/* format v2 */ - /* Extract the NCZ_V2_ARRAY dict */ - if((stat = NCJdictget(jvar,NCZ_V2_ARRAY,&jncvar))) goto done; - } - if(jncvar == NULL) {stat = NC_ENCZARR; goto done;} - assert((NCJsort(jncvar) == NCJ_DICT)); - /* Extract storage flag */ - if((stat = NCJdictget(jncvar,"storage",&jvalue))) - goto done; - if(jvalue != NULL) { - if(strcmp(NCJstring(jvalue),"chunked") == 0) { - var->storage = NC_CHUNKED; - } else if(strcmp(NCJstring(jvalue),"compact") == 0) { - var->storage = NC_COMPACT; - } else if(strcmp(NCJstring(jvalue),"scalar") == 0) { - var->storage = NC_CONTIGUOUS; - zvar->scalar = 1; - } else { /*storage = NC_CONTIGUOUS;*/ - var->storage = NC_CONTIGUOUS; - } - } - /* Extract dimnames list */ - switch ((stat = NCJdictget(jncvar,"dimrefs",&jdimrefs))) { - case NC_NOERR: /* Extract the dimref names */ - assert((NCJsort(jdimrefs) == NCJ_ARRAY)); - assert(NCJlength(jdimrefs) == rank); - for(j=0;jdim))) - goto done; - - /* Extract the dimids */ - for(j=0;jdimids[j] = var->dim[j]->hdr.id; - - /* At this point, we can finalize the filters */ - if((stat = NCZ_filter_setup(var))) goto done; - - /* Clean up from last cycle */ - nclistfreeall(dimnames); dimnames = nclistnew(); - nullfree(varpath); varpath = NULL; - nullfree(shapes); shapes = NULL; - if(formatv1) {NCJreclaim(jncvar); jncvar = NULL;} - NCJreclaim(jvar); jvar = NULL; - } - -done: - nullfree(shapes); - nullfree(varpath); - nullfree(key); - nclistfreeall(dimnames); - NCJreclaim(jvar); - if(formatv1) NCJreclaim(jncvar); - return THROW(stat); -} - -/** - * @internal Materialize subgroups into memory - * - * @param file Pointer to file info struct. - * @param grp Pointer to grp info struct. - * @param subgrpnames List of names of subgroups in this group - * - * @return ::NC_NOERR No error. - * @author Dennis Heimbigner - */ -static int -define_subgrps(NC_FILE_INFO_T* file, NC_GRP_INFO_T* grp, NClist* subgrpnames) -{ - int i,stat = NC_NOERR; - - /* Load each subgroup name in turn */ - for(i = 0; i < nclistlength(subgrpnames); i++) { - NC_GRP_INFO_T* g = NULL; - const char* gname = nclistget(subgrpnames,i); - char norm_name[NC_MAX_NAME]; - /* Check and normalize the name. */ - if((stat = nc4_check_name(gname, norm_name))) - goto done; - if((stat = nc4_grp_list_add(file, grp, norm_name, &g))) - goto done; - if(!(g->format_grp_info = calloc(1, sizeof(NCZ_GRP_INFO_T)))) - {stat = NC_ENOMEM; goto done;} - ((NCZ_GRP_INFO_T*)g->format_grp_info)->common.file = file; - } - - /* Recurse to fill in subgroups */ - for(i=0;ichildren);i++) { - NC_GRP_INFO_T* g = (NC_GRP_INFO_T*)ncindexith(grp->children,i); - if((stat = define_grp(file,g))) - goto done; - } - -done: - return THROW(stat); -} - -int -ncz_read_superblock(NC_FILE_INFO_T* file, char** nczarrvp, char** zarrfp) -{ - int stat = NC_NOERR; - NCjson* jnczgroup = NULL; - NCjson* jzgroup = NULL; - NCjson* jsuper = NULL; - NCjson* jtmp = NULL; - char* nczarr_version = NULL; - char* zarr_format = NULL; - NCZ_FILE_INFO_T* zinfo = (NCZ_FILE_INFO_T*)file->format_file_info; - - /* See if the V1 META-Root is being used */ - switch(stat = NCZ_downloadjson(zinfo->map, NCZMETAROOT, &jnczgroup)) { - case NC_EEMPTY: /* not there */ - stat = NC_NOERR; - break; - case NC_NOERR: - if((stat = NCJdictget(jnczgroup,"nczarr_version",&jtmp))) goto done; - nczarr_version = strdup(NCJstring(jtmp)); - break; - default: goto done; - } - /* Also gett Zarr Root Group */ - switch(stat = NCZ_downloadjson(zinfo->map, ZMETAROOT, &jzgroup)) { - case NC_NOERR: - break; - case NC_EEMPTY: /* not there */ - stat = NC_NOERR; - assert(jzgroup == NULL); - break; - default: goto done; - } - if(jzgroup != NULL) { - /* See if this NCZarr V2 */ - if((stat = NCJdictget(jzgroup,NCZ_V2_SUPERBLOCK,&jsuper))) goto done; - if(jsuper != NULL) { - /* Extract the equivalent attribute */ - if(jsuper->sort != NCJ_DICT) - {stat = NC_ENCZARR; goto done;} - if((stat = NCJdictget(jsuper,"version",&jtmp))) goto done; - nczarr_version = nulldup(NCJstring(jtmp)); - } - /* In any case, extract the zarr format */ - if((stat = NCJdictget(jzgroup,"zarr_format",&jtmp))) goto done; - zarr_format = nulldup(NCJstring(jtmp)); - } - /* Set the controls */ - if(jnczgroup == NULL && jsuper == NULL) { - zinfo->controls.flags |= FLAG_PUREZARR; - } else if(jnczgroup != NULL) { - zinfo->controls.flags |= FLAG_NCZARR_V1; - /* Also means file is read only */ - file->no_write = 1; - } else if(jsuper != NULL) { - /* ! FLAG_NCZARR_V1 && ! FLAG_PUREZARR */ - } - if(nczarrvp) {*nczarrvp = nczarr_version; nczarr_version = NULL;} - if(zarrfp) {*zarrfp = zarr_format; zarr_format = NULL;} -done: - nullfree(zarr_format); - nullfree(nczarr_version); - NCJreclaim(jzgroup); - NCJreclaim(jnczgroup); - return THROW(stat); -} - -/**************************************************/ -/* Utilities */ - -static int -parse_group_content(NCjson* jcontent, NClist* dimdefs, NClist* varnames, NClist* subgrps) -{ - int i,stat = NC_NOERR; - NCjson* jvalue = NULL; - - if((stat=NCJdictget(jcontent,"dims",&jvalue))) goto done; - if(jvalue != NULL) { - if(NCJsort(jvalue) != NCJ_DICT) {stat = THROW(NC_ENCZARR); goto done;} - /* Extract the dimensions defined in this group */ - for(i=0;imap,zakey,&jvar))) - goto done; - assert((NCJsort(jvar) == NCJ_DICT)); - nullfree(varkey); varkey = NULL; - nullfree(zakey); zakey = NULL; - /* Extract the shape */ - if((stat=NCJdictget(jvar,"shape",&jvalue))) goto done; - if((stat = decodeints(jvalue, shapes))) goto done; - -done: - NCJreclaim(jvar); - NCJreclaim(jvalue); - nullfree(varkey); varkey = NULL; - nullfree(zakey); zakey = NULL; - return THROW(stat); -} -#endif - -static int -searchvars(NCZ_FILE_INFO_T* zfile, NC_GRP_INFO_T* grp, NClist* varnames) -{ - int i,stat = NC_NOERR; - char* grpkey = NULL; - char* varkey = NULL; - char* zarray = NULL; - NClist* matches = nclistnew(); - - /* Compute the key for the grp */ - if((stat = NCZ_grpkey(grp,&grpkey))) goto done; - /* Get the map and search group */ - if((stat = nczmap_search(zfile->map,grpkey,matches))) goto done; - for(i=0;imap,zarray)) == NC_NOERR) - nclistpush(varnames,strdup(name)); - stat = NC_NOERR; - nullfree(varkey); varkey = NULL; - nullfree(zarray); zarray = NULL; - } - -done: - nullfree(grpkey); - nullfree(varkey); - nullfree(zarray); - nclistfreeall(matches); - return stat; -} - -static int -searchsubgrps(NCZ_FILE_INFO_T* zfile, NC_GRP_INFO_T* grp, NClist* subgrpnames) -{ - int i,stat = NC_NOERR; - char* grpkey = NULL; - char* subkey = NULL; - char* zgroup = NULL; - NClist* matches = nclistnew(); - - /* Compute the key for the grp */ - if((stat = NCZ_grpkey(grp,&grpkey))) goto done; - /* Get the map and search group */ - if((stat = nczmap_search(zfile->map,grpkey,matches))) goto done; - for(i=0;imap,zgroup)) == NC_NOERR) - nclistpush(subgrpnames,strdup(name)); - stat = NC_NOERR; - nullfree(subkey); subkey = NULL; - nullfree(zgroup); zgroup = NULL; - } - -done: - nullfree(grpkey); - nullfree(subkey); - nullfree(zgroup); - nclistfreeall(matches); - return stat; -} - -/* Convert a list of integer strings to 64 bit integers */ -static int -decodeints(NCjson* jshape, size64_t* shapes) -{ - int i, stat = NC_NOERR; - - for(i=0;iroot_grp; - NC_DIM_INFO_T* thed = NULL; - if((stat = nc4_dim_list_add(root, name, (size_t)dimlen, -1, &thed))) - goto done; - assert(thed != NULL); - /* Create struct for NCZ-specific dim info. */ - if (!(thed->format_dim_info = calloc(1, sizeof(NCZ_DIM_INFO_T)))) - {stat = NC_ENOMEM; goto done;} - ((NCZ_DIM_INFO_T*)thed->format_dim_info)->common.file = file; - *dimp = thed; thed = NULL; -done: - return stat; -} - - -/* -Given a list of segments, find corresponding group. -*/ -static int -locategroup(NC_FILE_INFO_T* file, size_t nsegs, NClist* segments, NC_GRP_INFO_T** grpp) -{ - int i, j, found, stat = NC_NOERR; - NC_GRP_INFO_T* grp = NULL; - - grp = file->root_grp; - for(i=0;ichildren);j++) { - NC_GRP_INFO_T* subgrp = (NC_GRP_INFO_T*)ncindexith(grp->children,j); - if(strcmp(subgrp->hdr.name,norm_name)==0) { - grp = subgrp; - found = 1; - break; - } - } - if(!found) {stat = NC_ENOGRP; goto done;} - } - /* grp should be group of interest */ - if(grpp) *grpp = grp; - -done: - return THROW(stat); -} - -static int -parsedimrefs(NC_FILE_INFO_T* file, NClist* dimnames, size64_t* shape, NC_DIM_INFO_T** dims, int create) -{ - int i, stat = NC_NOERR; - NClist* segments = NULL; - - for(i=0;idim);j++) { - d = (NC_DIM_INFO_T*)ncindexith(g->dim,j); - if(strcmp(d->hdr.name,dimname)==0) { - dims[i] = d; - break; - } - } - if(dims[i] == NULL && create) { - /* If not found and create then create it */ - if((stat = createdim(file, dimname, shape[i], &dims[i]))) - goto done; - } - assert(dims[i] != NULL); - assert(dims[i]->len == shape[i]); - } -done: - nclistfreeall(segments); - return THROW(stat); -} - -/** - * @internal Get the metadata for a variable. - * - * @param var Pointer to var info struct. - * - * @return ::NC_NOERR No error. - * @return ::NC_EBADID Bad ncid. - * @return ::NC_ENOMEM Out of memory. - * @return ::NC_EHDFERR HDF5 returned error. - * @return ::NC_EVARMETA Error with var metadata. - * @author Ed Hartnett - */ -int -ncz_get_var_meta(NC_FILE_INFO_T* file, NC_VAR_INFO_T* var) -{ - int retval = NC_NOERR; - - assert(file && var && var->format_var_info); - LOG((3, "%s: var %s", __func__, var->hdr.name)); - - /* Have we already read the var metadata? */ - if (var->meta_read) - return NC_NOERR; - -#ifdef LOOK - /* Get the current chunk cache settings. */ - if ((access_pid = H5Dget_access_plist(hdf5_var->hdf_datasetid)) < 0) - BAIL(NC_EVARMETA); - - /* Learn about current chunk cache settings. */ - if ((H5Pget_chunk_cache(access_pid, &(var->chunk_cache_nelems), - &(var->chunk_cache_size), &rdcc_w0)) < 0) - BAIL(NC_EHDFERR); - var->chunk_cache_preemption = rdcc_w0; - - /* Get the dataset creation properties. */ - if ((propid = H5Dget_create_plist(hdf5_var->hdf_datasetid)) < 0) - BAIL(NC_EHDFERR); - - /* Get var chunking info. */ - if ((retval = get_chunking_info(propid, var))) - BAIL(retval); - - /* Get filter info for a var. */ - if ((retval = get_filter_info(propid, var))) - BAIL(retval); - - /* Get fill value, if defined. */ - if ((retval = get_fill_info(propid, var))) - BAIL(retval); - - /* Is this a deflated variable with a chunksize greater than the - * current cache size? */ - if ((retval = nc4_adjust_var_cache(var))) - BAIL(retval); - - if (var->coords_read && !var->dimscale) - if ((retval = get_attached_info(var, hdf5_var, var->ndims, hdf5_var->hdf_datasetid))) - return retval; -#endif - - /* Remember that we have read the metadata for this var. */ - var->meta_read = NC_TRUE; - - return retval; -} - -#if 0 -int -ncz_create_superblock(NCZ_FILE_INFO_T* zinfo) -{ - int stat = NC_NOERR; - NCjson* json = NULL; - NCZMAP* map = NULL; - char version[1024]; - - ZTRACE(4,"zinfo=%s",zinfo->common.file->controller->path); - - /* If V2, then do not create a superblock per-se */ - if(!(zinfo->controls.flags & FLAG_NCZARR_V1)) goto done; - - map = zinfo->map; - - /* create superblock json */ - if((stat = NCJnew(NCJ_DICT,&json))) - goto done; - - /* fill */ - snprintf(version,sizeof(version),"%d",zinfo->zarr.zarr_version); - if((stat = NCJaddstring(json,NCJ_STRING,"zarr_format"))) goto done; - if((stat = NCJaddstring(json,NCJ_INT,version))) goto done; - if((stat = NCJaddstring(json,NCJ_STRING,NCZ_V2_VERSION))) goto done; - { - char ver[1024]; - snprintf(ver,sizeof(ver),"%lu.%lu.%lu", - zinfo->zarr.nczarr_version.major, - zinfo->zarr.nczarr_version.minor, - zinfo->zarr.nczarr_version.release); - if((stat = NCJaddstring(json,NCJ_STRING,ver))) goto done; - } - /* Write back to map */ - if((stat=NCZ_uploadjson(map,NCZMETAROOT,json))) - goto done; -done: - NCJreclaim(json); - return ZUNTRACE(stat); -} -#endif - -/* Compute the set of dim refs for this variable, taking purezarr and xarray into account */ -static int -computedimrefs(NC_FILE_INFO_T* file, NC_VAR_INFO_T* var, int purezarr, int xarray, int ndims, NClist* dimnames, size64_t* shapes, NC_DIM_INFO_T** dims) -{ - int stat = NC_NOERR; - int i; - int createdims = 0; /* 1 => we need to create the dims in root if they do not already exist */ - NCZ_FILE_INFO_T* zfile = (NCZ_FILE_INFO_T*)file->format_file_info; - NCZ_VAR_INFO_T* zvar = (NCZ_VAR_INFO_T*)(var->format_var_info); - NCjson* jatts = NULL; - - assert(zfile && zvar); - - /* xarray => purezarr */ - assert(!xarray || purezarr); - - if(xarray) {/* Read in the attributes to get xarray dimdef attribute; Note that it might not exist */ - char zdimname[4096]; - if(zvar->xarray == NULL) { - assert(nclistlength(dimnames) == 0); - if((stat = ncz_read_atts(file,(NC_OBJ*)var))) goto done; - } - if(zvar->xarray != NULL) { - /* convert xarray to the dimnames */ - for(i=0;ixarray);i++) { - snprintf(zdimname,sizeof(zdimname),"/%s",(const char*)nclistget(zvar->xarray,i)); - nclistpush(dimnames,strdup(zdimname)); - } - } - createdims = 1; /* may need to create them */ - } - - /* If pure zarr and we have no dimref names, then fake it */ - if(purezarr && nclistlength(dimnames) == 0) { - createdims = 1; - for(i=0;iformat_var_info; + + assert(zvar != NULL && zvar->cache != NULL); + assert(var->fill_value != NULL); + nullfree(zvar->cache->fillchunk); zvar->cache->fillchunk = NULL; + if((stat = NCZ_create_fill_chunk(zvar->cache->chunksize,var->type_info->size,var->fill_value,&zvar->cache->fillchunk))) + goto done; + +done: + return stat; +} + +#if 0 +/**************************************************/ +/* S3 utilities */ + +EXTERNL int +NCZ_s3urlprocess(NCURI* url, ZS3INFO* s3) +{ + int stat = NC_NOERR; + NCURI* url2 = NULL; + NClist* pathsegments = NULL; + const char* profile0 = NULL; + + if(url == NULL || s3 == NULL) + {stat = NC_EURL; goto done;} + /* Get current profile */ + if((stat = NC_getactives3profile(url,&profile0))) goto done; + if(profile0 == NULL) profile0 = "none"; + s3->profile = strdup(profile0); + + /* Rebuild the URL to path format and get a usable region*/ + if((stat = NC_s3urlrebuild(url,&url2,&s3->bucket,&s3->region))) goto done; + s3->host = strdup(url2->host); + /* construct the rootkey minus the leading bucket */ + pathsegments = nclistnew(); + if((stat = NC_split_delim(url2->path,'/',pathsegments))) goto done; + if(nclistlength(pathsegments) > 0) { + char* seg = nclistremove(pathsegments,0); + nullfree(seg); + } + if((stat = nczm_join(pathsegments,&s3->rootkey))) goto done; + +done: + ncurifree(url2); + nclistfreeall(pathsegments); + return stat; +} + +int +NCZ_s3clear(ZS3INFO* s3) +{ + if(s3) { + nullfree(s3->host); s3->host = NULL; + nullfree(s3->region); s3->region = NULL; + nullfree(s3->bucket); s3->bucket = NULL; + nullfree(s3->rootkey); s3->rootkey = NULL; + nullfree(s3->profile); s3->profile = NULL; + } + return NC_NOERR; +} +#endif + int NCZ_ischunkname(const char* name,char dimsep) { diff --git a/libnczarr/zvar.c b/libnczarr/zvar.c index 3346a8b125..8021d98614 100644 --- a/libnczarr/zvar.c +++ b/libnczarr/zvar.c @@ -12,6 +12,10 @@ #include "zincludes.h" #include /* For pow() used below. */ +/* Forward */ +static int NCZ_get_vars_scalar(int ncid, NC_FILE_INFO_T* h5, NC_VAR_INFO_T* var, size64_t* start, size64_t* count, size64_t* stride, void* data, nc_type mem_nc_type); +static int NCZ_put_vars_scalar(int ncid, NC_FILE_INFO_T* h5, NC_VAR_INFO_T* var, size64_t* start, size64_t* count, size64_t* stride, const void* data, nc_type mem_nc_type); + #ifdef LOGGING static void reportchunking(const char* title, NC_VAR_INFO_T* var) @@ -389,6 +393,9 @@ NCZ_def_var(int ncid, const char *name, nc_type xtype, int ndims, var->meta_read = NC_TRUE; var->atts_read = NC_TRUE; + /* Indicate we do not yet have a quantizer */ + var->quantize_mode = -1; + /* Set the filter list */ assert(var->filters == NULL); var->filters = (void*)nclistnew(); @@ -510,7 +517,8 @@ static int ncz_def_var_extra(int ncid, int varid, int *shuffle, int *unused1, int *unused2, int *fletcher32, int *storagep, const size_t *chunksizes, int *no_fill, - const void *fill_value, int *endianness) + const void *fill_value, int *endianness, + int *quantize_mode, int *nsd) { NC_GRP_INFO_T *grp; NC_FILE_INFO_T *h5; @@ -522,13 +530,15 @@ ncz_def_var_extra(int ncid, int varid, int *shuffle, int *unused1, LOG((2, "%s: ncid 0x%x varid %d", __func__, ncid, varid)); - ZTRACE(2,"ncid=%d varid=%d shuffle=%d fletcher32=%d no_fill=%d, fill_value=%p endianness=%d", + ZTRACE(2,"ncid=%d varid=%d shuffle=%d fletcher32=%d no_fill=%d, fill_value=%p endianness=%d quantize_mode=%d nsd=%d", ncid,varid, (shuffle?*shuffle:-1), (fletcher32?*fletcher32:-1), (no_fill?*no_fill:-1), fill_value, - (endianness?*endianness:-1)); + (quantize_mode?*quantize_mode:-1), + (nsd?*nsd:-1) + ); /* Find info for this file and group, and set pointer to each. */ if ((retval = nc4_find_nc_grp_h5(ncid, NULL, &grp, &h5))) @@ -717,13 +727,25 @@ ncz_def_var_extra(int ncid, int varid, int *shuffle, int *unused1, retval = NCZ_del_att(ncid, varid, _FillValue); if (retval && retval != NC_ENOTATT) goto done; - - /* Create a _FillValue attribute. */ + /* Create a _FillValue attribute; also sets var->fill_value */ if ((retval = nc_put_att(ncid, varid, _FillValue, var->type_info->hdr.id, 1, fill_value))) goto done; + /* Rebuild the fill chunk */ + if((retval = ncz_rebuild_fill_chunk(var))) goto done; } - +#if 0 + else if (fill_value) { + /* rebuild fill value even if no_fill */ + nullfree(var->fill_value); var->fill_value = NULL; + if((var->fill_value = malloc(var->type_info->size))==NULL) + {retval = NC_ENOMEM; goto done;} + memcpy(var->fill_value,fill_value,var->type_info->size); + /* Rebuild the fill chunk */ + if((retval = ncz_rebuild_fill_chunk(var))) goto done; + } +#endif + /* Is the user setting the endianness? */ if (endianness) { @@ -750,6 +772,46 @@ ncz_def_var_extra(int ncid, int varid, int *shuffle, int *unused1, var->endianness = *endianness; } + /* Remember quantization settings. They will be used when data are + * written. */ + if (quantize_mode) + { + /* Only two valid mode settings. */ + if (*quantize_mode != NC_NOQUANTIZE && + *quantize_mode != NC_QUANTIZE_BITGROOM) + return NC_EINVAL; + + if (*quantize_mode == NC_QUANTIZE_BITGROOM) + { + /* Only float and double types can have quantization. */ + if (var->type_info->hdr.id != NC_FLOAT && + var->type_info->hdr.id != NC_DOUBLE) + return NC_EINVAL; + + /* For bitgroom, number of significant digits is required. */ + if (!nsd) + return NC_EINVAL; + + /* NSD must be in range. */ + if (*nsd <= 0) + return NC_EINVAL; + if (var->type_info->hdr.id == NC_FLOAT && + *nsd > NC_QUANTIZE_MAX_FLOAT_NSD) + return NC_EINVAL; + if (var->type_info->hdr.id == NC_DOUBLE && + *nsd > NC_QUANTIZE_MAX_DOUBLE_NSD) + return NC_EINVAL; + + var->nsd = *nsd; + } + + var->quantize_mode = *quantize_mode; + + /* If quantization is turned off, then set nsd to 0. */ + if (*quantize_mode == NC_NOQUANTIZE) + var->nsd = 0; + } + done: return ZUNTRACE(retval); } @@ -782,7 +844,7 @@ NCZ_def_var_deflate(int ncid, int varid, int shuffle, int deflate, int stat = NC_NOERR; unsigned int level = (unsigned int)deflate_level; /* Set shuffle first */ - if((stat = ncz_def_var_extra(ncid, varid, &shuffle, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL))) goto done; + if((stat = ncz_def_var_extra(ncid, varid, &shuffle, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL))) goto done; if(deflate) stat = nc_def_var_filter(ncid, varid, H5Z_FILTER_DEFLATE,1,&level); if(stat) goto done; @@ -812,7 +874,7 @@ int NCZ_def_var_fletcher32(int ncid, int varid, int fletcher32) { return ncz_def_var_extra(ncid, varid, NULL, NULL, NULL, &fletcher32, - NULL, NULL, NULL, NULL, NULL); + NULL, NULL, NULL, NULL, NULL, NULL, NULL); } /** @@ -841,7 +903,7 @@ int NCZ_def_var_chunking(int ncid, int varid, int contiguous, const size_t *chunksizesp) { return ncz_def_var_extra(ncid, varid, NULL, NULL, NULL, NULL, - &contiguous, chunksizesp, NULL, NULL, NULL); + &contiguous, chunksizesp, NULL, NULL, NULL, NULL, NULL); } /** @@ -886,7 +948,7 @@ ncz_def_var_chunking_ints(int ncid, int varid, int contiguous, int *chunksizesp) cs[i] = chunksizesp[i]; retval = ncz_def_var_extra(ncid, varid, NULL, NULL, NULL, NULL, - &contiguous, cs, NULL, NULL, NULL); + &contiguous, cs, NULL, NULL, NULL, NULL, NULL); if (var->ndims) free(cs); @@ -920,7 +982,7 @@ int NCZ_def_var_fill(int ncid, int varid, int no_fill, const void *fill_value) { return ncz_def_var_extra(ncid, varid, NULL, NULL, NULL, NULL, NULL, - NULL, &no_fill, fill_value, NULL); + NULL, &no_fill, fill_value, NULL, NULL, NULL); } /** @@ -949,7 +1011,138 @@ int NCZ_def_var_endian(int ncid, int varid, int endianness) { return ncz_def_var_extra(ncid, varid, NULL, NULL, NULL, NULL, NULL, - NULL, NULL, NULL, &endianness); + NULL, NULL, NULL, &endianness, NULL, NULL); +} + +/** + * @internal Set quantization settings on a variable. This is + * called by nc_def_var_quantize(). + * + * Quantization allows the user to specify a number of significant + * digits for variables of type ::NC_FLOAT or ::NC_DOUBLE. (Attempting + * to set quantize for other types will result in an ::NC_EINVAL + * error.) + * + * When quantize is turned on, and the number of significant digits + * has been specified, then the netCDF library will apply all zeros or + * all ones (alternating) to bits which are not needed to specify the + * value to the number of significant digits. This will change the + * value of the data, but will make it more compressable. + * + * Quantizing the data does not reduce the size of the data on disk, + * but combining quantize with compression will allow for better + * compression. Since the data values are changed, the use of quantize + * and compression such as deflate constitute lossy compression. + * + * Producers of large datasets may find that using quantize with + * compression will result in significant improvent in the final data + * size. + * + * Variables which use quantize will have added an attribute with name + * ::NC_QUANTIZE_ATT_NAME, which will contain the number of + * significant digits. Users should not delete or change this + * attribute. This is the only record that quantize has been applied + * to the data. + * + * As with the deflate settings, quantize settings may only be + * modified before the first call to nc_enddef(). Once nc_enddef() is + * called for the file, quantize settings for any variable in the file + * may not be changed. + * + * Use of quantization is fully backwards compatible with existing + * versions and packages that can read compressed netCDF data. A + * variable which has been quantized is readable to older versions of + * the netCDF libraries, and to netCDF-Java. + * + * @param ncid File ID. + * @param varid Variable ID. NC_GLOBAL may not be used. + * @param quantize_mode Quantization mode. May be ::NC_NOQUANTIZE or + * ::NC_QUANTIZE_BITGROOM. + * @param nsd Number of significant digits. May be any integer from 1 + * to ::NC_QUANTIZE_MAX_FLOAT_NSD (for variables of type ::NC_FLOAT) or + * ::NC_QUANTIZE_MAX_DOUBLE_NSD (for variables of type ::NC_DOUBLE). + * + * @returns ::NC_NOERR No error. + * @returns ::NC_EBADID Bad ncid. + * @returns ::NC_ENOTVAR Invalid variable ID. + * @returns ::NC_ENOTNC4 Attempting netcdf-4 operation on file that is + * not netCDF-4/HDF5. + * @returns ::NC_ELATEDEF Too late to change settings for this variable. + * @returns ::NC_ENOTINDEFINE Not in define mode. + * @returns ::NC_EINVAL Invalid input + * @author Ed Hartnett, Dennis Heimbigner + */ +int +NCZ_def_var_quantize(int ncid, int varid, int quantize_mode, int nsd) +{ + return ncz_def_var_extra(ncid, varid, NULL, NULL, NULL, NULL, + NULL, NULL, NULL, NULL, NULL, + &quantize_mode, &nsd); +} + +/** +Ensure that the quantize information for a variable is defined. +Keep a flag in the NCZ_VAR_INFO_T struct to indicate if quantize +info is defined, and if not, read the attribute. +*/ +int +NCZ_ensure_quantizer(int ncid, NC_VAR_INFO_T* var) +{ + int nsd = 0; + + /* Read the attribute */ + if(NCZ_get_att(ncid,var->hdr.id,NC_QUANTIZE_ATT_NAME,&nsd,NC_INT)) { + var->quantize_mode = NC_NOQUANTIZE; + var->nsd = 0; + } else { + var->quantize_mode = NC_QUANTIZE_BITGROOM; + var->nsd = nsd; + } + if(var->quantize_mode < 0) var->quantize_mode = 0; + return NC_NOERR; +} + +/** + * @internal Get quantize information about a variable. Pass NULL for + * whatever you don't care about. Note that this can require reading + * all the attributes for the variable. + * + * @param ncid File ID. + * @param varid Variable ID. + * @param quantize_modep Gets quantize mode. + * @param nsdp Gets Number of Significant Digits if quantize is in use. + * + * @returns ::NC_NOERR No error. + * @returns ::NC_EBADID Bad ncid. + * @returns ::NC_ENOTVAR Bad varid. + * @returns ::NC_EINVAL Invalid input. + * @author Ed Hartnett + */ +int +NCZ_inq_var_quantize(int ncid, int varid, int *quantize_modep, + int *nsdp) +{ + NC_VAR_INFO_T *var; + int retval; + + LOG((2, "%s: ncid 0x%x varid %d", __func__, ncid, varid)); + + /* Find info for this file and group, and set pointer to each. */ + /* Get pointer to the var. */ + if ((retval = nc4_find_grp_h5_var(ncid, varid, NULL, NULL, &var))) + return retval; + if (!var) + return NC_ENOTVAR; + assert(var->hdr.id == varid); + if(var->quantize_mode == -1) + {if((retval = NCZ_ensure_quantizer(ncid, var))) return retval;} + /* Copy the data to the user's data buffers. */ + if (quantize_modep) + *quantize_modep = var->quantize_mode; + if (nsdp) + *nsdp = var->nsd; + + return NC_NOERR; } /** @@ -1426,11 +1619,16 @@ NCZ_put_vars(int ncid, int varid, const size_t *startp, const size_t *countp, } #endif - /* Are we going to convert any data? (No converting of user-defined types - except enum). */ - if (mem_nc_type != var->type_info->hdr.id && + /* If this is a scalar, then do that separately */ + if(var->ndims == 0) + return NCZ_put_vars_scalar(ncid,h5,var,start,count,stride,data,mem_nc_type); + + /* Are we going to convert any data? (No converting of compound or + * opaque or vlen types.) We also need to call this code if we are doing + * quantization. */ + if ((mem_nc_type != var->type_info->hdr.id && mem_nc_type != NC_COMPOUND && mem_nc_type != NC_OPAQUE - && mem_nc_type != NC_VLEN) + && mem_nc_type != NC_VLEN) || var->quantize_mode > 0) { size_t file_type_size; @@ -1525,6 +1723,7 @@ NCZ_put_vars(int ncid, int varid, const size_t *startp, const size_t *countp, /* Do we need to convert the data? */ if (need_to_convert) { + if(var->quantize_mode < 0) {if((retval = NCZ_ensure_quantizer(ncid,var))) BAIL(retval);} if ((retval = nc4_convert_type(data, bufr, mem_nc_type, var->type_info->hdr.id, len, &range_error, var->fill_value, (h5->cmode & NC_CLASSIC_MODEL), @@ -1621,7 +1820,6 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, size64_t fmaxdims[NC_MAX_VAR_DIMS]; size64_t start[NC_MAX_VAR_DIMS]; size64_t stride[NC_MAX_VAR_DIMS]; - void *fillvalue = NULL; int no_read = 0, provide_fill = 0; int fill_value_size[NC_MAX_VAR_DIMS]; int retval, range_error = 0, i, d2; @@ -1694,6 +1892,10 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, assert(var->type_info && var->type_info->size && var->type_info->format_type_info); + /* If this is a scalar, then do that separately */ + if(var->ndims == 0) + return NCZ_get_vars_scalar(ncid,h5,var,start,count,stride,data,mem_nc_type); + /* Later on, we will need to know the size of this type in the * file. */ file_type_size = var->type_info->size; @@ -1705,7 +1907,7 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, { /* We must convert - allocate a buffer. */ need_to_convert++; - for (d2 = 0; d2 < (var->ndims+zvar->scalar); d2++) + for (d2 = 0; d2 < (var->ndims); d2++) len *= countp[d2]; LOG((4, "converting data for var %s type=%d len=%d", var->hdr.name, var->type_info->hdr.id, len)); @@ -1861,8 +2063,8 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, /* Get the fill value from the ZARR variable. Memory will be * allocated. */ - if (ncz_get_fill_value(h5, var, &fillvalue) < 0) - BAIL(NC_EHDFERR); + if ((retval = ncz_ensure_fill_value(var))) + BAIL(retval); /* How many fill values do we need? */ for (fill_len = 1, d2 = 0; d2 < var->ndims; d2++) @@ -1875,25 +2077,27 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, if (var->type_info->nc_type_class == NC_STRING) { - if (*(char **)fillvalue) + const char** strfill = (const char**)var->fill_value; + if (*strfill) { - if (!(*(char **)filldata = strdup(*(char **)fillvalue))) + filldata = strdup(*strfill); + if (!filldata) BAIL(NC_ENOMEM); } else - *(char **)filldata = NULL; + filldata = NULL; } else if (var->type_info->nc_type_class == NC_VLEN) { - if (fillvalue) + if (var->fill_value) { - memcpy(filldata,fillvalue,file_type_size); + memcpy(filldata,var->fill_value,file_type_size); } else { - *(char **)filldata = NULL; + filldata = NULL; } } else - memcpy(filldata, fillvalue, file_type_size); + memcpy(filldata, var->fill_value, file_type_size); filldata = (char *)filldata + file_type_size; } } @@ -1901,6 +2105,7 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, /* Convert data type if needed. */ if (need_to_convert) { + if(var->quantize_mode < 0) {if((retval = NCZ_ensure_quantizer(ncid,var))) BAIL(retval);} if ((retval = nc4_convert_type(bufr, data, var->type_info->hdr.id, mem_nc_type, len, &range_error, var->fill_value, (h5->cmode & NC_CLASSIC_MODEL), var->quantize_mode, @@ -1930,15 +2135,63 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, #endif if (need_to_convert && bufr) free(bufr); - if (fillvalue) + + /* If there was an error return it, otherwise return any potential + range error value. If none, return NC_NOERR as usual.*/ + if (retval) + return THROW(retval); + if (range_error) + return THROW(NC_ERANGE); + return NC_NOERR; +} + +static int +NCZ_get_vars_scalar(int ncid, NC_FILE_INFO_T* h5, NC_VAR_INFO_T* var, size64_t* start, size64_t* count, size64_t* stride, void* data, nc_type mem_nc_type) +{ + int retval = NC_NOERR; + size_t file_type_size = var->type_info->size; + int range_error = 0; + void *bufr = NULL; + int need_to_convert = 0; + + /* Are we going to convert any data? (No converting of compound or + * opaque types.) */ + if((mem_nc_type != var->type_info->hdr.id && + mem_nc_type != NC_COMPOUND && mem_nc_type != NC_OPAQUE) + || var->quantize_mode > 0) + { + /* We must convert - allocate a buffer. */ + need_to_convert++; + if (!(bufr = malloc(file_type_size))) + BAIL(NC_ENOMEM); + } else { + if (!bufr) + bufr = data; + } + if((retval = NCZ_transferslice(var, READING, start, count, stride, bufr, var->type_info->hdr.id))) + BAIL(retval); + + /* Convert data type if needed. */ + if (need_to_convert) { - if (var->type_info->nc_type_class == NC_VLEN) - nc_free_vlen((nc_vlen_t *)fillvalue); - else if (var->type_info->nc_type_class == NC_STRING && *(char **)fillvalue) - free(*(char **)fillvalue); - free(fillvalue); + if(var->quantize_mode < 0) {if((retval = NCZ_ensure_quantizer(ncid,var))) BAIL(retval);} + if ((retval = nc4_convert_type(bufr, data, var->type_info->hdr.id, mem_nc_type, + 1, &range_error, var->fill_value, + (h5->cmode & NC_CLASSIC_MODEL), var->quantize_mode, + var->nsd))) + BAIL(retval); + /* For strict netcdf-3 rules, ignore erange errors between UBYTE + * and BYTE types. */ + if ((h5->cmode & NC_CLASSIC_MODEL) && + (var->type_info->hdr.id == NC_UBYTE || var->type_info->hdr.id == NC_BYTE) && + (mem_nc_type == NC_UBYTE || mem_nc_type == NC_BYTE) && + range_error) + range_error = 0; } +exit: + if (need_to_convert && bufr) + free(bufr); /* If there was an error return it, otherwise return any potential range error value. If none, return NC_NOERR as usual.*/ if (retval) @@ -1948,6 +2201,70 @@ NCZ_get_vars(int ncid, int varid, const size_t *startp, const size_t *countp, return NC_NOERR; } +static int +NCZ_put_vars_scalar(int ncid, NC_FILE_INFO_T* h5, NC_VAR_INFO_T* var, size64_t* start, size64_t* count, size64_t* stride, const void* data, nc_type mem_nc_type) +{ + int retval = NC_NOERR; + size_t file_type_size = var->type_info->size; + int range_error = 0; + void *bufr = NULL; + int need_to_convert = 0; + + if (mem_nc_type >= NC_FIRSTUSERTYPEID) + return THROW(NC_EINVAL); + + /* Are we going to convert any data? (No converting of compound or + * opaque types.) */ + if((mem_nc_type != var->type_info->hdr.id && + mem_nc_type != NC_COMPOUND && mem_nc_type != NC_OPAQUE) + || var->quantize_mode > 0) + { + /* We must convert - allocate a buffer. */ + need_to_convert++; + if (!(bufr = malloc(file_type_size))) + BAIL(NC_ENOMEM); + } else { + if (!bufr) + bufr = (void*)data; + } + + /* Convert data type if needed. */ + if (need_to_convert) + { + if(var->quantize_mode < 0) {if((retval = NCZ_ensure_quantizer(ncid,var))) BAIL(retval);} + if ((retval = nc4_convert_type(data, bufr, var->type_info->hdr.id, mem_nc_type, + 1, &range_error, var->fill_value, + (h5->cmode & NC_CLASSIC_MODEL), var->quantize_mode, + var->nsd))) + BAIL(retval); + } + + if((retval = NCZ_transferslice(var, WRITING, start, count, stride, bufr, var->type_info->hdr.id))) + BAIL(retval); + + /* Remember that we have written to this var so that Fill Value + * can't be set for it. */ + if (!var->written_to) + var->written_to = NC_TRUE; + + /* For strict netcdf-3 rules, ignore erange errors between UBYTE + * and BYTE types. */ + if ((h5->cmode & NC_CLASSIC_MODEL) && + (var->type_info->hdr.id == NC_UBYTE || var->type_info->hdr.id == NC_BYTE) && + (mem_nc_type == NC_UBYTE || mem_nc_type == NC_BYTE) && + range_error) + range_error = 0; + +exit: + if (need_to_convert && bufr) + free(bufr); + if (retval) + return THROW(retval); + if (range_error) + return THROW(NC_ERANGE); + return NC_NOERR; +} + /** * @internal Get all the information about a variable. Pass NULL for * whatever you don't care about. diff --git a/libnczarr/zwalk.c b/libnczarr/zwalk.c index 635925ab36..db68c09d06 100644 --- a/libnczarr/zwalk.c +++ b/libnczarr/zwalk.c @@ -122,18 +122,23 @@ NCZ_transferslice(NC_VAR_INFO_T* var, int reading, common.typesize = typesize; common.cache = zvar->cache; - /* We need to talk scalar into account */ - common.rank = var->ndims + zvar->scalar; + /* We need to take scalar into account */ + common.rank = var->ndims; common.scalar = zvar->scalar; common.swap = (zfile->native_endianness == var->endianness ? 0 : 1); common.chunkcount = 1; for(r=0;rchunksizes); + assert(r == 0); + assert(var->chunksizes[0] == 1); + } else { dimlens[r] = var->dim[r]->len; - chunklens[r] = var->chunksizes[r]; + } + chunklens[r] = var->chunksizes[r]; slices[r].start = start[r]; slices[r].stride = stride[r]; slices[r].stop = minimum(start[r]+(count[r]*stride[r]),dimlens[r]); @@ -157,9 +162,6 @@ NCZ_transferslice(NC_VAR_INFO_T* var, int reading, common.reader.source = ((NCZ_VAR_INFO_T*)(var->format_var_info))->cache; common.reader.read = readfromcache; - /* verify */ - assert(var->no_fill || var->fill_value != NULL); - if(common.scalar) { if((stat = NCZ_transferscalar(&common))) goto done; } diff --git a/libnczarr/zxcache.c b/libnczarr/zxcache.c index e8602af477..f4ca32c573 100644 --- a/libnczarr/zxcache.c +++ b/libnczarr/zxcache.c @@ -121,15 +121,6 @@ fprintf(stderr,"xxx: adjusting cache for: %s\n",var->hdr.name); #endif /* One more thing, adjust the chunksize */ zvar->cache->chunksize = zvar->chunksize; - /* and also rebuild the fillchunk */ - nullfree(zvar->cache->fillchunk); - zvar->cache->fillchunk = NULL; - if(var->no_fill) - stat = NCZ_create_fill_chunk(zvar->cache->chunksize,var->type_info->size,NULL,&zvar->cache->fillchunk); - else { - assert(var->fill_value != NULL); - stat = NCZ_create_fill_chunk(zvar->cache->chunksize,var->type_info->size,var->fill_value,&zvar->cache->fillchunk); - } return stat; } @@ -172,6 +163,20 @@ NCZ_create_chunk_cache(NC_VAR_INFO_T* var, size64_t chunksize, char dimsep, NCZC cache->maxentries = 1; #endif + /* build the fillchunk */ + nullfree(cache->fillchunk); + cache->fillchunk = NULL; + /* Make sure the fill_value is defined */ + if((stat = ncz_ensure_fill_value(var))) goto done; /* ensure var->fill_value is set */ + assert(var->fill_value != NULL); + if(var->no_fill) + stat = NCZ_create_fill_chunk(cache->chunksize,var->type_info->size,NULL,&cache->fillchunk); + else { + assert(var->fill_value != NULL); + stat = NCZ_create_fill_chunk(cache->chunksize,var->type_info->size,var->fill_value,&cache->fillchunk); + } + if(stat) goto done; + #ifdef DEBUG fprintf(stderr,"%s.cache: nelems=%ld size=%ld\n", var->hdr.name,(unsigned long)cache->maxentries,(unsigned long)cache->maxsize); @@ -180,6 +185,7 @@ NCZ_create_chunk_cache(NC_VAR_INFO_T* var, size64_t chunksize, char dimsep, NCZC if((cache->mru = nclistnew()) == NULL) {stat = NC_ENOMEM; goto done;} nclistsetalloc(cache->mru,cache->maxentries); + if(cachep) {*cachep = cache; cache = NULL;} done: nullfree(fill); diff --git a/ncgen/genbin.c b/ncgen/genbin.c index 6becaa7580..3768a8a960 100644 --- a/ncgen/genbin.c +++ b/ncgen/genbin.c @@ -257,6 +257,11 @@ genbin_definespecialattributes(Symbol* var) } CHECK_ERR(stat); } + if(special->flags & _QUANTIZE_FLAG) { + stat = nc_def_var_quantize(var->container->nc_id, + var->nc_id, special->_Quantizer, special->_NSD); + CHECK_ERR(stat); + } return stat; } #endif /*USE_NETCDF4*/ diff --git a/ncgen/genc.c b/ncgen/genc.c index b3b0c414b8..6843e88bc5 100644 --- a/ncgen/genc.c +++ b/ncgen/genc.c @@ -539,6 +539,21 @@ genc_definespecialattributes(Symbol* vsym) codelined(1,"CHECK_ERR(stat);"); } } + if(special->flags & _QUANTIZE_FLAG) { + const char* alg = NULL; + switch(special->_Quantizer) { + case NC_QUANTIZE_BITGROOM: alg = "NC_QUANTIZE_BITGROOM"; + default: alg = "NC_NOQUANTIZE"; + } + bbprintf0(stmt, + " stat = nc_def_var_quantize(%s, %s, %s, %d);\n", + groupncid(vsym->container), + varncid(vsym), + alg, special->_NSD + ); + codedump(stmt); + codelined(1,"CHECK_ERR(stat);"); + } } #endif /*USE_NETCDF4*/ diff --git a/ncgen/ncgen.h b/ncgen/ncgen.h index 855d39ce84..781d541f13 100644 --- a/ncgen/ncgen.h +++ b/ncgen/ncgen.h @@ -80,6 +80,7 @@ various C global variables #define _FORMAT_FLAG 0x800 #define _FILTER_FLAG 0x1000 #define _CODECS_FLAG 0x2000 +#define _QUANTIZE_FLAG 0x4000 extern struct Specialtoken { char* name; @@ -122,6 +123,8 @@ typedef struct Specialdata { int _Shuffle; /* 0 => false, 1 => true*/ int _Endianness; /* 1 =>little, 2 => big*/ int _Fill ; /* 0 => false, 1 => true WATCHOUT: this is inverse of NOFILL*/ + int _Quantizer; /* algorithm */ + int _NSD; /* No. of significant digits */ NC_H5_Filterspec** _Filters; size_t nfilters; /* |filters| */ char* _Codecs; /* in JSON form */ diff --git a/ncgen/ncgen.l b/ncgen/ncgen.l index fd09595a12..4b9423bb90 100644 --- a/ncgen/ncgen.l +++ b/ncgen/ncgen.l @@ -138,6 +138,7 @@ struct Specialtoken specials[] = { {"_SuperblockVersion",_SUPERBLOCK,_SUPERBLOCK_FLAG}, {"_Filter",_FILTER,_FILTER_FLAG}, {"_Codecs",_CODECS,_CODECS_FLAG}, +{"_QuantizeBitgroomNumberOfSignificantDigits",_QUANTIZE,_QUANTIZE_FLAG}, {NULL,0} /* null terminate */ }; @@ -216,7 +217,7 @@ NUMBER [+-]?[0-9][0-9]*[Uu]?([BbSs]|[Ll]|[Ll][Ll])? DBLNUMBER [+-]?[0-9]*\.[0-9]*{exp}?[LlDd]?|[+-]?[0-9]*{exp}[LlDd]? FLTNUMBER [+-]?[0-9]*\.[0-9]*{exp}?[Ff]|[+-]?[0-9]*{exp}[Ff] -SPECIAL "_FillValue"|"_Format"|"_Storage"|"_ChunkSizes"|"_Fletcher32"|"_DeflateLevel"|"_Shuffle"|"_Endianness"|"_NoFill"|"_NCProperties"|"_IsNetcdf4"|"_SuperblockVersion"|"_Filter"|"_Codecs" +SPECIAL "_FillValue"|"_Format"|"_Storage"|"_ChunkSizes"|"_Fletcher32"|"_DeflateLevel"|"_Shuffle"|"_Endianness"|"_NoFill"|"_NCProperties"|"_IsNetcdf4"|"_SuperblockVersion"|"_Filter"|"_Codecs"|"_QuantizeBitgroomNumberOfSignificantDIgits" USASCII [\x01-\x7F] diff --git a/ncgen/ncgen.y b/ncgen/ncgen.y index feba80248e..78fa44f21e 100644 --- a/ncgen/ncgen.y +++ b/ncgen/ncgen.y @@ -214,6 +214,7 @@ NCConstant* constant; _SUPERBLOCK _FILTER _CODECS + _QUANTIZE DATASETID %type ident typename primtype dimd varspec @@ -769,6 +770,8 @@ attrdecl: {$$ = makespecial(_FILTER_FLAG,$1,NULL,(void*)$5,ISCONST);} | ambiguous_ref ':' _CODECS '=' conststring {$$ = makespecial(_CODECS_FLAG,$1,NULL,(void*)$5,ISCONST);} + | ambiguous_ref ':' _QUANTIZE '=' constint + {$$ = makespecial(_QUANTIZE_FLAG,$1,NULL,(void*)$5,ISCONST);} | ambiguous_ref ':' _NOFILL '=' constbool {$$ = makespecial(_NOFILL_FLAG,$1,NULL,(void*)$5,ISCONST);} | ':' _FORMAT '=' conststring @@ -1230,6 +1233,7 @@ makespecial(int tag, Symbol* vsym, Symbol* tsym, void* data, int isconst) break; case _SUPERBLOCK_FLAG: case _DEFLATE_FLAG: + case _QUANTIZE_FLAG: tmp = nullconst(); tmp->nctype = NC_INT; convert1(con,tmp); @@ -1324,6 +1328,11 @@ makespecial(int tag, Symbol* vsym, Symbol* tsym, void* data, int isconst) special->_DeflateLevel = idata; special->flags |= _DEFLATE_FLAG; break; + case _QUANTIZE_FLAG: + special->_Quantizer = NC_QUANTIZE_BITGROOM; + special->_NSD = idata; + special->flags |= _QUANTIZE_FLAG; + break; case _SHUFFLE_FLAG: special->_Shuffle = tf; special->flags |= _SHUFFLE_FLAG; diff --git a/ncgen/ncgenl.c b/ncgen/ncgenl.c index 2880e0e8d7..8fe8f41d3a 100644 --- a/ncgen/ncgenl.c +++ b/ncgen/ncgenl.c @@ -619,7 +619,7 @@ struct yy_trans_info flex_int32_t yy_verify; flex_int32_t yy_nxt; }; -static const flex_int16_t yy_accept[429] = +static const flex_int16_t yy_accept[469] = { 0, 0, 0, 51, 51, 0, 0, 55, 53, 1, 49, 53, 53, 53, 53, 43, 37, 41, 41, 40, 40, @@ -634,39 +634,43 @@ static const flex_int16_t yy_accept[429] = 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, - 40, 40, 40, 0, 0, 51, 0, 52, 39, 45, - 0, 0, 0, 0, 43, 0, 0, 43, 2, 37, - 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, - 40, 40, 40, 40, 40, 40, 36, 33, 40, 40, + 40, 40, 40, 40, 0, 0, 51, 0, 52, 39, + 45, 0, 0, 0, 0, 43, 0, 0, 43, 2, + 37, 0, 0, 0, 0, 0, 0, 0, 4, 0, + 0, 40, 40, 40, 40, 40, 40, 36, 33, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, - 40, 13, 40, 33, 40, 40, 40, 40, 40, 40, - 40, 40, 40, 40, 40, 40, 0, 48, 0, 0, + 40, 40, 40, 13, 40, 33, 40, 40, 40, 40, + 40, 40, 40, 40, 40, 40, 40, 40, 0, 48, + + 0, 0, 43, 0, 37, 0, 0, 0, 0, 0, + 0, 0, 4, 42, 42, 0, 40, 40, 34, 40, + 40, 35, 40, 40, 40, 40, 40, 40, 40, 40, + 40, 40, 40, 40, 40, 40, 40, 11, 10, 40, + 40, 40, 40, 6, 40, 40, 40, 40, 21, 40, + 40, 40, 20, 40, 40, 40, 40, 40, 16, 40, + 40, 40, 40, 0, 0, 34, 0, 37, 0, 0, + 0, 0, 0, 0, 0, 40, 40, 40, 40, 40, + 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, + 40, 40, 40, 40, 40, 40, 29, 40, 40, 8, - 43, 0, 37, 0, 0, 0, 0, 0, 0, 0, - 4, 42, 42, 0, 40, 40, 34, 40, 40, 35, + 40, 17, 40, 40, 40, 40, 12, 40, 40, 40, + 14, 40, 40, 23, 40, 40, 40, 46, 47, 0, + 0, 0, 0, 40, 40, 40, 31, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, - 40, 40, 40, 40, 11, 10, 40, 40, 40, 40, - 6, 40, 40, 40, 40, 21, 40, 40, 40, 20, - 40, 40, 40, 40, 40, 16, 40, 40, 40, 40, - 0, 0, 34, 0, 37, 0, 0, 0, 0, 0, - 0, 0, 40, 40, 40, 40, 40, 40, 40, 40, + 40, 40, 40, 40, 40, 9, 30, 40, 7, 19, + 5, 26, 18, 40, 40, 15, 40, 0, 0, 40, + 40, 40, 40, 40, 38, 40, 40, 40, 40, 40, + 40, 40, 40, 40, 40, 40, 40, 22, 40, 40, + 40, 40, 0, 40, 32, 40, 40, 40, 40, 40, + 40, 40, 40, 40, 40, 5, 40, 40, 24, 40, + + 40, 32, 32, 25, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, - 40, 40, 29, 40, 40, 8, 40, 17, 40, 40, - - 40, 40, 12, 40, 40, 40, 14, 40, 40, 23, - 40, 40, 40, 46, 47, 0, 0, 0, 0, 40, - 40, 40, 31, 40, 40, 40, 40, 40, 40, 40, + 40, 40, 40, 28, 40, 40, 40, 40, 27, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, - 9, 30, 40, 7, 19, 5, 26, 18, 40, 40, - 15, 40, 0, 0, 40, 40, 40, 40, 40, 38, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, - 40, 22, 40, 40, 40, 40, 0, 40, 32, 40, - 40, 40, 40, 40, 40, 40, 40, 40, 5, 40, - 40, 24, 40, 40, 32, 32, 25, 40, 40, 40, - 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, - 40, 40, 40, 40, 28, 40, 40, 40, 27, 40, 40, 40, 40, 40, 40, 40, 40, 0 } ; @@ -679,429 +683,459 @@ static const YY_CHAR yy_ec[256] = 5, 8, 9, 5, 10, 11, 12, 13, 14, 15, 16, 17, 14, 18, 14, 19, 19, 20, 5, 5, 5, 5, 5, 21, 22, 23, 24, 25, 26, 27, - 28, 28, 29, 28, 28, 30, 31, 32, 28, 33, - 28, 28, 34, 35, 36, 37, 28, 38, 28, 28, - 5, 39, 5, 5, 40, 5, 41, 42, 43, 44, - - 45, 46, 47, 48, 49, 28, 50, 51, 52, 53, - 54, 55, 56, 57, 58, 59, 60, 61, 28, 38, - 62, 63, 64, 5, 5, 5, 1, 65, 65, 65, - 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, - 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, - 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, - 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, - 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, - 65, 65, 65, 65, 65, 65, 65, 65, 65, 65, - 65, 66, 66, 66, 66, 66, 66, 66, 66, 66, - - 66, 66, 66, 66, 66, 66, 66, 66, 66, 66, - 66, 66, 66, 66, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 67, 67, 67, 67, 67, 67, 67, - 67, 67, 67, 67, 67, 67, 67, 67, 67, 68, - 68, 68, 68, 68, 68, 68, 68, 1, 1, 1, + 28, 28, 29, 28, 28, 30, 31, 32, 33, 34, + 35, 28, 36, 37, 38, 39, 28, 40, 28, 28, + 5, 41, 5, 5, 42, 5, 43, 44, 45, 46, + + 47, 48, 49, 50, 51, 28, 52, 53, 54, 55, + 56, 57, 58, 59, 60, 61, 62, 63, 28, 40, + 64, 65, 66, 5, 5, 5, 1, 67, 67, 67, + 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, + 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, + 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, + 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, + 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, + 67, 67, 67, 67, 67, 67, 67, 67, 67, 67, + 67, 68, 68, 68, 68, 68, 68, 68, 68, 68, + + 68, 68, 68, 68, 68, 68, 68, 68, 68, 68, + 68, 68, 68, 68, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 69, 69, 69, 69, 69, 69, 69, + 69, 69, 69, 69, 69, 69, 69, 69, 69, 70, + 70, 70, 70, 70, 70, 70, 70, 1, 1, 1, 1, 1, 1, 1, 1 } ; -static const YY_CHAR yy_meta[69] = +static const YY_CHAR yy_meta[71] = { 0, 1, 1, 2, 1, 1, 1, 3, 4, 5, 5, 6, 7, 8, 8, 8, 8, 8, 8, 8, 1, 5, 9, 9, 9, 9, 10, 9, 11, 12, 13, - 11, 11, 11, 13, 11, 11, 11, 11, 11, 11, - 9, 9, 9, 9, 10, 9, 11, 11, 11, 11, - 13, 11, 11, 11, 11, 11, 11, 13, 11, 11, - 11, 11, 11, 14, 1, 11, 11, 11 + 11, 11, 11, 11, 11, 13, 11, 11, 11, 11, + 11, 11, 9, 9, 9, 9, 10, 9, 11, 11, + 11, 11, 13, 11, 11, 11, 11, 11, 11, 13, + 11, 11, 11, 11, 11, 14, 1, 11, 11, 11 } ; -static const flex_int16_t yy_base[447] = +static const flex_int16_t yy_base[487] = { 0, - 0, 0, 325, 321, 264, 255, 318, 2458, 67, 2458, - 64, 269, 61, 62, 95, 77, 136, 259, 51, 61, - 188, 97, 118, 150, 65, 195, 233, 153, 183, 222, - 241, 202, 207, 213, 238, 246, 243, 257, 268, 264, - 298, 276, 221, 219, 218, 270, 236, 0, 2458, 79, - 87, 2458, 244, 238, 344, 0, 206, 358, 190, 0, - 2458, 370, 2458, 2458, 0, 342, 377, 177, 175, 174, - 200, 2458, 54, 377, 0, 254, 397, 171, 170, 169, - 358, 85, 404, 373, 376, 398, 391, 406, 412, 417, - 421, 428, 432, 451, 454, 458, 488, 484, 447, 468, - - 494, 471, 501, 506, 510, 525, 532, 536, 541, 544, - 557, 562, 576, 579, 567, 583, 593, 600, 588, 618, - 613, 626, 630, 168, 167, 222, 217, 2458, 0, 2458, - 221, 688, 219, 694, 701, 179, 719, 740, 0, 709, - 683, 774, 159, 158, 135, 130, 128, 710, 125, 123, - 721, 730, 733, 744, 756, 768, 751, 775, 764, 786, - 789, 781, 800, 811, 806, 819, 831, 826, 842, 822, - 852, 845, 856, 864, 878, 875, 889, 896, 900, 908, - 911, 941, 922, 926, 957, 931, 944, 962, 966, 975, - 978, 988, 982, 1012, 999, 1019, 118, 2458, 1056, 0, - - 2458, 50, 1024, 1080, 117, 115, 112, 110, 108, 104, - 1030, 72, 2458, 103, 1038, 1043, 1061, 1064, 1076, 1068, - 1081, 1084, 1094, 1087, 1107, 1118, 1124, 1127, 1131, 1139, - 1161, 1165, 1169, 1173, 1148, 1178, 1184, 1181, 1203, 1214, - 1191, 1195, 1221, 1238, 1225, 1228, 1258, 1234, 1247, 1251, - 1264, 1269, 1273, 1288, 1299, 1303, 1306, 1295, 1311, 1318, - 160, 154, 2458, 107, 1321, 1386, 93, 91, 77, 73, - 72, 70, 1336, 1366, 1360, 1369, 1372, 1379, 1376, 1410, - 1413, 1416, 1419, 1422, 1427, 1432, 1453, 1457, 1463, 1471, - 1475, 1467, 2458, 1478, 1483, 1487, 1520, 1508, 1513, 1529, - - 1523, 1526, 1533, 1538, 1543, 1564, 1559, 1590, 1569, 1573, - 1579, 1583, 1613, 2458, 2458, 85, 65, 59, 36, 1594, - 1620, 1604, 1599, 1609, 1630, 1635, 1645, 1639, 1650, 1669, - 1661, 1655, 1676, 1685, 1681, 1694, 1691, 1724, 1702, 1707, - 1711, 2458, 1715, 1728, 1732, 1737, 2458, 1741, 1748, 1762, - 1745, 1767, 39, 27, 1771, 1780, 1783, 1797, 1792, 1786, - 1816, 1803, 1823, 1827, 1834, 1840, 1847, 1858, 1865, 1873, - 1880, 1870, 1888, 1896, 1883, 1913, 24, 1920, 1926, 1932, - 1938, 1943, 1929, 1952, 1964, 1968, 1976, 1983, 1986, 2002, - 2017, 1999, 2020, 2024, 36, 2006, 2009, 2050, 2054, 2039, - - 2057, 2042, 2062, 2072, 2076, 2087, 2093, 2098, 2102, 2109, - 2112, 2123, 2132, 2153, 2458, 2142, 2149, 2146, 2458, 2135, - 2165, 2179, 2168, 2172, 2188, 2203, 2209, 2458, 2277, 2291, - 2305, 2319, 2328, 2337, 2346, 2359, 2373, 2386, 2400, 2410, - 2416, 2424, 2426, 2432, 2438, 2444 + 0, 0, 345, 344, 284, 283, 340, 2654, 69, 2654, + 66, 295, 63, 64, 99, 80, 142, 283, 62, 67, + 196, 101, 122, 190, 152, 81, 243, 159, 176, 220, + 264, 183, 232, 239, 236, 252, 257, 269, 274, 277, + 307, 213, 250, 230, 229, 279, 274, 0, 2654, 81, + 78, 2654, 282, 267, 315, 0, 222, 355, 202, 0, + 2654, 369, 2654, 2654, 0, 322, 380, 188, 183, 182, + 208, 2654, 56, 143, 0, 339, 396, 180, 176, 175, + 364, 92, 411, 374, 377, 382, 398, 408, 412, 428, + 386, 433, 438, 445, 468, 458, 464, 422, 481, 484, + + 498, 494, 514, 503, 517, 528, 524, 533, 547, 536, + 554, 558, 572, 569, 577, 580, 592, 602, 588, 612, + 622, 632, 625, 628, 173, 172, 230, 223, 2654, 0, + 2654, 227, 692, 223, 698, 705, 177, 723, 746, 0, + 713, 687, 780, 151, 141, 140, 134, 132, 457, 131, + 130, 728, 709, 733, 748, 763, 766, 754, 771, 779, + 788, 784, 796, 814, 810, 820, 827, 830, 804, 850, + 844, 840, 860, 835, 870, 865, 874, 880, 885, 915, + 904, 919, 895, 928, 937, 951, 967, 959, 941, 911, + 962, 975, 985, 997, 992, 1000, 1007, 1023, 127, 2654, + + 730, 0, 2654, 90, 1011, 1069, 121, 118, 116, 114, + 109, 108, 965, 74, 2654, 107, 1037, 1048, 1030, 1053, + 1070, 1056, 1073, 1060, 1086, 1078, 1093, 1096, 1109, 1104, + 1112, 1126, 1135, 1142, 1150, 1147, 1160, 1167, 1172, 1182, + 1202, 1190, 1186, 1193, 1205, 1208, 1216, 1238, 1212, 1223, + 1242, 1227, 1249, 1253, 1258, 1264, 1275, 1283, 1307, 1288, + 1295, 1300, 1319, 160, 157, 2654, 107, 1326, 1377, 84, + 77, 76, 73, 72, 67, 1331, 1345, 1339, 1352, 1362, + 1365, 1357, 1369, 1399, 1405, 1382, 1408, 1413, 1418, 1424, + 1431, 1444, 1448, 1454, 1461, 1457, 2654, 1468, 1474, 1465, + + 1491, 1498, 1504, 1513, 1507, 1516, 1521, 1529, 1539, 1551, + 1546, 1562, 1555, 1565, 1571, 1576, 1585, 2654, 2654, 82, + 61, 55, 53, 1607, 1597, 1601, 1610, 1619, 1622, 1633, + 1643, 1627, 1640, 1657, 1664, 1674, 1677, 1680, 1683, 1687, + 1713, 1690, 1720, 1724, 1727, 1730, 2654, 1743, 1735, 1739, + 1760, 2654, 1746, 1776, 1765, 1750, 1783, 45, 38, 1786, + 1790, 1798, 1806, 1820, 1780, 1816, 1823, 1828, 1832, 1836, + 1862, 1842, 1867, 1872, 1875, 1853, 1883, 1879, 1897, 1905, + 1913, 1916, 27, 1920, 1928, 1909, 1939, 1961, 1946, 1951, + 1958, 1964, 1977, 1954, 1969, 1994, 1999, 2002, 2007, 2010, + + 2014, 39, 2017, 2020, 2050, 2053, 2057, 2060, 2065, 2075, + 2070, 2091, 2096, 2107, 2083, 2113, 2116, 2127, 2121, 2137, + 2151, 2157, 2160, 2654, 2163, 2167, 2177, 2172, 2654, 2181, + 2198, 2101, 2207, 2202, 2214, 2218, 2221, 2224, 2228, 2254, + 2244, 2269, 2261, 2237, 2274, 2277, 2285, 2292, 2307, 2311, + 2300, 2315, 2322, 2331, 2337, 2326, 2346, 2357, 2367, 2362, + 2352, 2387, 2382, 2392, 2378, 2397, 2403, 2654, 2473, 2487, + 2501, 2515, 2524, 2533, 2542, 2555, 2569, 2582, 2596, 2606, + 2612, 2620, 2622, 2628, 2634, 2640 } ; -static const flex_int16_t yy_def[447] = +static const flex_int16_t yy_def[487] = { 0, - 428, 1, 429, 429, 430, 430, 428, 428, 428, 428, - 431, 432, 428, 433, 428, 434, 428, 17, 435, 435, - 435, 435, 435, 435, 435, 428, 435, 435, 435, 435, - 21, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 428, 428, 428, 436, 436, 437, 428, 428, - 431, 428, 431, 428, 438, 15, 17, 428, 428, 15, - 428, 428, 428, 428, 439, 440, 428, 428, 428, 428, - 17, 428, 428, 428, 441, 435, 428, 428, 428, 428, - 435, 21, 21, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 428, 428, 436, 436, 428, 437, 428, - 428, 428, 442, 428, 428, 428, 428, 428, 439, 440, - 443, 428, 428, 428, 428, 428, 428, 444, 428, 428, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 428, 428, 428, 445, - - 428, 428, 446, 428, 428, 428, 428, 428, 428, 428, - 444, 428, 428, 428, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 428, 428, 428, 428, 446, 428, 428, 428, 428, 428, - 428, 428, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 428, 435, 435, 435, 435, 435, 435, 435, - - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 428, 428, 428, 428, 428, 428, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 428, 435, 435, 435, 435, 428, 435, 435, 435, - 435, 435, 428, 428, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 428, 435, 435, 435, - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 428, 435, 435, 435, 435, 435, - - 435, 435, 435, 435, 435, 435, 435, 435, 435, 435, - 435, 435, 435, 435, 428, 435, 435, 435, 428, 435, - 435, 435, 435, 435, 435, 435, 435, 0, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428 + 468, 1, 469, 469, 470, 470, 468, 468, 468, 468, + 471, 472, 468, 473, 468, 474, 468, 17, 475, 475, + 475, 475, 475, 475, 475, 468, 475, 475, 475, 475, + 21, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 468, 468, 468, 476, 476, 477, 468, 468, + 471, 468, 471, 468, 478, 15, 17, 468, 468, 15, + 468, 468, 468, 468, 479, 480, 468, 468, 468, 468, + 17, 468, 468, 468, 481, 475, 468, 468, 468, 468, + 475, 21, 21, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 468, 468, 476, 476, 468, 477, + 468, 468, 468, 482, 468, 468, 468, 468, 468, 479, + 480, 483, 468, 468, 468, 468, 468, 468, 484, 468, + 468, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 468, 468, + + 468, 485, 468, 468, 486, 468, 468, 468, 468, 468, + 468, 468, 484, 468, 468, 468, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 468, 468, 468, 468, 486, 468, 468, + 468, 468, 468, 468, 468, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 468, 475, 475, 475, + + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 468, 468, 468, + 468, 468, 468, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 468, 475, 475, 475, + 475, 468, 475, 475, 475, 475, 475, 468, 468, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 468, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + + 475, 468, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 468, 475, 475, 475, 475, 468, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 475, 475, 475, + 475, 475, 475, 475, 475, 475, 475, 0, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468 } ; -static const flex_int16_t yy_nxt[2527] = +static const flex_int16_t yy_nxt[2725] = { 0, 8, 9, 10, 9, 8, 11, 12, 8, 13, 14, 15, 16, 17, 18, 18, 18, 18, 18, 18, 8, 8, 19, 19, 19, 20, 21, 22, 19, 23, 19, - 19, 24, 19, 19, 19, 25, 19, 19, 26, 27, - 19, 28, 29, 30, 31, 32, 33, 19, 34, 19, - 35, 19, 36, 37, 19, 19, 38, 39, 40, 41, - 42, 19, 19, 8, 8, 43, 44, 45, 50, 52, - 50, 56, 56, 57, 57, 57, 57, 57, 57, 57, - 50, 263, 50, 72, 64, 395, 58, 58, 65, 77, - 59, 265, 52, 76, 76, 263, 90, 377, 264, 77, - - 203, 213, 53, 77, 72, 58, 58, 60, 60, 60, - 60, 60, 60, 60, 81, 67, 78, 79, 80, 61, - 62, 63, 213, 354, 61, 53, 78, 79, 80, 265, - 78, 79, 80, 353, 140, 77, 319, 203, 61, 62, - 63, 318, 68, 69, 70, 61, 56, 84, 71, 71, - 71, 71, 71, 71, 71, 317, 77, 265, 72, 316, - 315, 58, 78, 79, 80, 73, 314, 76, 66, 72, - 85, 74, 272, 75, 140, 86, 271, 72, 87, 270, - 58, 203, 91, 78, 79, 80, 73, 214, 77, 76, - 88, 77, 210, 72, 66, 74, 82, 82, 89, 209, - - 83, 83, 83, 83, 83, 83, 83, 91, 91, 91, - 91, 91, 91, 91, 99, 78, 79, 80, 78, 79, - 80, 77, 208, 140, 202, 198, 77, 198, 428, 127, - 100, 197, 91, 150, 149, 76, 101, 428, 147, 146, - 77, 66, 136, 428, 130, 77, 428, 128, 78, 79, - 80, 77, 106, 78, 79, 80, 92, 93, 94, 95, - 77, 96, 102, 107, 97, 108, 98, 78, 79, 80, - 103, 77, 78, 79, 80, 104, 77, 127, 78, 79, - 80, 77, 125, 124, 77, 91, 110, 78, 79, 80, - 111, 109, 77, 105, 112, 77, 428, 113, 78, 79, - - 80, 114, 77, 78, 79, 80, 77, 55, 78, 79, - 80, 78, 79, 80, 77, 115, 123, 428, 49, 78, - 79, 80, 78, 79, 80, 117, 116, 49, 47, 78, - 79, 80, 47, 78, 79, 80, 77, 428, 428, 118, - 428, 78, 79, 80, 428, 428, 119, 428, 120, 428, - 121, 428, 428, 141, 428, 122, 132, 132, 132, 132, - 132, 132, 428, 78, 79, 80, 134, 134, 428, 428, - 135, 135, 135, 135, 135, 135, 135, 428, 137, 137, - 142, 133, 138, 138, 138, 138, 138, 138, 138, 66, - 66, 66, 66, 66, 66, 66, 77, 428, 428, 72, - - 76, 76, 76, 76, 76, 428, 73, 143, 144, 145, - 72, 77, 76, 76, 77, 428, 76, 151, 72, 428, - 157, 155, 428, 78, 79, 80, 154, 73, 152, 77, - 153, 428, 156, 152, 72, 76, 77, 158, 78, 79, - 80, 78, 79, 80, 77, 428, 159, 152, 428, 153, - 77, 428, 428, 428, 152, 77, 78, 79, 80, 77, - 76, 428, 157, 78, 79, 80, 77, 428, 428, 428, - 77, 78, 79, 80, 428, 160, 162, 78, 79, 80, - 428, 161, 78, 79, 80, 77, 78, 79, 80, 77, - 428, 428, 77, 78, 79, 80, 77, 78, 79, 80, - - 428, 428, 164, 163, 165, 173, 77, 166, 174, 77, - 428, 168, 78, 79, 80, 167, 78, 79, 80, 78, - 79, 80, 77, 78, 79, 80, 77, 428, 428, 176, - 428, 170, 77, 78, 79, 80, 78, 79, 80, 77, - 428, 169, 171, 172, 77, 175, 428, 428, 77, 78, - 79, 80, 177, 78, 79, 80, 428, 428, 428, 78, - 79, 80, 428, 77, 428, 178, 78, 79, 80, 179, - 77, 78, 79, 80, 77, 78, 79, 80, 180, 77, - 428, 428, 77, 428, 428, 181, 428, 428, 428, 428, - 78, 79, 80, 183, 182, 77, 184, 78, 79, 80, - - 77, 78, 79, 80, 428, 77, 78, 79, 80, 78, - 79, 80, 157, 428, 77, 185, 186, 77, 428, 187, - 188, 77, 78, 79, 80, 428, 77, 78, 79, 80, - 428, 77, 78, 79, 80, 428, 428, 428, 77, 189, - 192, 78, 79, 80, 78, 79, 80, 190, 78, 79, - 80, 77, 428, 78, 79, 80, 77, 428, 78, 79, - 80, 191, 428, 194, 77, 78, 79, 80, 77, 428, - 428, 193, 428, 195, 428, 428, 428, 428, 78, 79, - 80, 428, 428, 78, 79, 80, 196, 428, 428, 428, - 428, 78, 79, 80, 198, 78, 79, 80, 428, 428, - - 199, 199, 199, 199, 199, 199, 135, 135, 135, 135, - 135, 135, 135, 135, 135, 135, 135, 135, 135, 135, - 141, 204, 428, 428, 428, 201, 428, 63, 428, 428, - 201, 138, 138, 138, 138, 138, 138, 138, 428, 212, - 428, 428, 428, 213, 201, 428, 63, 142, 205, 206, - 207, 201, 138, 138, 138, 138, 138, 138, 138, 77, - 212, 428, 215, 428, 61, 428, 63, 213, 77, 61, - 428, 77, 428, 428, 143, 144, 145, 140, 140, 140, - 140, 140, 77, 61, 216, 63, 78, 79, 80, 77, - 61, 219, 221, 140, 77, 78, 79, 80, 78, 79, - - 80, 217, 77, 428, 218, 428, 77, 428, 428, 78, - 79, 80, 140, 77, 428, 428, 78, 79, 80, 77, - 220, 78, 79, 80, 77, 428, 224, 77, 428, 78, - 79, 80, 223, 78, 79, 80, 428, 140, 77, 428, - 78, 79, 80, 225, 77, 222, 78, 79, 80, 77, - 227, 78, 79, 80, 78, 79, 80, 77, 230, 428, - 77, 226, 229, 428, 77, 78, 79, 80, 231, 77, - 428, 78, 79, 80, 428, 228, 78, 79, 80, 428, - 77, 232, 428, 77, 78, 79, 80, 78, 79, 80, - 77, 78, 79, 80, 77, 428, 78, 79, 80, 234, - - 235, 428, 77, 428, 428, 233, 428, 78, 79, 80, - 78, 79, 80, 77, 428, 238, 77, 78, 79, 80, - 236, 78, 79, 80, 428, 428, 428, 77, 428, 78, - 79, 80, 237, 239, 77, 428, 428, 240, 77, 428, - 78, 79, 80, 78, 79, 80, 77, 428, 242, 77, - 428, 241, 428, 428, 78, 79, 80, 428, 244, 428, - 77, 78, 79, 80, 77, 78, 79, 80, 246, 77, - 243, 220, 428, 78, 79, 80, 78, 79, 80, 77, - 247, 428, 77, 428, 428, 245, 249, 78, 79, 80, - 428, 78, 79, 80, 250, 77, 78, 79, 80, 248, - - 77, 428, 428, 428, 77, 428, 78, 79, 80, 78, - 79, 80, 428, 77, 252, 428, 77, 428, 251, 254, - 77, 428, 78, 79, 80, 253, 77, 78, 79, 80, - 428, 78, 79, 80, 257, 141, 255, 77, 428, 428, - 78, 79, 80, 78, 79, 80, 256, 78, 79, 80, - 77, 428, 259, 78, 79, 80, 428, 77, 428, 212, - 258, 428, 266, 213, 78, 79, 80, 260, 261, 261, - 261, 261, 261, 261, 428, 428, 77, 78, 79, 80, - 212, 77, 428, 428, 78, 79, 80, 213, 273, 267, - 268, 269, 203, 203, 203, 203, 203, 203, 203, 77, - - 276, 274, 77, 78, 79, 80, 77, 428, 78, 79, - 80, 277, 428, 428, 77, 428, 275, 428, 428, 77, - 428, 428, 77, 428, 428, 77, 78, 79, 80, 78, - 79, 80, 77, 78, 79, 80, 278, 280, 279, 428, - 428, 78, 79, 80, 428, 77, 78, 79, 80, 78, - 79, 80, 78, 79, 80, 281, 77, 428, 428, 78, - 79, 80, 77, 428, 428, 77, 428, 428, 282, 77, - 428, 428, 78, 79, 80, 286, 283, 77, 285, 428, - 428, 428, 284, 78, 79, 80, 77, 428, 428, 78, - 79, 80, 78, 79, 80, 287, 78, 79, 80, 77, - - 293, 428, 428, 77, 78, 79, 80, 77, 428, 288, - 289, 77, 428, 78, 79, 80, 77, 291, 428, 77, - 428, 428, 77, 428, 428, 290, 78, 79, 80, 77, - 78, 79, 80, 77, 78, 79, 80, 292, 78, 79, - 80, 77, 428, 78, 79, 80, 78, 79, 80, 78, - 79, 80, 77, 296, 298, 294, 78, 79, 80, 77, - 78, 79, 80, 77, 295, 428, 77, 428, 78, 79, - 80, 299, 77, 428, 428, 297, 77, 301, 428, 78, - 79, 80, 300, 428, 428, 77, 78, 79, 80, 77, - 78, 79, 80, 78, 79, 80, 77, 428, 428, 78, - - 79, 80, 77, 78, 79, 80, 302, 77, 428, 428, - 428, 77, 78, 79, 80, 305, 78, 79, 80, 428, - 308, 304, 303, 78, 79, 80, 77, 428, 428, 78, - 79, 80, 141, 77, 78, 79, 80, 77, 78, 79, - 80, 77, 428, 307, 77, 306, 311, 309, 428, 77, - 428, 428, 310, 78, 79, 80, 77, 428, 313, 266, - 78, 79, 80, 428, 78, 79, 80, 312, 78, 79, - 80, 78, 79, 80, 77, 428, 78, 79, 80, 428, - 320, 428, 428, 78, 79, 80, 267, 268, 269, 265, - 265, 265, 265, 265, 321, 323, 428, 428, 77, 428, - - 324, 78, 79, 80, 77, 265, 428, 77, 322, 428, - 77, 428, 428, 428, 77, 428, 428, 77, 326, 428, - 428, 428, 428, 428, 265, 78, 79, 80, 325, 428, - 428, 78, 79, 80, 78, 79, 80, 78, 79, 80, - 428, 78, 79, 80, 78, 79, 80, 428, 77, 265, - 327, 77, 329, 328, 77, 428, 428, 77, 428, 428, - 77, 428, 428, 330, 331, 77, 428, 332, 428, 428, - 77, 428, 428, 428, 428, 78, 79, 80, 78, 79, - 80, 78, 79, 80, 78, 79, 80, 78, 79, 80, - 333, 77, 78, 79, 80, 77, 428, 78, 79, 80, - - 428, 77, 428, 428, 428, 77, 334, 335, 336, 77, - 428, 337, 428, 77, 428, 428, 77, 428, 78, 79, - 80, 77, 78, 79, 80, 77, 339, 341, 78, 79, - 80, 338, 78, 79, 80, 340, 78, 79, 80, 342, - 78, 79, 80, 78, 79, 80, 77, 428, 78, 79, - 80, 77, 78, 79, 80, 323, 428, 343, 77, 428, - 428, 77, 428, 428, 77, 428, 428, 77, 323, 428, - 344, 77, 428, 78, 79, 80, 77, 428, 78, 79, - 80, 77, 428, 347, 345, 78, 79, 80, 78, 79, - 80, 78, 79, 80, 78, 79, 80, 77, 78, 79, - - 80, 346, 77, 78, 79, 80, 348, 77, 78, 79, - 80, 77, 428, 428, 428, 349, 428, 77, 428, 428, - 428, 77, 355, 428, 78, 79, 80, 350, 77, 78, - 79, 80, 77, 428, 78, 79, 80, 77, 78, 79, - 80, 351, 77, 358, 78, 79, 80, 77, 78, 79, - 80, 77, 428, 428, 352, 78, 79, 80, 77, 78, - 79, 80, 357, 359, 78, 79, 80, 428, 77, 78, - 79, 80, 356, 77, 78, 79, 80, 77, 78, 79, - 80, 428, 428, 77, 428, 78, 79, 80, 77, 428, - 363, 362, 360, 77, 428, 78, 79, 80, 428, 77, - - 78, 79, 80, 361, 78, 79, 80, 77, 364, 428, - 78, 79, 80, 360, 77, 78, 79, 80, 365, 77, - 78, 79, 80, 77, 428, 360, 78, 79, 80, 77, - 428, 360, 77, 428, 78, 79, 80, 368, 428, 366, - 77, 78, 79, 80, 367, 77, 78, 79, 80, 77, - 78, 79, 80, 77, 370, 371, 78, 79, 80, 78, - 79, 80, 77, 428, 428, 369, 77, 78, 79, 80, - 77, 372, 78, 79, 80, 77, 78, 79, 80, 77, - 78, 79, 80, 77, 428, 428, 77, 428, 428, 78, - 79, 80, 374, 78, 79, 80, 373, 78, 79, 80, - - 77, 428, 78, 79, 80, 77, 78, 79, 80, 77, - 78, 79, 80, 78, 79, 80, 428, 376, 77, 428, - 375, 77, 380, 378, 77, 217, 428, 78, 79, 80, - 77, 428, 78, 79, 80, 77, 78, 79, 80, 428, - 381, 77, 428, 428, 379, 78, 79, 80, 78, 79, - 80, 78, 79, 80, 77, 383, 428, 78, 79, 80, - 382, 77, 78, 79, 80, 77, 428, 428, 78, 79, - 80, 385, 77, 384, 428, 428, 428, 386, 77, 428, - 428, 78, 79, 80, 387, 77, 428, 428, 78, 79, - 80, 360, 78, 79, 80, 428, 77, 428, 428, 78, - - 79, 80, 360, 77, 428, 78, 79, 80, 77, 428, - 428, 77, 78, 79, 80, 388, 389, 428, 77, 428, - 428, 77, 428, 78, 79, 80, 77, 393, 428, 428, - 78, 79, 80, 390, 77, 78, 79, 80, 78, 79, - 80, 428, 428, 428, 391, 78, 79, 80, 78, 79, - 80, 77, 392, 78, 79, 80, 397, 394, 77, 428, - 428, 78, 79, 80, 77, 396, 428, 77, 428, 428, - 77, 217, 399, 400, 428, 428, 77, 428, 78, 79, - 80, 77, 428, 428, 428, 78, 79, 80, 428, 428, + 19, 24, 19, 19, 19, 19, 19, 25, 19, 19, + 26, 27, 19, 28, 29, 30, 31, 32, 33, 19, + 34, 19, 35, 19, 36, 37, 19, 19, 38, 39, + 40, 41, 42, 19, 19, 8, 8, 43, 44, 45, + 50, 52, 50, 56, 56, 57, 57, 57, 57, 57, + 57, 57, 50, 52, 50, 72, 266, 64, 58, 58, + 402, 65, 59, 91, 91, 91, 91, 91, 91, 91, + + 76, 76, 77, 215, 268, 383, 53, 77, 72, 58, + 58, 60, 60, 60, 60, 60, 60, 60, 53, 205, + 67, 359, 81, 61, 62, 63, 215, 268, 61, 78, + 79, 80, 358, 141, 78, 79, 80, 266, 323, 205, + 267, 77, 322, 321, 61, 62, 63, 68, 69, 70, + 268, 61, 56, 84, 71, 71, 71, 71, 71, 71, + 71, 320, 77, 319, 72, 72, 318, 58, 78, 79, + 80, 73, 73, 76, 66, 275, 85, 72, 72, 74, + 141, 75, 274, 90, 273, 72, 72, 205, 58, 78, + 79, 80, 77, 91, 73, 73, 216, 76, 212, 77, + + 66, 72, 72, 74, 82, 82, 211, 210, 83, 83, + 83, 83, 83, 83, 83, 86, 77, 141, 87, 78, + 79, 80, 100, 77, 204, 101, 78, 79, 80, 200, + 77, 102, 88, 200, 468, 107, 77, 128, 199, 91, + 89, 151, 150, 78, 79, 80, 76, 468, 148, 147, + 78, 79, 80, 77, 66, 124, 137, 78, 79, 80, + 77, 468, 103, 78, 79, 80, 92, 93, 94, 95, + 104, 96, 77, 131, 97, 105, 77, 98, 99, 77, + 78, 79, 80, 77, 468, 129, 128, 78, 79, 80, + 108, 110, 77, 109, 111, 126, 125, 77, 112, 78, + + 79, 80, 113, 78, 79, 80, 78, 79, 80, 77, + 78, 79, 80, 114, 77, 115, 91, 77, 106, 78, + 79, 80, 468, 116, 78, 79, 80, 133, 133, 133, + 133, 133, 133, 142, 117, 55, 78, 79, 80, 468, + 118, 78, 79, 80, 78, 79, 80, 77, 49, 49, + 119, 47, 47, 468, 134, 468, 468, 120, 468, 121, + 468, 122, 143, 135, 135, 468, 123, 136, 136, 136, + 136, 136, 136, 136, 78, 79, 80, 138, 138, 77, + 468, 139, 139, 139, 139, 139, 139, 139, 468, 144, + 145, 146, 66, 66, 66, 66, 66, 66, 66, 76, + + 76, 76, 76, 76, 77, 468, 78, 79, 80, 468, + 468, 468, 468, 468, 77, 76, 468, 77, 157, 76, + 76, 468, 77, 468, 156, 152, 77, 158, 468, 155, + 468, 78, 79, 80, 468, 153, 76, 154, 77, 159, + 153, 78, 79, 80, 78, 79, 80, 468, 77, 78, + 79, 80, 77, 78, 79, 80, 153, 160, 154, 468, + 468, 76, 77, 153, 158, 78, 79, 80, 77, 468, + 468, 468, 468, 77, 468, 78, 79, 80, 77, 78, + 79, 80, 161, 171, 163, 77, 214, 169, 162, 78, + 79, 80, 215, 468, 468, 78, 79, 80, 77, 164, + + 78, 79, 80, 468, 77, 78, 79, 80, 77, 214, + 468, 468, 78, 79, 80, 468, 215, 168, 165, 170, + 166, 77, 468, 167, 77, 78, 79, 80, 468, 468, + 172, 78, 79, 80, 77, 78, 79, 80, 77, 468, + 176, 173, 174, 77, 175, 468, 468, 177, 78, 79, + 80, 78, 79, 80, 77, 468, 179, 77, 468, 468, + 468, 78, 79, 80, 77, 78, 79, 80, 77, 468, + 78, 79, 80, 77, 178, 468, 77, 468, 180, 182, + 468, 78, 79, 80, 78, 79, 80, 77, 183, 181, + 185, 78, 79, 80, 77, 78, 79, 80, 77, 468, + + 78, 79, 80, 78, 79, 80, 468, 184, 186, 77, + 468, 188, 77, 468, 78, 79, 80, 77, 187, 189, + 77, 78, 79, 80, 158, 78, 79, 80, 77, 468, + 468, 468, 77, 468, 468, 190, 78, 79, 80, 78, + 79, 80, 77, 468, 78, 79, 80, 78, 79, 80, + 191, 193, 77, 468, 468, 78, 79, 80, 192, 78, + 79, 80, 77, 468, 468, 77, 194, 468, 77, 78, + 79, 80, 77, 468, 197, 468, 468, 195, 468, 78, + 79, 80, 468, 468, 196, 468, 198, 468, 468, 78, + 79, 80, 78, 79, 80, 78, 79, 80, 200, 78, + + 79, 80, 468, 468, 201, 201, 201, 201, 201, 201, + 136, 136, 136, 136, 136, 136, 136, 136, 136, 136, + 136, 136, 136, 136, 142, 468, 468, 206, 468, 203, + 468, 63, 468, 468, 203, 139, 139, 139, 139, 139, + 139, 139, 264, 264, 264, 264, 264, 264, 468, 77, + 203, 468, 63, 143, 207, 208, 209, 203, 139, 139, + 139, 139, 139, 139, 139, 468, 468, 468, 77, 468, + 61, 217, 63, 77, 468, 61, 78, 79, 80, 468, + 144, 145, 146, 141, 141, 141, 141, 141, 77, 221, + 218, 61, 468, 63, 77, 78, 79, 80, 61, 141, + + 78, 79, 80, 77, 468, 468, 77, 223, 468, 468, + 219, 77, 468, 220, 468, 78, 79, 80, 222, 77, + 141, 78, 79, 80, 77, 468, 468, 468, 77, 225, + 78, 79, 80, 78, 79, 80, 77, 232, 78, 79, + 80, 468, 468, 226, 77, 141, 78, 79, 80, 224, + 77, 78, 79, 80, 77, 78, 79, 80, 468, 227, + 77, 231, 228, 78, 79, 80, 229, 77, 468, 468, + 77, 78, 79, 80, 468, 77, 233, 78, 79, 80, + 77, 78, 79, 80, 77, 230, 234, 78, 79, 80, + 77, 237, 468, 468, 78, 79, 80, 78, 79, 80, + + 77, 235, 78, 79, 80, 77, 468, 78, 79, 80, + 77, 78, 79, 80, 77, 236, 238, 78, 79, 80, + 77, 468, 241, 239, 468, 77, 468, 78, 79, 80, + 240, 242, 78, 79, 80, 77, 468, 78, 79, 80, + 468, 78, 79, 80, 77, 247, 468, 78, 79, 80, + 468, 77, 78, 79, 80, 77, 246, 244, 243, 77, + 468, 245, 78, 79, 80, 468, 468, 468, 77, 254, + 468, 78, 79, 80, 248, 468, 468, 77, 78, 79, + 80, 77, 78, 79, 80, 249, 78, 79, 80, 468, + 250, 77, 468, 253, 214, 78, 79, 80, 222, 77, + + 215, 468, 77, 468, 78, 79, 80, 77, 78, 79, + 80, 251, 255, 468, 468, 77, 252, 214, 78, 79, + 80, 257, 142, 256, 215, 77, 78, 79, 80, 78, + 79, 80, 77, 468, 78, 79, 80, 77, 468, 468, + 77, 468, 78, 79, 80, 258, 260, 77, 468, 468, + 261, 269, 78, 79, 80, 468, 468, 259, 468, 78, + 79, 80, 262, 77, 78, 79, 80, 78, 79, 80, + 77, 468, 468, 263, 78, 79, 80, 77, 270, 271, + 272, 205, 205, 205, 205, 205, 205, 205, 77, 276, + 78, 79, 80, 77, 279, 468, 77, 78, 79, 80, + + 77, 468, 468, 280, 78, 79, 80, 278, 277, 468, + 77, 468, 468, 77, 281, 78, 79, 80, 77, 468, + 78, 79, 80, 78, 79, 80, 77, 78, 79, 80, + 283, 468, 282, 77, 468, 468, 77, 78, 79, 80, + 78, 79, 80, 284, 77, 78, 79, 80, 285, 77, + 468, 468, 77, 78, 79, 80, 286, 288, 289, 468, + 78, 79, 80, 78, 79, 80, 77, 468, 468, 287, + 468, 78, 79, 80, 468, 77, 78, 79, 80, 78, + 79, 80, 77, 468, 290, 291, 468, 77, 468, 468, + 77, 468, 468, 78, 79, 80, 292, 293, 468, 468, + + 77, 468, 78, 79, 80, 294, 295, 77, 468, 78, + 79, 80, 77, 468, 78, 79, 80, 78, 79, 80, + 468, 297, 77, 468, 468, 468, 77, 78, 79, 80, + 77, 468, 302, 77, 78, 79, 80, 296, 299, 78, + 79, 80, 77, 468, 298, 77, 468, 304, 77, 78, + 79, 80, 77, 78, 79, 80, 77, 78, 79, 80, + 78, 79, 80, 77, 301, 300, 468, 77, 468, 78, + 79, 80, 78, 79, 80, 78, 79, 80, 77, 78, + 79, 80, 77, 78, 79, 80, 303, 305, 306, 77, + 78, 79, 80, 77, 78, 79, 80, 468, 77, 468, + + 468, 468, 468, 468, 77, 78, 79, 80, 309, 78, + 79, 80, 308, 307, 468, 77, 78, 79, 80, 468, + 78, 79, 80, 77, 312, 78, 79, 80, 77, 311, + 468, 78, 79, 80, 310, 77, 314, 142, 468, 468, + 77, 468, 78, 79, 80, 468, 468, 77, 315, 468, + 78, 79, 80, 313, 468, 78, 79, 80, 316, 77, + 468, 317, 78, 79, 80, 468, 269, 78, 79, 80, + 468, 77, 468, 325, 78, 79, 80, 324, 327, 77, + 268, 268, 268, 268, 268, 77, 78, 79, 80, 326, + 328, 468, 77, 270, 271, 272, 268, 77, 78, 79, + + 80, 330, 77, 468, 468, 77, 78, 79, 80, 77, + 468, 331, 78, 79, 80, 468, 329, 268, 468, 78, + 79, 80, 77, 468, 78, 79, 80, 468, 334, 78, + 79, 80, 78, 79, 80, 468, 78, 79, 80, 77, + 468, 332, 268, 333, 468, 77, 468, 468, 77, 78, + 79, 80, 335, 77, 468, 336, 468, 468, 77, 468, + 468, 468, 468, 468, 77, 468, 78, 79, 80, 468, + 468, 77, 78, 79, 80, 78, 79, 80, 337, 338, + 78, 79, 80, 339, 77, 78, 79, 80, 77, 468, + 468, 78, 79, 80, 77, 341, 342, 77, 78, 79, + + 80, 77, 468, 468, 340, 77, 468, 468, 77, 468, + 347, 78, 79, 80, 77, 78, 79, 80, 344, 343, + 346, 78, 79, 80, 78, 79, 80, 345, 78, 79, + 80, 77, 78, 79, 80, 78, 79, 80, 77, 327, + 468, 78, 79, 80, 77, 468, 468, 77, 468, 468, + 348, 468, 468, 77, 327, 468, 77, 468, 78, 79, + 80, 77, 349, 468, 468, 78, 79, 80, 468, 77, + 352, 78, 79, 80, 78, 79, 80, 350, 353, 77, + 78, 79, 80, 78, 79, 80, 77, 468, 78, 79, + 80, 77, 468, 468, 468, 77, 78, 79, 80, 351, + + 468, 468, 77, 354, 468, 77, 78, 79, 80, 468, + 468, 77, 468, 78, 79, 80, 77, 468, 78, 79, + 80, 355, 78, 79, 80, 77, 468, 468, 357, 78, + 79, 80, 78, 79, 80, 360, 356, 77, 78, 79, + 80, 77, 468, 78, 79, 80, 468, 77, 468, 468, + 77, 361, 78, 79, 80, 363, 468, 364, 468, 77, + 468, 362, 77, 468, 78, 79, 80, 77, 78, 79, + 80, 468, 468, 77, 78, 79, 80, 78, 79, 80, + 77, 367, 368, 77, 468, 468, 78, 79, 80, 78, + 79, 80, 365, 468, 78, 79, 80, 77, 468, 468, + + 78, 79, 80, 366, 77, 468, 468, 78, 79, 80, + 78, 79, 80, 369, 77, 365, 468, 77, 468, 468, + 77, 370, 468, 77, 78, 79, 80, 77, 468, 468, + 77, 78, 79, 80, 365, 365, 371, 372, 374, 468, + 468, 78, 79, 80, 78, 79, 80, 78, 79, 80, + 78, 79, 80, 77, 78, 79, 80, 78, 79, 80, + 77, 468, 468, 375, 77, 373, 468, 77, 468, 468, + 77, 468, 468, 468, 468, 77, 468, 377, 376, 77, + 78, 79, 80, 77, 468, 468, 77, 78, 79, 80, 77, 78, 79, 80, 78, 79, 80, 78, 79, 80, - 398, 428, 77, 78, 79, 80, 77, 428, 78, 79, - 80, 401, 428, 403, 77, 428, 428, 78, 79, 80, - 402, 77, 428, 428, 77, 428, 428, 428, 428, 78, - 79, 80, 404, 78, 79, 80, 405, 77, 428, 428, - 77, 78, 79, 80, 77, 428, 428, 77, 78, 79, - 80, 78, 79, 80, 406, 77, 428, 411, 77, 428, - 428, 389, 77, 397, 78, 79, 80, 78, 79, 80, - 428, 78, 79, 80, 78, 79, 80, 77, 360, 428, - 77, 407, 78, 79, 80, 78, 79, 80, 77, 78, - 79, 80, 77, 428, 408, 77, 410, 428, 409, 428, - - 77, 360, 428, 428, 78, 79, 80, 78, 79, 80, - 77, 428, 415, 428, 77, 78, 79, 80, 413, 78, - 79, 80, 78, 79, 80, 77, 360, 78, 79, 80, - 412, 77, 428, 428, 428, 428, 77, 78, 79, 80, - 77, 78, 79, 80, 414, 428, 428, 77, 428, 428, - 77, 428, 78, 79, 80, 360, 428, 428, 78, 79, - 80, 77, 416, 78, 79, 80, 360, 78, 79, 80, - 77, 417, 419, 77, 78, 79, 80, 78, 79, 80, - 77, 418, 422, 428, 77, 360, 420, 77, 78, 79, - 80, 77, 428, 421, 428, 428, 428, 78, 79, 80, - - 78, 79, 80, 77, 428, 428, 77, 78, 79, 80, - 77, 78, 79, 80, 78, 79, 80, 77, 78, 79, - 80, 428, 360, 423, 424, 428, 77, 428, 428, 425, - 78, 79, 80, 78, 79, 80, 426, 78, 79, 80, - 428, 77, 428, 428, 78, 79, 80, 77, 428, 428, - 428, 428, 428, 78, 79, 80, 427, 428, 428, 428, - 428, 360, 428, 428, 428, 428, 428, 428, 78, 79, - 80, 428, 428, 428, 78, 79, 80, 46, 46, 46, - 46, 46, 46, 46, 46, 46, 46, 46, 46, 46, - 46, 48, 48, 48, 48, 48, 48, 48, 48, 48, - - 48, 48, 48, 48, 48, 51, 51, 51, 51, 51, - 51, 51, 51, 51, 51, 51, 51, 51, 51, 54, - 54, 54, 54, 54, 54, 54, 54, 54, 54, 54, - 54, 54, 54, 57, 428, 57, 428, 57, 428, 57, - 66, 428, 428, 66, 428, 66, 66, 66, 66, 66, - 76, 76, 428, 76, 76, 76, 76, 76, 76, 126, - 126, 126, 126, 126, 126, 126, 126, 126, 126, 126, - 126, 126, 126, 129, 129, 129, 129, 129, 129, 129, - 129, 129, 129, 129, 129, 129, 131, 428, 131, 131, - 131, 131, 131, 131, 131, 131, 131, 131, 131, 131, - - 139, 428, 139, 139, 139, 139, 139, 139, 139, 139, - 139, 139, 139, 139, 140, 140, 140, 140, 140, 140, - 140, 140, 140, 148, 148, 148, 200, 428, 428, 428, - 428, 200, 200, 200, 203, 203, 203, 203, 203, 211, - 211, 211, 428, 428, 211, 262, 262, 262, 265, 265, - 265, 265, 265, 265, 265, 265, 265, 7, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428 + 77, 378, 78, 79, 80, 77, 78, 79, 80, 468, + 78, 79, 80, 78, 79, 80, 77, 78, 79, 80, + 77, 379, 380, 77, 468, 381, 77, 78, 79, 80, + 77, 386, 78, 79, 80, 382, 468, 219, 77, 468, + 384, 468, 468, 78, 79, 80, 77, 78, 79, 80, + 78, 79, 80, 78, 79, 80, 77, 78, 79, 80, + 77, 385, 388, 77, 468, 78, 79, 80, 77, 468, + 387, 468, 77, 78, 79, 80, 77, 389, 391, 468, + 390, 392, 77, 78, 79, 80, 468, 78, 79, 80, + 78, 79, 80, 77, 468, 78, 79, 80, 396, 78, + + 79, 80, 77, 78, 79, 80, 394, 77, 393, 78, + 79, 80, 77, 365, 468, 77, 468, 468, 365, 77, + 78, 79, 80, 77, 468, 468, 468, 395, 468, 78, + 79, 80, 468, 404, 78, 79, 80, 77, 397, 78, + 79, 80, 78, 79, 80, 77, 78, 79, 80, 77, + 78, 79, 80, 77, 468, 398, 77, 468, 468, 400, + 77, 468, 401, 399, 78, 79, 80, 403, 77, 468, + 468, 468, 78, 79, 80, 219, 78, 79, 80, 77, + 78, 79, 80, 78, 79, 80, 77, 78, 79, 80, + 406, 77, 407, 468, 77, 78, 79, 80, 77, 468, + + 412, 77, 468, 405, 77, 468, 78, 79, 80, 77, + 468, 410, 408, 78, 79, 80, 409, 77, 78, 79, + 80, 78, 79, 80, 413, 78, 79, 80, 78, 79, + 80, 78, 79, 80, 77, 411, 78, 79, 80, 77, + 468, 468, 77, 468, 78, 79, 80, 77, 396, 468, + 77, 468, 468, 414, 77, 404, 468, 77, 468, 468, + 77, 78, 79, 80, 468, 468, 78, 79, 80, 78, + 79, 80, 468, 415, 78, 79, 80, 78, 79, 80, + 419, 78, 79, 80, 78, 79, 80, 78, 79, 80, + 77, 365, 468, 77, 468, 468, 416, 77, 468, 417, + + 77, 468, 424, 468, 468, 77, 365, 468, 468, 468, + 77, 468, 468, 421, 468, 77, 418, 78, 79, 80, + 78, 79, 80, 77, 78, 79, 80, 78, 79, 80, + 420, 77, 78, 79, 80, 365, 77, 78, 79, 80, + 422, 77, 78, 79, 80, 468, 468, 77, 468, 434, + 78, 79, 80, 77, 468, 468, 77, 468, 78, 79, + 80, 77, 468, 78, 79, 80, 423, 77, 78, 79, + 80, 468, 365, 468, 78, 79, 80, 77, 425, 429, + 78, 79, 80, 78, 79, 80, 365, 426, 78, 79, + 80, 77, 468, 468, 78, 79, 80, 77, 468, 468, + + 77, 427, 468, 77, 78, 79, 80, 77, 428, 430, + 433, 468, 77, 431, 468, 468, 468, 77, 78, 79, + 80, 77, 468, 468, 78, 79, 80, 78, 79, 80, + 78, 79, 80, 365, 78, 79, 80, 432, 77, 78, + 79, 80, 77, 468, 78, 79, 80, 77, 78, 79, + 80, 468, 468, 435, 77, 468, 468, 365, 77, 468, + 436, 77, 468, 468, 77, 78, 79, 80, 77, 78, + 79, 80, 437, 438, 78, 79, 80, 77, 441, 440, + 439, 78, 79, 80, 77, 78, 79, 80, 78, 79, + 80, 78, 79, 80, 77, 78, 79, 80, 445, 443, + + 444, 77, 468, 468, 78, 79, 80, 442, 468, 77, + 468, 78, 79, 80, 77, 365, 468, 77, 468, 468, + 447, 78, 79, 80, 468, 77, 468, 446, 78, 79, + 80, 448, 77, 468, 468, 452, 78, 79, 80, 450, + 77, 78, 79, 80, 78, 79, 80, 77, 468, 468, + 449, 77, 78, 79, 80, 77, 468, 468, 451, 78, + 79, 80, 77, 468, 468, 453, 77, 78, 79, 80, + 454, 77, 468, 457, 78, 79, 80, 77, 78, 79, + 80, 468, 78, 79, 80, 455, 77, 456, 468, 78, + 79, 80, 77, 78, 79, 80, 458, 77, 78, 79, + + 80, 459, 77, 468, 78, 79, 80, 77, 468, 460, + 464, 463, 462, 78, 79, 80, 461, 468, 77, 78, + 79, 80, 77, 468, 78, 79, 80, 77, 466, 78, + 79, 80, 77, 468, 78, 79, 80, 77, 468, 468, + 465, 468, 468, 77, 468, 78, 79, 80, 468, 78, + 79, 80, 468, 468, 78, 79, 80, 467, 468, 78, + 79, 80, 365, 468, 78, 79, 80, 468, 468, 468, + 78, 79, 80, 46, 46, 46, 46, 46, 46, 46, + 46, 46, 46, 46, 46, 46, 46, 48, 48, 48, + 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, + + 48, 51, 51, 51, 51, 51, 51, 51, 51, 51, + 51, 51, 51, 51, 51, 54, 54, 54, 54, 54, + 54, 54, 54, 54, 54, 54, 54, 54, 54, 57, + 468, 57, 468, 57, 468, 57, 66, 468, 468, 66, + 468, 66, 66, 66, 66, 66, 76, 76, 468, 76, + 76, 76, 76, 76, 76, 127, 127, 127, 127, 127, + 127, 127, 127, 127, 127, 127, 127, 127, 127, 130, + 130, 130, 130, 130, 130, 130, 130, 130, 130, 130, + 130, 130, 132, 468, 132, 132, 132, 132, 132, 132, + 132, 132, 132, 132, 132, 132, 140, 468, 140, 140, + + 140, 140, 140, 140, 140, 140, 140, 140, 140, 140, + 141, 141, 141, 141, 141, 141, 141, 141, 141, 149, + 149, 149, 202, 468, 468, 468, 468, 202, 202, 202, + 205, 205, 205, 205, 205, 213, 213, 213, 468, 468, + 213, 265, 265, 265, 268, 268, 268, 268, 268, 268, + 268, 268, 268, 7, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468 } ; -static const flex_int16_t yy_chk[2527] = +static const flex_int16_t yy_chk[2725] = { 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, @@ -1109,278 +1143,300 @@ static const flex_int16_t yy_chk[2527] = 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 1, 1, 9, 11, - 9, 13, 14, 13, 13, 13, 13, 13, 13, 13, - 50, 395, 50, 73, 16, 377, 13, 14, 16, 19, - 14, 354, 51, 82, 82, 202, 25, 353, 202, 20, - - 319, 212, 11, 25, 73, 13, 14, 15, 15, 15, - 15, 15, 15, 15, 20, 16, 19, 19, 19, 15, - 15, 15, 212, 318, 15, 51, 20, 20, 20, 317, - 25, 25, 25, 316, 272, 22, 271, 270, 15, 15, - 15, 269, 16, 16, 16, 15, 17, 22, 17, 17, - 17, 17, 17, 17, 17, 268, 23, 267, 17, 264, - 262, 17, 22, 22, 22, 17, 261, 214, 210, 17, - 23, 17, 209, 17, 208, 24, 207, 17, 24, 206, - 17, 205, 197, 23, 23, 23, 17, 150, 24, 149, - 24, 28, 147, 17, 146, 17, 21, 21, 24, 145, - - 21, 21, 21, 21, 21, 21, 21, 26, 26, 26, - 26, 26, 26, 26, 28, 24, 24, 24, 28, 28, - 28, 29, 144, 143, 136, 133, 21, 131, 127, 126, - 29, 125, 124, 80, 79, 78, 29, 71, 70, 69, - 32, 68, 59, 57, 54, 33, 53, 47, 29, 29, - 29, 34, 32, 21, 21, 21, 27, 27, 27, 27, - 30, 27, 30, 33, 27, 34, 27, 32, 32, 32, - 30, 27, 33, 33, 33, 30, 35, 46, 34, 34, - 34, 37, 45, 44, 36, 43, 36, 30, 30, 30, - 36, 35, 76, 31, 36, 38, 18, 37, 27, 27, - - 27, 38, 40, 35, 35, 35, 39, 12, 37, 37, - 37, 36, 36, 36, 42, 39, 42, 7, 6, 76, - 76, 76, 38, 38, 38, 40, 39, 5, 4, 40, - 40, 40, 3, 39, 39, 39, 41, 0, 0, 41, - 0, 42, 42, 42, 0, 0, 41, 0, 41, 0, - 41, 0, 0, 66, 0, 41, 55, 55, 55, 55, - 55, 55, 0, 41, 41, 41, 58, 58, 0, 0, - 58, 58, 58, 58, 58, 58, 58, 0, 62, 62, - 66, 55, 62, 62, 62, 62, 62, 62, 62, 67, - 67, 67, 67, 67, 67, 67, 81, 0, 0, 74, - - 77, 77, 77, 77, 77, 0, 74, 66, 66, 66, - 74, 84, 83, 83, 85, 0, 77, 81, 74, 0, - 87, 85, 0, 81, 81, 81, 84, 74, 83, 87, - 83, 0, 86, 83, 74, 77, 86, 88, 84, 84, - 84, 85, 85, 85, 88, 0, 90, 83, 0, 83, - 89, 0, 0, 0, 83, 90, 87, 87, 87, 91, - 77, 0, 89, 86, 86, 86, 92, 0, 0, 0, - 93, 88, 88, 88, 0, 92, 93, 89, 89, 89, - 0, 92, 90, 90, 90, 99, 91, 91, 91, 94, - 0, 0, 95, 92, 92, 92, 96, 93, 93, 93, - - 0, 0, 95, 94, 95, 99, 100, 95, 100, 102, - 0, 97, 99, 99, 99, 96, 94, 94, 94, 95, - 95, 95, 98, 96, 96, 96, 97, 0, 0, 102, - 0, 98, 101, 100, 100, 100, 102, 102, 102, 103, - 0, 97, 98, 98, 104, 101, 0, 0, 105, 98, - 98, 98, 103, 97, 97, 97, 0, 0, 0, 101, - 101, 101, 0, 106, 0, 104, 103, 103, 103, 105, - 107, 104, 104, 104, 108, 105, 105, 105, 106, 109, - 0, 0, 110, 0, 0, 107, 0, 0, 0, 0, - 106, 106, 106, 109, 108, 111, 110, 107, 107, 107, - - 112, 108, 108, 108, 0, 115, 109, 109, 109, 110, - 110, 110, 112, 0, 113, 111, 113, 114, 0, 114, - 115, 116, 111, 111, 111, 0, 119, 112, 112, 112, - 0, 117, 115, 115, 115, 0, 0, 0, 118, 116, - 119, 113, 113, 113, 114, 114, 114, 117, 116, 116, - 116, 121, 0, 119, 119, 119, 120, 0, 117, 117, - 117, 118, 0, 121, 122, 118, 118, 118, 123, 0, - 0, 120, 0, 122, 0, 0, 0, 0, 121, 121, - 121, 0, 0, 120, 120, 120, 123, 0, 0, 0, - 0, 122, 122, 122, 132, 123, 123, 123, 0, 0, - - 132, 132, 132, 132, 132, 132, 134, 134, 134, 134, - 134, 134, 134, 135, 135, 135, 135, 135, 135, 135, - 140, 141, 0, 0, 0, 135, 0, 135, 0, 0, - 135, 137, 137, 137, 137, 137, 137, 137, 0, 148, - 0, 0, 0, 148, 135, 0, 135, 140, 141, 141, - 141, 135, 138, 138, 138, 138, 138, 138, 138, 151, - 148, 0, 151, 0, 138, 0, 138, 148, 152, 138, - 0, 153, 0, 0, 140, 140, 140, 142, 142, 142, - 142, 142, 154, 138, 154, 138, 151, 151, 151, 157, - 138, 156, 159, 142, 155, 152, 152, 152, 153, 153, - - 153, 155, 159, 0, 155, 0, 156, 0, 0, 154, - 154, 154, 142, 158, 0, 0, 157, 157, 157, 162, - 158, 155, 155, 155, 160, 0, 162, 161, 0, 159, - 159, 159, 161, 156, 156, 156, 0, 142, 163, 0, - 158, 158, 158, 163, 165, 160, 162, 162, 162, 164, - 165, 160, 160, 160, 161, 161, 161, 166, 168, 0, - 170, 164, 167, 0, 168, 163, 163, 163, 169, 167, - 0, 165, 165, 165, 0, 166, 164, 164, 164, 0, - 169, 170, 0, 172, 166, 166, 166, 170, 170, 170, - 171, 168, 168, 168, 173, 0, 167, 167, 167, 172, - - 173, 0, 174, 0, 0, 171, 0, 169, 169, 169, - 172, 172, 172, 176, 0, 176, 175, 171, 171, 171, - 174, 173, 173, 173, 0, 0, 0, 177, 0, 174, - 174, 174, 175, 177, 178, 0, 0, 178, 179, 0, - 176, 176, 176, 175, 175, 175, 180, 0, 180, 181, - 0, 179, 0, 0, 177, 177, 177, 0, 182, 0, - 183, 178, 178, 178, 184, 179, 179, 179, 183, 186, - 181, 184, 0, 180, 180, 180, 181, 181, 181, 182, - 185, 0, 187, 0, 0, 182, 186, 183, 183, 183, - 0, 184, 184, 184, 187, 185, 186, 186, 186, 185, - - 188, 0, 0, 0, 189, 0, 182, 182, 182, 187, - 187, 187, 0, 190, 189, 0, 191, 0, 188, 190, - 193, 0, 185, 185, 185, 189, 192, 188, 188, 188, - 0, 189, 189, 189, 193, 203, 191, 195, 0, 0, - 190, 190, 190, 191, 191, 191, 192, 193, 193, 193, - 194, 0, 195, 192, 192, 192, 0, 196, 0, 211, - 194, 0, 203, 211, 195, 195, 195, 196, 199, 199, - 199, 199, 199, 199, 0, 0, 215, 194, 194, 194, - 211, 216, 0, 0, 196, 196, 196, 211, 215, 203, - 203, 203, 204, 204, 204, 204, 204, 204, 204, 217, - - 219, 216, 218, 215, 215, 215, 220, 0, 216, 216, - 216, 221, 0, 0, 219, 0, 218, 0, 0, 221, - 0, 0, 222, 0, 0, 224, 217, 217, 217, 218, - 218, 218, 223, 220, 220, 220, 222, 224, 223, 0, - 0, 219, 219, 219, 0, 225, 221, 221, 221, 222, - 222, 222, 224, 224, 224, 225, 226, 0, 0, 223, - 223, 223, 227, 0, 0, 228, 0, 0, 226, 229, - 0, 0, 225, 225, 225, 229, 226, 230, 228, 0, - 0, 0, 227, 226, 226, 226, 235, 0, 0, 227, - 227, 227, 228, 228, 228, 230, 229, 229, 229, 231, - - 238, 0, 0, 232, 230, 230, 230, 233, 0, 231, - 232, 234, 0, 235, 235, 235, 236, 234, 0, 238, - 0, 0, 237, 0, 0, 233, 231, 231, 231, 241, - 232, 232, 232, 242, 233, 233, 233, 237, 234, 234, - 234, 239, 0, 236, 236, 236, 238, 238, 238, 237, - 237, 237, 240, 242, 244, 239, 241, 241, 241, 243, - 242, 242, 242, 245, 240, 0, 246, 0, 239, 239, - 239, 245, 248, 0, 0, 243, 244, 248, 0, 240, - 240, 240, 247, 0, 0, 249, 243, 243, 243, 250, - 245, 245, 245, 246, 246, 246, 247, 0, 0, 248, - - 248, 248, 251, 244, 244, 244, 249, 252, 0, 0, - 0, 253, 249, 249, 249, 253, 250, 250, 250, 0, - 256, 252, 251, 247, 247, 247, 254, 0, 0, 251, - 251, 251, 265, 258, 252, 252, 252, 255, 253, 253, - 253, 256, 0, 255, 257, 254, 258, 256, 0, 259, - 0, 0, 257, 254, 254, 254, 260, 0, 260, 265, - 258, 258, 258, 0, 255, 255, 255, 259, 256, 256, - 256, 257, 257, 257, 273, 0, 259, 259, 259, 0, - 273, 0, 0, 260, 260, 260, 265, 265, 265, 266, - 266, 266, 266, 266, 274, 276, 0, 0, 275, 0, - - 277, 273, 273, 273, 274, 266, 0, 276, 275, 0, - 277, 0, 0, 0, 279, 0, 0, 278, 279, 0, - 0, 0, 0, 0, 266, 275, 275, 275, 278, 0, - 0, 274, 274, 274, 276, 276, 276, 277, 277, 277, - 0, 279, 279, 279, 278, 278, 278, 0, 280, 266, - 280, 281, 282, 281, 282, 0, 0, 283, 0, 0, - 284, 0, 0, 283, 284, 285, 0, 285, 0, 0, - 286, 0, 0, 0, 0, 280, 280, 280, 281, 281, - 281, 282, 282, 282, 283, 283, 283, 284, 284, 284, - 286, 287, 285, 285, 285, 288, 0, 286, 286, 286, - - 0, 289, 0, 0, 0, 292, 287, 288, 289, 290, - 0, 290, 0, 291, 0, 0, 294, 0, 287, 287, - 287, 295, 288, 288, 288, 296, 292, 295, 289, 289, - 289, 291, 292, 292, 292, 294, 290, 290, 290, 297, - 291, 291, 291, 294, 294, 294, 298, 0, 295, 295, - 295, 299, 296, 296, 296, 300, 0, 299, 297, 0, - 0, 301, 0, 0, 302, 0, 0, 300, 301, 0, - 302, 303, 0, 298, 298, 298, 304, 0, 299, 299, - 299, 305, 0, 306, 304, 297, 297, 297, 301, 301, - 301, 302, 302, 302, 300, 300, 300, 307, 303, 303, - - 303, 305, 306, 304, 304, 304, 308, 309, 305, 305, - 305, 310, 0, 0, 0, 309, 0, 311, 0, 0, - 0, 312, 320, 0, 307, 307, 307, 311, 308, 306, - 306, 306, 320, 0, 309, 309, 309, 323, 310, 310, - 310, 312, 322, 324, 311, 311, 311, 324, 312, 312, - 312, 313, 0, 0, 313, 308, 308, 308, 321, 320, - 320, 320, 322, 325, 323, 323, 323, 0, 325, 322, - 322, 322, 321, 326, 324, 324, 324, 328, 313, 313, - 313, 0, 0, 327, 0, 321, 321, 321, 329, 0, - 329, 328, 326, 332, 0, 325, 325, 325, 0, 331, - - 326, 326, 326, 327, 328, 328, 328, 330, 331, 0, - 327, 327, 327, 332, 333, 329, 329, 329, 333, 335, - 332, 332, 332, 334, 0, 330, 331, 331, 331, 337, - 0, 335, 336, 0, 330, 330, 330, 337, 0, 334, - 339, 333, 333, 333, 336, 340, 335, 335, 335, 341, - 334, 334, 334, 343, 339, 340, 337, 337, 337, 336, - 336, 336, 338, 0, 0, 338, 344, 339, 339, 339, - 345, 343, 340, 340, 340, 346, 341, 341, 341, 348, - 343, 343, 343, 351, 0, 0, 349, 0, 0, 338, - 338, 338, 349, 344, 344, 344, 346, 345, 345, 345, - - 350, 0, 346, 346, 346, 352, 348, 348, 348, 355, - 351, 351, 351, 349, 349, 349, 0, 352, 356, 0, - 350, 357, 358, 355, 360, 356, 0, 350, 350, 350, - 359, 0, 352, 352, 352, 358, 355, 355, 355, 0, - 359, 362, 0, 0, 357, 356, 356, 356, 357, 357, - 357, 360, 360, 360, 361, 362, 0, 359, 359, 359, - 361, 363, 358, 358, 358, 364, 0, 0, 362, 362, - 362, 364, 365, 363, 0, 0, 0, 365, 366, 0, - 0, 361, 361, 361, 366, 367, 0, 0, 363, 363, - 363, 367, 364, 364, 364, 0, 368, 0, 0, 365, - - 365, 365, 368, 369, 0, 366, 366, 366, 372, 0, - 0, 370, 367, 367, 367, 369, 370, 0, 371, 0, - 0, 375, 0, 368, 368, 368, 373, 375, 0, 0, - 369, 369, 369, 371, 374, 372, 372, 372, 370, 370, - 370, 0, 0, 0, 373, 371, 371, 371, 375, 375, - 375, 376, 374, 373, 373, 373, 380, 376, 378, 0, - 0, 374, 374, 374, 379, 378, 0, 383, 0, 0, - 380, 379, 382, 383, 0, 0, 381, 0, 376, 376, - 376, 382, 0, 0, 0, 378, 378, 378, 0, 0, - 384, 379, 379, 379, 383, 383, 383, 380, 380, 380, - - 381, 0, 385, 381, 381, 381, 386, 0, 382, 382, - 382, 384, 0, 386, 387, 0, 0, 384, 384, 384, - 385, 388, 0, 0, 389, 0, 0, 0, 0, 385, - 385, 385, 387, 386, 386, 386, 388, 392, 0, 0, - 390, 387, 387, 387, 396, 0, 0, 397, 388, 388, - 388, 389, 389, 389, 390, 391, 0, 402, 393, 0, - 0, 391, 394, 393, 392, 392, 392, 390, 390, 390, - 0, 396, 396, 396, 397, 397, 397, 400, 403, 0, - 402, 394, 391, 391, 391, 393, 393, 393, 398, 394, - 394, 394, 399, 0, 398, 401, 400, 0, 399, 0, - - 403, 401, 0, 0, 400, 400, 400, 402, 402, 402, - 404, 0, 407, 0, 405, 398, 398, 398, 405, 399, - 399, 399, 401, 401, 401, 406, 411, 403, 403, 403, - 404, 407, 0, 0, 0, 0, 408, 404, 404, 404, - 409, 405, 405, 405, 406, 0, 0, 410, 0, 0, - 411, 0, 406, 406, 406, 408, 0, 0, 407, 407, - 407, 412, 409, 408, 408, 408, 410, 409, 409, 409, - 413, 412, 414, 420, 410, 410, 410, 411, 411, 411, - 416, 413, 418, 0, 418, 420, 416, 417, 412, 412, - 412, 414, 0, 417, 0, 0, 0, 413, 413, 413, - - 420, 420, 420, 421, 0, 0, 423, 416, 416, 416, - 424, 418, 418, 418, 417, 417, 417, 422, 414, 414, - 414, 0, 421, 422, 423, 0, 425, 0, 0, 424, - 421, 421, 421, 423, 423, 423, 425, 424, 424, 424, - 0, 426, 0, 0, 422, 422, 422, 427, 0, 0, - 0, 0, 0, 425, 425, 425, 426, 0, 0, 0, - 0, 427, 0, 0, 0, 0, 0, 0, 426, 426, - 426, 0, 0, 0, 427, 427, 427, 429, 429, 429, - 429, 429, 429, 429, 429, 429, 429, 429, 429, 429, - 429, 430, 430, 430, 430, 430, 430, 430, 430, 430, - - 430, 430, 430, 430, 430, 431, 431, 431, 431, 431, - 431, 431, 431, 431, 431, 431, 431, 431, 431, 432, - 432, 432, 432, 432, 432, 432, 432, 432, 432, 432, - 432, 432, 432, 433, 0, 433, 0, 433, 0, 433, - 434, 0, 0, 434, 0, 434, 434, 434, 434, 434, - 435, 435, 0, 435, 435, 435, 435, 435, 435, 436, - 436, 436, 436, 436, 436, 436, 436, 436, 436, 436, - 436, 436, 436, 437, 437, 437, 437, 437, 437, 437, - 437, 437, 437, 437, 437, 437, 438, 0, 438, 438, - 438, 438, 438, 438, 438, 438, 438, 438, 438, 438, - - 439, 0, 439, 439, 439, 439, 439, 439, 439, 439, - 439, 439, 439, 439, 440, 440, 440, 440, 440, 440, - 440, 440, 440, 441, 441, 441, 442, 0, 0, 0, - 0, 442, 442, 442, 443, 443, 443, 443, 443, 444, - 444, 444, 0, 0, 444, 445, 445, 445, 446, 446, - 446, 446, 446, 446, 446, 446, 446, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428, 428, 428, 428, 428, - 428, 428, 428, 428, 428, 428 + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 9, 11, 9, 13, 14, 13, 13, 13, 13, 13, + 13, 13, 50, 51, 50, 73, 402, 16, 13, 14, + 383, 16, 14, 26, 26, 26, 26, 26, 26, 26, + + 82, 82, 19, 214, 359, 358, 11, 20, 73, 13, + 14, 15, 15, 15, 15, 15, 15, 15, 51, 323, + 16, 322, 20, 15, 15, 15, 214, 321, 15, 19, + 19, 19, 320, 275, 20, 20, 20, 204, 274, 273, + 204, 22, 272, 271, 15, 15, 15, 16, 16, 16, + 270, 15, 17, 22, 17, 17, 17, 17, 17, 17, + 17, 267, 23, 265, 17, 74, 264, 17, 22, 22, + 22, 17, 74, 216, 212, 211, 23, 17, 74, 17, + 210, 17, 209, 25, 208, 17, 74, 207, 17, 23, + 23, 23, 25, 199, 17, 74, 151, 150, 148, 28, + + 147, 17, 74, 17, 21, 21, 146, 145, 21, 21, + 21, 21, 21, 21, 21, 24, 29, 144, 24, 25, + 25, 25, 28, 32, 137, 29, 28, 28, 28, 134, + 24, 29, 24, 132, 128, 32, 21, 127, 126, 125, + 24, 80, 79, 29, 29, 29, 78, 71, 70, 69, + 32, 32, 32, 42, 68, 42, 59, 24, 24, 24, + 30, 57, 30, 21, 21, 21, 27, 27, 27, 27, + 30, 27, 33, 54, 27, 30, 35, 27, 27, 34, + 42, 42, 42, 27, 53, 47, 46, 30, 30, 30, + 33, 35, 36, 34, 36, 45, 44, 37, 36, 33, + + 33, 33, 36, 35, 35, 35, 34, 34, 34, 38, + 27, 27, 27, 37, 39, 38, 43, 40, 31, 36, + 36, 36, 18, 39, 37, 37, 37, 55, 55, 55, + 55, 55, 55, 66, 39, 12, 38, 38, 38, 7, + 40, 39, 39, 39, 40, 40, 40, 41, 6, 5, + 41, 4, 3, 0, 55, 0, 0, 41, 0, 41, + 0, 41, 66, 58, 58, 0, 41, 58, 58, 58, + 58, 58, 58, 58, 41, 41, 41, 62, 62, 76, + 0, 62, 62, 62, 62, 62, 62, 62, 0, 66, + 66, 66, 67, 67, 67, 67, 67, 67, 67, 77, + + 77, 77, 77, 77, 81, 0, 76, 76, 76, 0, + 0, 0, 0, 0, 84, 77, 0, 85, 86, 83, + 83, 0, 86, 0, 85, 81, 91, 87, 0, 84, + 0, 81, 81, 81, 0, 83, 77, 83, 87, 88, + 83, 84, 84, 84, 85, 85, 85, 0, 88, 86, + 86, 86, 89, 91, 91, 91, 83, 90, 83, 0, + 0, 77, 98, 83, 89, 87, 87, 87, 90, 0, + 0, 0, 0, 92, 0, 88, 88, 88, 93, 89, + 89, 89, 92, 98, 93, 94, 149, 97, 92, 98, + 98, 98, 149, 0, 0, 90, 90, 90, 96, 94, + + 92, 92, 92, 0, 97, 93, 93, 93, 95, 149, + 0, 0, 94, 94, 94, 0, 149, 96, 95, 97, + 95, 99, 0, 95, 100, 96, 96, 96, 0, 0, + 99, 97, 97, 97, 102, 95, 95, 95, 101, 0, + 101, 99, 99, 104, 100, 0, 0, 102, 99, 99, + 99, 100, 100, 100, 103, 0, 104, 105, 0, 0, + 0, 102, 102, 102, 107, 101, 101, 101, 106, 0, + 104, 104, 104, 108, 103, 0, 110, 0, 105, 107, + 0, 103, 103, 103, 105, 105, 105, 109, 108, 106, + 110, 107, 107, 107, 111, 106, 106, 106, 112, 0, + + 108, 108, 108, 110, 110, 110, 0, 109, 111, 114, + 0, 114, 113, 0, 109, 109, 109, 115, 112, 115, + 116, 111, 111, 111, 113, 112, 112, 112, 119, 0, + 0, 0, 117, 0, 0, 116, 114, 114, 114, 113, + 113, 113, 118, 0, 115, 115, 115, 116, 116, 116, + 117, 119, 120, 0, 0, 119, 119, 119, 118, 117, + 117, 117, 121, 0, 0, 123, 120, 0, 124, 118, + 118, 118, 122, 0, 123, 0, 0, 121, 0, 120, + 120, 120, 0, 0, 122, 0, 124, 0, 0, 121, + 121, 121, 123, 123, 123, 124, 124, 124, 133, 122, + + 122, 122, 0, 0, 133, 133, 133, 133, 133, 133, + 135, 135, 135, 135, 135, 135, 135, 136, 136, 136, + 136, 136, 136, 136, 141, 0, 0, 142, 0, 136, + 0, 136, 0, 0, 136, 138, 138, 138, 138, 138, + 138, 138, 201, 201, 201, 201, 201, 201, 0, 153, + 136, 0, 136, 141, 142, 142, 142, 136, 139, 139, + 139, 139, 139, 139, 139, 0, 0, 0, 152, 0, + 139, 152, 139, 154, 0, 139, 153, 153, 153, 0, + 141, 141, 141, 143, 143, 143, 143, 143, 155, 157, + 155, 139, 0, 139, 158, 152, 152, 152, 139, 143, + + 154, 154, 154, 156, 0, 0, 157, 160, 0, 0, + 156, 159, 0, 156, 0, 155, 155, 155, 159, 160, + 143, 158, 158, 158, 162, 0, 0, 0, 161, 162, + 156, 156, 156, 157, 157, 157, 163, 169, 159, 159, + 159, 0, 0, 163, 169, 143, 160, 160, 160, 161, + 165, 162, 162, 162, 164, 161, 161, 161, 0, 164, + 166, 168, 165, 163, 163, 163, 166, 167, 0, 0, + 168, 169, 169, 169, 0, 174, 170, 165, 165, 165, + 172, 164, 164, 164, 171, 167, 171, 166, 166, 166, + 170, 174, 0, 0, 167, 167, 167, 168, 168, 168, + + 173, 172, 174, 174, 174, 176, 0, 172, 172, 172, + 175, 171, 171, 171, 177, 173, 175, 170, 170, 170, + 178, 0, 178, 176, 0, 179, 0, 173, 173, 173, + 177, 179, 176, 176, 176, 183, 0, 175, 175, 175, + 0, 177, 177, 177, 181, 184, 0, 178, 178, 178, + 0, 190, 179, 179, 179, 180, 183, 181, 180, 182, + 0, 182, 183, 183, 183, 0, 0, 0, 184, 190, + 0, 181, 181, 181, 184, 0, 0, 185, 190, 190, + 190, 189, 180, 180, 180, 185, 182, 182, 182, 0, + 187, 186, 0, 189, 213, 184, 184, 184, 186, 188, + + 213, 0, 191, 0, 185, 185, 185, 187, 189, 189, + 189, 187, 191, 0, 0, 192, 188, 213, 186, 186, + 186, 192, 205, 191, 213, 193, 188, 188, 188, 191, + 191, 191, 195, 0, 187, 187, 187, 194, 0, 0, + 196, 0, 192, 192, 192, 193, 195, 197, 0, 0, + 196, 205, 193, 193, 193, 0, 0, 194, 0, 195, + 195, 195, 197, 198, 194, 194, 194, 196, 196, 196, + 219, 0, 0, 198, 197, 197, 197, 217, 205, 205, + 205, 206, 206, 206, 206, 206, 206, 206, 218, 217, + 198, 198, 198, 220, 221, 0, 222, 219, 219, 219, + + 224, 0, 0, 223, 217, 217, 217, 220, 218, 0, + 221, 0, 0, 223, 224, 218, 218, 218, 226, 0, + 220, 220, 220, 222, 222, 222, 225, 224, 224, 224, + 226, 0, 225, 227, 0, 0, 228, 221, 221, 221, + 223, 223, 223, 227, 230, 226, 226, 226, 228, 229, + 0, 0, 231, 225, 225, 225, 228, 230, 231, 0, + 227, 227, 227, 228, 228, 228, 232, 0, 0, 229, + 0, 230, 230, 230, 0, 233, 229, 229, 229, 231, + 231, 231, 234, 0, 232, 233, 0, 236, 0, 0, + 235, 0, 0, 232, 232, 232, 234, 235, 0, 0, + + 237, 0, 233, 233, 233, 236, 237, 238, 0, 234, + 234, 234, 239, 0, 236, 236, 236, 235, 235, 235, + 0, 241, 240, 0, 0, 0, 243, 237, 237, 237, + 242, 0, 247, 244, 238, 238, 238, 240, 243, 239, + 239, 239, 241, 0, 242, 245, 0, 250, 246, 240, + 240, 240, 249, 243, 243, 243, 247, 242, 242, 242, + 244, 244, 244, 250, 246, 245, 0, 252, 0, 241, + 241, 241, 245, 245, 245, 246, 246, 246, 248, 249, + 249, 249, 251, 247, 247, 247, 248, 251, 252, 253, + 250, 250, 250, 254, 252, 252, 252, 0, 255, 0, + + 0, 0, 0, 0, 256, 248, 248, 248, 256, 251, + 251, 251, 255, 254, 0, 257, 253, 253, 253, 0, + 254, 254, 254, 258, 259, 255, 255, 255, 260, 258, + 0, 256, 256, 256, 257, 261, 260, 268, 0, 0, + 262, 0, 257, 257, 257, 0, 0, 259, 261, 0, + 258, 258, 258, 259, 0, 260, 260, 260, 262, 263, + 0, 263, 261, 261, 261, 0, 268, 262, 262, 262, + 0, 276, 0, 277, 259, 259, 259, 276, 279, 278, + 269, 269, 269, 269, 269, 277, 263, 263, 263, 278, + 280, 0, 279, 268, 268, 268, 269, 282, 276, 276, + + 276, 282, 280, 0, 0, 281, 278, 278, 278, 283, + 0, 283, 277, 277, 277, 0, 281, 269, 0, 279, + 279, 279, 286, 0, 282, 282, 282, 0, 286, 280, + 280, 280, 281, 281, 281, 0, 283, 283, 283, 284, + 0, 284, 269, 285, 0, 285, 0, 0, 287, 286, + 286, 286, 287, 288, 0, 288, 0, 0, 289, 0, + 0, 0, 0, 0, 290, 0, 284, 284, 284, 0, + 0, 291, 285, 285, 285, 287, 287, 287, 289, 290, + 288, 288, 288, 291, 292, 289, 289, 289, 293, 0, + 0, 290, 290, 290, 294, 293, 294, 296, 291, 291, + + 291, 295, 0, 0, 292, 300, 0, 0, 298, 0, + 301, 292, 292, 292, 299, 293, 293, 293, 296, 295, + 299, 294, 294, 294, 296, 296, 296, 298, 295, 295, + 295, 301, 300, 300, 300, 298, 298, 298, 302, 304, + 0, 299, 299, 299, 303, 0, 0, 305, 0, 0, + 303, 0, 0, 304, 305, 0, 306, 0, 301, 301, + 301, 307, 306, 0, 0, 302, 302, 302, 0, 308, + 310, 303, 303, 303, 305, 305, 305, 308, 312, 309, + 304, 304, 304, 306, 306, 306, 311, 0, 307, 307, + 307, 310, 0, 0, 0, 313, 308, 308, 308, 309, + + 0, 0, 312, 313, 0, 314, 309, 309, 309, 0, + 0, 315, 0, 311, 311, 311, 316, 0, 310, 310, + 310, 315, 313, 313, 313, 317, 0, 0, 317, 312, + 312, 312, 314, 314, 314, 324, 316, 325, 315, 315, + 315, 326, 0, 316, 316, 316, 0, 324, 0, 0, + 327, 325, 317, 317, 317, 328, 0, 329, 0, 328, + 0, 326, 329, 0, 325, 325, 325, 332, 326, 326, + 326, 0, 0, 330, 324, 324, 324, 327, 327, 327, + 333, 332, 333, 331, 0, 0, 328, 328, 328, 329, + 329, 329, 330, 0, 332, 332, 332, 334, 0, 0, + + 330, 330, 330, 331, 335, 0, 0, 333, 333, 333, + 331, 331, 331, 335, 336, 334, 0, 337, 0, 0, + 338, 337, 0, 339, 334, 334, 334, 340, 0, 0, + 342, 335, 335, 335, 336, 339, 338, 340, 342, 0, + 0, 336, 336, 336, 337, 337, 337, 338, 338, 338, + 339, 339, 339, 341, 340, 340, 340, 342, 342, 342, + 343, 0, 0, 343, 344, 341, 0, 345, 0, 0, + 346, 0, 0, 0, 0, 349, 0, 345, 344, 350, + 341, 341, 341, 348, 0, 0, 353, 343, 343, 343, + 356, 344, 344, 344, 345, 345, 345, 346, 346, 346, + + 351, 348, 349, 349, 349, 355, 350, 350, 350, 0, + 348, 348, 348, 353, 353, 353, 354, 356, 356, 356, + 365, 351, 354, 357, 0, 355, 360, 351, 351, 351, + 361, 363, 355, 355, 355, 357, 0, 361, 362, 0, + 360, 0, 0, 354, 354, 354, 363, 365, 365, 365, + 357, 357, 357, 360, 360, 360, 366, 361, 361, 361, + 364, 362, 366, 367, 0, 362, 362, 362, 368, 0, + 364, 0, 369, 363, 363, 363, 370, 367, 369, 0, + 368, 370, 372, 366, 366, 366, 0, 364, 364, 364, + 367, 367, 367, 376, 0, 368, 368, 368, 376, 369, + + 369, 369, 371, 370, 370, 370, 372, 373, 371, 372, + 372, 372, 374, 373, 0, 375, 0, 0, 374, 378, + 376, 376, 376, 377, 0, 0, 0, 375, 0, 371, + 371, 371, 0, 386, 373, 373, 373, 379, 377, 374, + 374, 374, 375, 375, 375, 380, 378, 378, 378, 386, + 377, 377, 377, 381, 0, 379, 382, 0, 0, 381, + 384, 0, 382, 380, 379, 379, 379, 384, 385, 0, + 0, 0, 380, 380, 380, 385, 386, 386, 386, 387, + 381, 381, 381, 382, 382, 382, 389, 384, 384, 384, + 388, 390, 389, 0, 394, 385, 385, 385, 391, 0, + + 394, 388, 0, 387, 392, 0, 387, 387, 387, 395, + 0, 392, 390, 389, 389, 389, 391, 393, 390, 390, + 390, 394, 394, 394, 395, 391, 391, 391, 388, 388, + 388, 392, 392, 392, 396, 393, 395, 395, 395, 397, + 0, 0, 398, 0, 393, 393, 393, 399, 398, 0, + 400, 0, 0, 397, 401, 400, 0, 403, 0, 0, + 404, 396, 396, 396, 0, 0, 397, 397, 397, 398, + 398, 398, 0, 401, 399, 399, 399, 400, 400, 400, + 409, 401, 401, 401, 403, 403, 403, 404, 404, 404, + 405, 410, 0, 406, 0, 0, 405, 407, 0, 406, + + 408, 0, 415, 0, 0, 409, 408, 0, 0, 0, + 411, 0, 0, 412, 0, 410, 407, 405, 405, 405, + 406, 406, 406, 415, 407, 407, 407, 408, 408, 408, + 411, 412, 409, 409, 409, 419, 413, 411, 411, 411, + 413, 432, 410, 410, 410, 0, 0, 414, 0, 432, + 415, 415, 415, 416, 0, 0, 417, 0, 412, 412, + 412, 419, 0, 413, 413, 413, 414, 418, 432, 432, + 432, 0, 416, 0, 414, 414, 414, 420, 417, 423, + 416, 416, 416, 417, 417, 417, 418, 420, 419, 419, + 419, 421, 0, 0, 418, 418, 418, 422, 0, 0, + + 423, 421, 0, 425, 420, 420, 420, 426, 422, 425, + 428, 0, 428, 426, 0, 0, 0, 427, 421, 421, + 421, 430, 0, 0, 422, 422, 422, 423, 423, 423, + 425, 425, 425, 430, 426, 426, 426, 427, 431, 428, + 428, 428, 434, 0, 427, 427, 427, 433, 430, 430, + 430, 0, 0, 433, 435, 0, 0, 431, 436, 0, + 434, 437, 0, 0, 438, 431, 431, 431, 439, 434, + 434, 434, 435, 436, 433, 433, 433, 444, 439, 438, + 437, 435, 435, 435, 441, 436, 436, 436, 437, 437, + 437, 438, 438, 438, 440, 439, 439, 439, 444, 441, + + 442, 443, 0, 0, 444, 444, 444, 440, 0, 442, + 0, 441, 441, 441, 445, 443, 0, 446, 0, 0, + 446, 440, 440, 440, 0, 447, 0, 445, 443, 443, + 443, 447, 448, 0, 0, 451, 442, 442, 442, 449, + 451, 445, 445, 445, 446, 446, 446, 449, 0, 0, + 448, 450, 447, 447, 447, 452, 0, 0, 450, 448, + 448, 448, 453, 0, 0, 452, 456, 451, 451, 451, + 453, 454, 0, 456, 449, 449, 449, 455, 450, 450, + 450, 0, 452, 452, 452, 454, 457, 455, 0, 453, + 453, 453, 461, 456, 456, 456, 457, 458, 454, 454, + + 454, 458, 460, 0, 455, 455, 455, 459, 0, 459, + 463, 462, 461, 457, 457, 457, 460, 0, 465, 461, + 461, 461, 463, 0, 458, 458, 458, 462, 465, 460, + 460, 460, 464, 0, 459, 459, 459, 466, 0, 0, + 464, 0, 0, 467, 0, 465, 465, 465, 0, 463, + 463, 463, 0, 0, 462, 462, 462, 466, 0, 464, + 464, 464, 467, 0, 466, 466, 466, 0, 0, 0, + 467, 467, 467, 469, 469, 469, 469, 469, 469, 469, + 469, 469, 469, 469, 469, 469, 469, 470, 470, 470, + 470, 470, 470, 470, 470, 470, 470, 470, 470, 470, + + 470, 471, 471, 471, 471, 471, 471, 471, 471, 471, + 471, 471, 471, 471, 471, 472, 472, 472, 472, 472, + 472, 472, 472, 472, 472, 472, 472, 472, 472, 473, + 0, 473, 0, 473, 0, 473, 474, 0, 0, 474, + 0, 474, 474, 474, 474, 474, 475, 475, 0, 475, + 475, 475, 475, 475, 475, 476, 476, 476, 476, 476, + 476, 476, 476, 476, 476, 476, 476, 476, 476, 477, + 477, 477, 477, 477, 477, 477, 477, 477, 477, 477, + 477, 477, 478, 0, 478, 478, 478, 478, 478, 478, + 478, 478, 478, 478, 478, 478, 479, 0, 479, 479, + + 479, 479, 479, 479, 479, 479, 479, 479, 479, 479, + 480, 480, 480, 480, 480, 480, 480, 480, 480, 481, + 481, 481, 482, 0, 0, 0, 0, 482, 482, 482, + 483, 483, 483, 483, 483, 484, 484, 484, 0, 0, + 484, 485, 485, 485, 486, 486, 486, 486, 486, 486, + 486, 486, 486, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468, 468, 468, 468, 468, 468, 468, + 468, 468, 468, 468 } ; static yy_state_type yy_last_accepting_state; @@ -1538,10 +1594,11 @@ struct Specialtoken specials[] = { {"_SuperblockVersion",_SUPERBLOCK,_SUPERBLOCK_FLAG}, {"_Filter",_FILTER,_FILTER_FLAG}, {"_Codecs",_CODECS,_CODECS_FLAG}, +{"_QuantizeBitgroomNumberOfSignificantDigits",_QUANTIZE,_QUANTIZE_FLAG}, {NULL,0} /* null terminate */ }; -#line 1545 "ncgenl.c" +#line 1602 "ncgenl.c" /* The most correct (validating) version of UTF8 character set (Taken from: http://www.w3.org/2005/03/23-lex-U) @@ -1584,7 +1641,7 @@ ID ([A-Za-z_]|{UTF8})([A-Z.@#\[\]a-z_0-9+-]|{UTF8})* /* Note: this definition of string will work for utf8 as well, although it is a very relaxed definition */ -#line 1588 "ncgenl.c" +#line 1645 "ncgenl.c" #define INITIAL 0 #define ST_C_COMMENT 1 @@ -1803,9 +1860,9 @@ YY_DECL } { -#line 223 "ncgen.l" +#line 224 "ncgen.l" -#line 1809 "ncgenl.c" +#line 1866 "ncgenl.c" while ( /*CONSTCOND*/1 ) /* loops until end-of-file is reached */ { @@ -1832,13 +1889,13 @@ YY_DECL while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; - if ( yy_current_state >= 429 ) + if ( yy_current_state >= 469 ) yy_c = yy_meta[yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + yy_c]; ++yy_cp; } - while ( yy_base[yy_current_state] != 2458 ); + while ( yy_base[yy_current_state] != 2654 ); yy_find_action: yy_act = yy_accept[yy_current_state]; @@ -1864,14 +1921,14 @@ YY_DECL case 1: YY_RULE_SETUP -#line 224 "ncgen.l" +#line 225 "ncgen.l" { /* whitespace */ break; } YY_BREAK case 2: YY_RULE_SETUP -#line 228 "ncgen.l" +#line 229 "ncgen.l" { /* comment */ break; } @@ -1879,7 +1936,7 @@ YY_RULE_SETUP case 3: /* rule 3 can match eol */ YY_RULE_SETUP -#line 232 "ncgen.l" +#line 233 "ncgen.l" {int len; char* s = NULL; /* In netcdf4, this will be used in a variety of places, so only remove escapes */ @@ -1903,7 +1960,7 @@ yytext[MAXTRST-1] = '\0'; YY_BREAK case 4: YY_RULE_SETUP -#line 253 "ncgen.l" +#line 254 "ncgen.l" { /* drop leading 0x; pad to even number of chars */ char* p = yytext+2; int len = yyleng - 2; @@ -1918,143 +1975,143 @@ YY_RULE_SETUP YY_BREAK case 5: YY_RULE_SETUP -#line 265 "ncgen.l" +#line 266 "ncgen.l" {return lexdebug(COMPOUND);} YY_BREAK case 6: YY_RULE_SETUP -#line 266 "ncgen.l" +#line 267 "ncgen.l" {return lexdebug(ENUM);} YY_BREAK case 7: YY_RULE_SETUP -#line 267 "ncgen.l" +#line 268 "ncgen.l" {return lexdebug(OPAQUE_);} YY_BREAK case 8: YY_RULE_SETUP -#line 269 "ncgen.l" +#line 270 "ncgen.l" {return lexdebug(FLOAT_K);} YY_BREAK case 9: YY_RULE_SETUP -#line 270 "ncgen.l" +#line 271 "ncgen.l" {return lexdebug(DOUBLE_K);} YY_BREAK case 10: YY_RULE_SETUP -#line 271 "ncgen.l" +#line 272 "ncgen.l" {return lexdebug(CHAR_K);} YY_BREAK case 11: YY_RULE_SETUP -#line 272 "ncgen.l" +#line 273 "ncgen.l" {return lexdebug(BYTE_K);} YY_BREAK case 12: YY_RULE_SETUP -#line 273 "ncgen.l" +#line 274 "ncgen.l" {return lexdebug(SHORT_K);} YY_BREAK case 13: YY_RULE_SETUP -#line 274 "ncgen.l" +#line 275 "ncgen.l" {return lexdebug(INT_K);} YY_BREAK case 14: YY_RULE_SETUP -#line 275 "ncgen.l" +#line 276 "ncgen.l" {return lexdebug(identcheck(UBYTE_K));} YY_BREAK case 15: YY_RULE_SETUP -#line 276 "ncgen.l" +#line 277 "ncgen.l" {return lexdebug(identcheck(USHORT_K));} YY_BREAK case 16: YY_RULE_SETUP -#line 277 "ncgen.l" +#line 278 "ncgen.l" {return lexdebug(identcheck(UINT_K));} YY_BREAK case 17: YY_RULE_SETUP -#line 278 "ncgen.l" +#line 279 "ncgen.l" {return lexdebug(identcheck(INT64_K));} YY_BREAK case 18: YY_RULE_SETUP -#line 279 "ncgen.l" +#line 280 "ncgen.l" {return lexdebug(identcheck(UINT64_K));} YY_BREAK case 19: YY_RULE_SETUP -#line 280 "ncgen.l" +#line 281 "ncgen.l" {return lexdebug(identcheck(STRING_K));} YY_BREAK case 20: YY_RULE_SETUP -#line 282 "ncgen.l" +#line 283 "ncgen.l" {return lexdebug(FLOAT_K);} YY_BREAK case 21: YY_RULE_SETUP -#line 283 "ncgen.l" +#line 284 "ncgen.l" {return lexdebug(INT_K);} YY_BREAK case 22: YY_RULE_SETUP -#line 284 "ncgen.l" +#line 285 "ncgen.l" {return lexdebug(INT_K);} YY_BREAK case 23: YY_RULE_SETUP -#line 285 "ncgen.l" +#line 286 "ncgen.l" {return lexdebug(identcheck(UINT_K));} YY_BREAK case 24: YY_RULE_SETUP -#line 286 "ncgen.l" +#line 287 "ncgen.l" {return lexdebug(identcheck(UINT_K));} YY_BREAK case 25: YY_RULE_SETUP -#line 289 "ncgen.l" +#line 290 "ncgen.l" {int32_val = -1; return lexdebug(NC_UNLIMITED_K);} YY_BREAK case 26: YY_RULE_SETUP -#line 292 "ncgen.l" +#line 293 "ncgen.l" {return lexdebug(TYPES);} YY_BREAK case 27: YY_RULE_SETUP -#line 293 "ncgen.l" +#line 294 "ncgen.l" {return lexdebug(DIMENSIONS);} YY_BREAK case 28: YY_RULE_SETUP -#line 294 "ncgen.l" +#line 295 "ncgen.l" {return lexdebug(VARIABLES);} YY_BREAK case 29: YY_RULE_SETUP -#line 295 "ncgen.l" +#line 296 "ncgen.l" {return lexdebug(DATA);} YY_BREAK case 30: YY_RULE_SETUP -#line 296 "ncgen.l" +#line 297 "ncgen.l" {return lexdebug(GROUP);} YY_BREAK case 31: YY_RULE_SETUP -#line 298 "ncgen.l" +#line 299 "ncgen.l" {BEGIN(TEXT);return lexdebug(NETCDF);} YY_BREAK case 32: YY_RULE_SETUP -#line 300 "ncgen.l" +#line 301 "ncgen.l" { /* missing value (pre-2.4 backward compatibility) */ if (yytext[0] == '-') { double_val = -INFINITY; @@ -2067,7 +2124,7 @@ YY_RULE_SETUP YY_BREAK case 33: YY_RULE_SETUP -#line 309 "ncgen.l" +#line 310 "ncgen.l" { /* missing value (pre-2.4 backward compatibility) */ double_val = NAN; specialconstants = 1; @@ -2076,7 +2133,7 @@ YY_RULE_SETUP YY_BREAK case 34: YY_RULE_SETUP -#line 315 "ncgen.l" +#line 316 "ncgen.l" {/* missing value (pre-2.4 backward compatibility)*/ if (yytext[0] == '-') { float_val = -INFINITYF; @@ -2089,7 +2146,7 @@ YY_RULE_SETUP YY_BREAK case 35: YY_RULE_SETUP -#line 324 "ncgen.l" +#line 325 "ncgen.l" { /* missing value (pre-2.4 backward compatibility) */ float_val = NANF; specialconstants = 1; @@ -2098,7 +2155,7 @@ YY_RULE_SETUP YY_BREAK case 36: YY_RULE_SETUP -#line 330 "ncgen.l" +#line 331 "ncgen.l" { #ifdef USE_NETCDF4 if(l_flag == L_C || l_flag == L_BINARY) @@ -2111,7 +2168,7 @@ YY_RULE_SETUP YY_BREAK case 37: YY_RULE_SETUP -#line 340 "ncgen.l" +#line 341 "ncgen.l" { bbClear(lextext); bbAppendn(lextext,(char*)yytext,yyleng+1); /* include null */ @@ -2122,7 +2179,7 @@ YY_RULE_SETUP YY_BREAK case 38: YY_RULE_SETUP -#line 349 "ncgen.l" +#line 350 "ncgen.l" {struct Specialtoken* st; bbClear(lextext); bbAppendn(lextext,(char*)yytext,yyleng+1); /* include null */ @@ -2136,7 +2193,7 @@ YY_RULE_SETUP case 39: /* rule 39 can match eol */ YY_RULE_SETUP -#line 359 "ncgen.l" +#line 360 "ncgen.l" { int c; char* p; char* q; @@ -2156,7 +2213,7 @@ YY_RULE_SETUP YY_BREAK case 40: YY_RULE_SETUP -#line 376 "ncgen.l" +#line 377 "ncgen.l" { char* id = NULL; int len; len = strlen(yytext); len = unescape(yytext,len,ISIDENT,&id); @@ -2171,7 +2228,7 @@ YY_RULE_SETUP YY_BREAK case 41: YY_RULE_SETUP -#line 388 "ncgen.l" +#line 389 "ncgen.l" { /* We need to try to see what size of integer ((u)int). @@ -2252,7 +2309,7 @@ done: return 0; YY_BREAK case 42: YY_RULE_SETUP -#line 466 "ncgen.l" +#line 467 "ncgen.l" { int c; int token = 0; @@ -2303,7 +2360,7 @@ YY_RULE_SETUP YY_BREAK case 43: YY_RULE_SETUP -#line 513 "ncgen.l" +#line 514 "ncgen.l" { if (sscanf((char*)yytext, "%le", &double_val) != 1) { sprintf(errstr,"bad long or double constant: %s",(char*)yytext); @@ -2314,7 +2371,7 @@ YY_RULE_SETUP YY_BREAK case 44: YY_RULE_SETUP -#line 520 "ncgen.l" +#line 521 "ncgen.l" { if (sscanf((char*)yytext, "%e", &float_val) != 1) { sprintf(errstr,"bad float constant: %s",(char*)yytext); @@ -2326,7 +2383,7 @@ YY_RULE_SETUP case 45: /* rule 45 can match eol */ YY_RULE_SETUP -#line 527 "ncgen.l" +#line 528 "ncgen.l" { (void) sscanf((char*)&yytext[1],"%c",&byte_val); return lexdebug(BYTE_CONST); @@ -2334,7 +2391,7 @@ YY_RULE_SETUP YY_BREAK case 46: YY_RULE_SETUP -#line 531 "ncgen.l" +#line 532 "ncgen.l" { int oct = unescapeoct(&yytext[2]); if(oct < 0) { @@ -2347,7 +2404,7 @@ YY_RULE_SETUP YY_BREAK case 47: YY_RULE_SETUP -#line 540 "ncgen.l" +#line 541 "ncgen.l" { int hex = unescapehex(&yytext[3]); if(byte_val < 0) { @@ -2360,7 +2417,7 @@ YY_RULE_SETUP YY_BREAK case 48: YY_RULE_SETUP -#line 549 "ncgen.l" +#line 550 "ncgen.l" { switch ((char)yytext[2]) { case 'a': byte_val = '\007'; break; /* not everyone under- @@ -2382,7 +2439,7 @@ YY_RULE_SETUP case 49: /* rule 49 can match eol */ YY_RULE_SETUP -#line 567 "ncgen.l" +#line 568 "ncgen.l" { lineno++ ; break; @@ -2390,7 +2447,7 @@ YY_RULE_SETUP YY_BREAK case 50: YY_RULE_SETUP -#line 572 "ncgen.l" +#line 573 "ncgen.l" {/*initial*/ BEGIN(ST_C_COMMENT); break; @@ -2399,21 +2456,21 @@ YY_RULE_SETUP case 51: /* rule 51 can match eol */ YY_RULE_SETUP -#line 577 "ncgen.l" +#line 578 "ncgen.l" {/* continuation */ break; } YY_BREAK case 52: YY_RULE_SETUP -#line 581 "ncgen.l" +#line 582 "ncgen.l" {/* final */ BEGIN(INITIAL); break; } YY_BREAK case YY_STATE_EOF(ST_C_COMMENT): -#line 586 "ncgen.l" +#line 587 "ncgen.l" {/* final, error */ fprintf(stderr,"unterminated /**/ comment"); BEGIN(INITIAL); @@ -2422,17 +2479,17 @@ case YY_STATE_EOF(ST_C_COMMENT): YY_BREAK case 53: YY_RULE_SETUP -#line 592 "ncgen.l" +#line 593 "ncgen.l" {/* Note: this next rule will not work for UTF8 characters */ return lexdebug(yytext[0]) ; } YY_BREAK case 54: YY_RULE_SETUP -#line 595 "ncgen.l" +#line 596 "ncgen.l" ECHO; YY_BREAK -#line 2436 "ncgenl.c" +#line 2493 "ncgenl.c" case YY_STATE_EOF(INITIAL): case YY_STATE_EOF(TEXT): yyterminate(); @@ -2730,7 +2787,7 @@ static int yy_get_next_buffer (void) while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; - if ( yy_current_state >= 429 ) + if ( yy_current_state >= 469 ) yy_c = yy_meta[yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + yy_c]; @@ -2758,11 +2815,11 @@ static int yy_get_next_buffer (void) while ( yy_chk[yy_base[yy_current_state] + yy_c] != yy_current_state ) { yy_current_state = (int) yy_def[yy_current_state]; - if ( yy_current_state >= 429 ) + if ( yy_current_state >= 469 ) yy_c = yy_meta[yy_c]; } yy_current_state = yy_nxt[yy_base[yy_current_state] + yy_c]; - yy_is_jam = (yy_current_state == 428); + yy_is_jam = (yy_current_state == 468); return yy_is_jam ? 0 : yy_current_state; } @@ -3438,7 +3495,7 @@ void yyfree (void * ptr ) #define YYTABLES_NAME "yytables" -#line 595 "ncgen.l" +#line 596 "ncgen.l" static int lexdebug(int token) diff --git a/ncgen/ncgeny.c b/ncgen/ncgeny.c index d12ba56d54..dfee221fd3 100644 --- a/ncgen/ncgeny.c +++ b/ncgen/ncgeny.c @@ -299,83 +299,84 @@ enum yysymbol_kind_t YYSYMBOL__SUPERBLOCK = 53, /* _SUPERBLOCK */ YYSYMBOL__FILTER = 54, /* _FILTER */ YYSYMBOL__CODECS = 55, /* _CODECS */ - YYSYMBOL_DATASETID = 56, /* DATASETID */ - YYSYMBOL_57_ = 57, /* '{' */ - YYSYMBOL_58_ = 58, /* '}' */ - YYSYMBOL_59_ = 59, /* ';' */ - YYSYMBOL_60_ = 60, /* ',' */ - YYSYMBOL_61_ = 61, /* '=' */ - YYSYMBOL_62_ = 62, /* '(' */ - YYSYMBOL_63_ = 63, /* ')' */ - YYSYMBOL_64_ = 64, /* '*' */ - YYSYMBOL_65_ = 65, /* ':' */ - YYSYMBOL_YYACCEPT = 66, /* $accept */ - YYSYMBOL_ncdesc = 67, /* ncdesc */ - YYSYMBOL_datasetid = 68, /* datasetid */ - YYSYMBOL_rootgroup = 69, /* rootgroup */ - YYSYMBOL_groupbody = 70, /* groupbody */ - YYSYMBOL_subgrouplist = 71, /* subgrouplist */ - YYSYMBOL_namedgroup = 72, /* namedgroup */ - YYSYMBOL_73_1 = 73, /* $@1 */ - YYSYMBOL_74_2 = 74, /* $@2 */ - YYSYMBOL_typesection = 75, /* typesection */ - YYSYMBOL_typedecls = 76, /* typedecls */ - YYSYMBOL_typename = 77, /* typename */ - YYSYMBOL_type_or_attr_decl = 78, /* type_or_attr_decl */ - YYSYMBOL_typedecl = 79, /* typedecl */ - YYSYMBOL_optsemicolon = 80, /* optsemicolon */ - YYSYMBOL_enumdecl = 81, /* enumdecl */ - YYSYMBOL_enumidlist = 82, /* enumidlist */ - YYSYMBOL_enumid = 83, /* enumid */ - YYSYMBOL_opaquedecl = 84, /* opaquedecl */ - YYSYMBOL_vlendecl = 85, /* vlendecl */ - YYSYMBOL_compounddecl = 86, /* compounddecl */ - YYSYMBOL_fields = 87, /* fields */ - YYSYMBOL_field = 88, /* field */ - YYSYMBOL_primtype = 89, /* primtype */ - YYSYMBOL_dimsection = 90, /* dimsection */ - YYSYMBOL_dimdecls = 91, /* dimdecls */ - YYSYMBOL_dim_or_attr_decl = 92, /* dim_or_attr_decl */ - YYSYMBOL_dimdeclist = 93, /* dimdeclist */ - YYSYMBOL_dimdecl = 94, /* dimdecl */ - YYSYMBOL_dimd = 95, /* dimd */ - YYSYMBOL_vasection = 96, /* vasection */ - YYSYMBOL_vadecls = 97, /* vadecls */ - YYSYMBOL_vadecl_or_attr = 98, /* vadecl_or_attr */ - YYSYMBOL_vardecl = 99, /* vardecl */ - YYSYMBOL_varlist = 100, /* varlist */ - YYSYMBOL_varspec = 101, /* varspec */ - YYSYMBOL_dimspec = 102, /* dimspec */ - YYSYMBOL_dimlist = 103, /* dimlist */ - YYSYMBOL_dimref = 104, /* dimref */ - YYSYMBOL_fieldlist = 105, /* fieldlist */ - YYSYMBOL_fieldspec = 106, /* fieldspec */ - YYSYMBOL_fielddimspec = 107, /* fielddimspec */ - YYSYMBOL_fielddimlist = 108, /* fielddimlist */ - YYSYMBOL_fielddim = 109, /* fielddim */ - YYSYMBOL_varref = 110, /* varref */ - YYSYMBOL_typeref = 111, /* typeref */ - YYSYMBOL_ambiguous_ref = 112, /* ambiguous_ref */ - YYSYMBOL_attrdecllist = 113, /* attrdecllist */ - YYSYMBOL_attrdecl = 114, /* attrdecl */ - YYSYMBOL_path = 115, /* path */ - YYSYMBOL_datasection = 116, /* datasection */ - YYSYMBOL_datadecls = 117, /* datadecls */ - YYSYMBOL_datadecl = 118, /* datadecl */ - YYSYMBOL_datalist = 119, /* datalist */ - YYSYMBOL_datalist0 = 120, /* datalist0 */ - YYSYMBOL_datalist1 = 121, /* datalist1 */ - YYSYMBOL_dataitem = 122, /* dataitem */ - YYSYMBOL_constdata = 123, /* constdata */ - YYSYMBOL_econstref = 124, /* econstref */ - YYSYMBOL_function = 125, /* function */ - YYSYMBOL_arglist = 126, /* arglist */ - YYSYMBOL_simpleconstant = 127, /* simpleconstant */ - YYSYMBOL_intlist = 128, /* intlist */ - YYSYMBOL_constint = 129, /* constint */ - YYSYMBOL_conststring = 130, /* conststring */ - YYSYMBOL_constbool = 131, /* constbool */ - YYSYMBOL_ident = 132 /* ident */ + YYSYMBOL__QUANTIZE = 56, /* _QUANTIZE */ + YYSYMBOL_DATASETID = 57, /* DATASETID */ + YYSYMBOL_58_ = 58, /* '{' */ + YYSYMBOL_59_ = 59, /* '}' */ + YYSYMBOL_60_ = 60, /* ';' */ + YYSYMBOL_61_ = 61, /* ',' */ + YYSYMBOL_62_ = 62, /* '=' */ + YYSYMBOL_63_ = 63, /* '(' */ + YYSYMBOL_64_ = 64, /* ')' */ + YYSYMBOL_65_ = 65, /* '*' */ + YYSYMBOL_66_ = 66, /* ':' */ + YYSYMBOL_YYACCEPT = 67, /* $accept */ + YYSYMBOL_ncdesc = 68, /* ncdesc */ + YYSYMBOL_datasetid = 69, /* datasetid */ + YYSYMBOL_rootgroup = 70, /* rootgroup */ + YYSYMBOL_groupbody = 71, /* groupbody */ + YYSYMBOL_subgrouplist = 72, /* subgrouplist */ + YYSYMBOL_namedgroup = 73, /* namedgroup */ + YYSYMBOL_74_1 = 74, /* $@1 */ + YYSYMBOL_75_2 = 75, /* $@2 */ + YYSYMBOL_typesection = 76, /* typesection */ + YYSYMBOL_typedecls = 77, /* typedecls */ + YYSYMBOL_typename = 78, /* typename */ + YYSYMBOL_type_or_attr_decl = 79, /* type_or_attr_decl */ + YYSYMBOL_typedecl = 80, /* typedecl */ + YYSYMBOL_optsemicolon = 81, /* optsemicolon */ + YYSYMBOL_enumdecl = 82, /* enumdecl */ + YYSYMBOL_enumidlist = 83, /* enumidlist */ + YYSYMBOL_enumid = 84, /* enumid */ + YYSYMBOL_opaquedecl = 85, /* opaquedecl */ + YYSYMBOL_vlendecl = 86, /* vlendecl */ + YYSYMBOL_compounddecl = 87, /* compounddecl */ + YYSYMBOL_fields = 88, /* fields */ + YYSYMBOL_field = 89, /* field */ + YYSYMBOL_primtype = 90, /* primtype */ + YYSYMBOL_dimsection = 91, /* dimsection */ + YYSYMBOL_dimdecls = 92, /* dimdecls */ + YYSYMBOL_dim_or_attr_decl = 93, /* dim_or_attr_decl */ + YYSYMBOL_dimdeclist = 94, /* dimdeclist */ + YYSYMBOL_dimdecl = 95, /* dimdecl */ + YYSYMBOL_dimd = 96, /* dimd */ + YYSYMBOL_vasection = 97, /* vasection */ + YYSYMBOL_vadecls = 98, /* vadecls */ + YYSYMBOL_vadecl_or_attr = 99, /* vadecl_or_attr */ + YYSYMBOL_vardecl = 100, /* vardecl */ + YYSYMBOL_varlist = 101, /* varlist */ + YYSYMBOL_varspec = 102, /* varspec */ + YYSYMBOL_dimspec = 103, /* dimspec */ + YYSYMBOL_dimlist = 104, /* dimlist */ + YYSYMBOL_dimref = 105, /* dimref */ + YYSYMBOL_fieldlist = 106, /* fieldlist */ + YYSYMBOL_fieldspec = 107, /* fieldspec */ + YYSYMBOL_fielddimspec = 108, /* fielddimspec */ + YYSYMBOL_fielddimlist = 109, /* fielddimlist */ + YYSYMBOL_fielddim = 110, /* fielddim */ + YYSYMBOL_varref = 111, /* varref */ + YYSYMBOL_typeref = 112, /* typeref */ + YYSYMBOL_ambiguous_ref = 113, /* ambiguous_ref */ + YYSYMBOL_attrdecllist = 114, /* attrdecllist */ + YYSYMBOL_attrdecl = 115, /* attrdecl */ + YYSYMBOL_path = 116, /* path */ + YYSYMBOL_datasection = 117, /* datasection */ + YYSYMBOL_datadecls = 118, /* datadecls */ + YYSYMBOL_datadecl = 119, /* datadecl */ + YYSYMBOL_datalist = 120, /* datalist */ + YYSYMBOL_datalist0 = 121, /* datalist0 */ + YYSYMBOL_datalist1 = 122, /* datalist1 */ + YYSYMBOL_dataitem = 123, /* dataitem */ + YYSYMBOL_constdata = 124, /* constdata */ + YYSYMBOL_econstref = 125, /* econstref */ + YYSYMBOL_function = 126, /* function */ + YYSYMBOL_arglist = 127, /* arglist */ + YYSYMBOL_simpleconstant = 128, /* simpleconstant */ + YYSYMBOL_intlist = 129, /* intlist */ + YYSYMBOL_constint = 130, /* constint */ + YYSYMBOL_conststring = 131, /* conststring */ + YYSYMBOL_constbool = 132, /* constbool */ + YYSYMBOL_ident = 133 /* ident */ }; typedef enum yysymbol_kind_t yysymbol_kind_t; @@ -697,19 +698,19 @@ union yyalloc /* YYFINAL -- State number of the termination state. */ #define YYFINAL 5 /* YYLAST -- Last index in YYTABLE. */ -#define YYLAST 424 +#define YYLAST 405 /* YYNTOKENS -- Number of terminals. */ -#define YYNTOKENS 66 +#define YYNTOKENS 67 /* YYNNTS -- Number of nonterminals. */ #define YYNNTS 67 /* YYNRULES -- Number of rules. */ -#define YYNRULES 154 +#define YYNRULES 155 /* YYNSTATES -- Number of states. */ -#define YYNSTATES 265 +#define YYNSTATES 268 /* YYMAXUTOK -- Last valid token kind. */ -#define YYMAXUTOK 311 +#define YYMAXUTOK 312 /* YYTRANSLATE(TOKEN-NUM) -- Symbol number corresponding to TOKEN-NUM @@ -727,15 +728,15 @@ static const yytype_int8 yytranslate[] = 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, - 62, 63, 64, 2, 60, 2, 2, 2, 2, 2, - 2, 2, 2, 2, 2, 2, 2, 2, 65, 59, - 2, 61, 2, 2, 2, 2, 2, 2, 2, 2, + 63, 64, 65, 2, 61, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 66, 60, + 2, 62, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, - 2, 2, 2, 57, 2, 58, 2, 2, 2, 2, + 2, 2, 2, 58, 2, 59, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, @@ -754,29 +755,29 @@ static const yytype_int8 yytranslate[] = 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, - 55, 56 + 55, 56, 57 }; #if YYDEBUG /* YYRLINE[YYN] -- Source line where rule number YYN was defined. */ static const yytype_int16 yyrline[] = { - 0, 237, 237, 243, 245, 252, 259, 259, 262, 271, - 261, 276, 277, 278, 282, 282, 284, 294, 294, 297, - 298, 299, 300, 303, 303, 306, 336, 338, 355, 364, - 376, 390, 423, 424, 427, 441, 442, 443, 444, 445, - 446, 447, 448, 449, 450, 451, 452, 455, 456, 457, - 460, 461, 464, 464, 466, 467, 471, 479, 489, 501, - 502, 503, 506, 507, 510, 510, 512, 534, 538, 542, - 571, 572, 575, 576, 580, 594, 598, 603, 632, 633, - 637, 638, 643, 653, 673, 684, 695, 714, 721, 721, - 724, 726, 728, 730, 732, 741, 752, 754, 756, 758, - 760, 762, 764, 766, 768, 770, 772, 774, 779, 786, - 795, 796, 797, 800, 801, 804, 808, 809, 813, 817, - 818, 823, 824, 828, 829, 830, 831, 832, 833, 837, - 841, 845, 847, 852, 853, 854, 855, 856, 857, 858, - 859, 860, 861, 862, 863, 867, 868, 872, 874, 876, - 878, 883, 887, 888, 894 + 0, 238, 238, 244, 246, 253, 260, 260, 263, 272, + 262, 277, 278, 279, 283, 283, 285, 295, 295, 298, + 299, 300, 301, 304, 304, 307, 337, 339, 356, 365, + 377, 391, 424, 425, 428, 442, 443, 444, 445, 446, + 447, 448, 449, 450, 451, 452, 453, 456, 457, 458, + 461, 462, 465, 465, 467, 468, 472, 480, 490, 502, + 503, 504, 507, 508, 511, 511, 513, 535, 539, 543, + 572, 573, 576, 577, 581, 595, 599, 604, 633, 634, + 638, 639, 644, 654, 674, 685, 696, 715, 722, 722, + 725, 727, 729, 731, 733, 742, 753, 755, 757, 759, + 761, 763, 765, 767, 769, 771, 773, 775, 777, 782, + 789, 798, 799, 800, 803, 804, 807, 811, 812, 816, + 820, 821, 826, 827, 831, 832, 833, 834, 835, 836, + 840, 844, 848, 850, 855, 856, 857, 858, 859, 860, + 861, 862, 863, 864, 865, 866, 870, 871, 875, 877, + 879, 881, 886, 890, 891, 897 }; #endif @@ -802,21 +803,21 @@ static const char *const yytname[] = "GROUP", "PATH", "FILLMARKER", "NIL", "_FILLVALUE", "_FORMAT", "_STORAGE", "_CHUNKSIZES", "_DEFLATELEVEL", "_SHUFFLE", "_ENDIANNESS", "_NOFILL", "_FLETCHER32", "_NCPROPS", "_ISNETCDF4", "_SUPERBLOCK", - "_FILTER", "_CODECS", "DATASETID", "'{'", "'}'", "';'", "','", "'='", - "'('", "')'", "'*'", "':'", "$accept", "ncdesc", "datasetid", - "rootgroup", "groupbody", "subgrouplist", "namedgroup", "$@1", "$@2", - "typesection", "typedecls", "typename", "type_or_attr_decl", "typedecl", - "optsemicolon", "enumdecl", "enumidlist", "enumid", "opaquedecl", - "vlendecl", "compounddecl", "fields", "field", "primtype", "dimsection", - "dimdecls", "dim_or_attr_decl", "dimdeclist", "dimdecl", "dimd", - "vasection", "vadecls", "vadecl_or_attr", "vardecl", "varlist", - "varspec", "dimspec", "dimlist", "dimref", "fieldlist", "fieldspec", - "fielddimspec", "fielddimlist", "fielddim", "varref", "typeref", - "ambiguous_ref", "attrdecllist", "attrdecl", "path", "datasection", - "datadecls", "datadecl", "datalist", "datalist0", "datalist1", - "dataitem", "constdata", "econstref", "function", "arglist", - "simpleconstant", "intlist", "constint", "conststring", "constbool", - "ident", YY_NULLPTR + "_FILTER", "_CODECS", "_QUANTIZE", "DATASETID", "'{'", "'}'", "';'", + "','", "'='", "'('", "')'", "'*'", "':'", "$accept", "ncdesc", + "datasetid", "rootgroup", "groupbody", "subgrouplist", "namedgroup", + "$@1", "$@2", "typesection", "typedecls", "typename", + "type_or_attr_decl", "typedecl", "optsemicolon", "enumdecl", + "enumidlist", "enumid", "opaquedecl", "vlendecl", "compounddecl", + "fields", "field", "primtype", "dimsection", "dimdecls", + "dim_or_attr_decl", "dimdeclist", "dimdecl", "dimd", "vasection", + "vadecls", "vadecl_or_attr", "vardecl", "varlist", "varspec", "dimspec", + "dimlist", "dimref", "fieldlist", "fieldspec", "fielddimspec", + "fielddimlist", "fielddim", "varref", "typeref", "ambiguous_ref", + "attrdecllist", "attrdecl", "path", "datasection", "datadecls", + "datadecl", "datalist", "datalist0", "datalist1", "dataitem", + "constdata", "econstref", "function", "arglist", "simpleconstant", + "intlist", "constint", "conststring", "constbool", "ident", YY_NULLPTR }; static const char * @@ -836,17 +837,17 @@ static const yytype_int16 yytoknum[] = 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, - 305, 306, 307, 308, 309, 310, 311, 123, 125, 59, - 44, 61, 40, 41, 42, 58 + 305, 306, 307, 308, 309, 310, 311, 312, 123, 125, + 59, 44, 61, 40, 41, 42, 58 }; #endif -#define YYPACT_NINF (-147) +#define YYPACT_NINF (-149) #define yypact_value_is_default(Yyn) \ ((Yyn) == YYPACT_NINF) -#define YYTABLE_NINF (-109) +#define YYTABLE_NINF (-110) #define yytable_value_is_error(Yyn) \ 0 @@ -855,33 +856,33 @@ static const yytype_int16 yytoknum[] = STATE-NUM. */ static const yytype_int16 yypact[] = { - -12, -6, 56, -147, 1, -147, 229, -147, -147, -147, - -147, -147, -147, -147, -147, -147, -147, -147, -147, -147, - -147, -147, -4, -147, -147, 385, -3, 28, 12, -147, - -147, 14, 19, 21, 26, 33, -16, 31, 143, 216, - 48, 229, 82, 82, 76, 8, 347, 84, -147, -147, - -1, 42, 43, 46, 47, 50, 52, 55, 59, 60, - 61, 62, 84, 63, 216, -147, -147, 58, 58, 58, - 58, 89, 291, 67, 229, 97, -147, -147, -147, -147, - -147, -147, -147, -147, -147, -147, -147, -147, -147, -147, - -147, -147, -147, -147, -147, -147, -147, -147, -147, -147, - -147, -147, -147, 347, -147, 69, -147, -147, -147, -147, - -147, -147, -147, 73, 79, 77, 78, 347, 82, 8, - 8, 76, 82, 76, 76, 82, 82, 347, 80, -147, - 119, -147, -147, -147, -147, -147, -147, 84, 81, -147, - 229, 85, 83, -147, 88, -147, 92, 229, 109, 10, - 347, 239, -147, 347, 347, 69, -147, 86, -147, -147, - -147, -147, -147, -147, -147, -147, 69, 385, 87, 98, - 91, 99, -147, 84, 64, 229, 101, -147, 385, -147, - 385, -147, -147, -147, -37, -147, 229, 69, 69, 8, - 304, 102, 84, -147, 84, 84, 84, -147, -147, -147, - -147, -147, 103, -147, 104, -147, -30, 105, -147, 385, - 108, 239, -147, -147, -147, -147, 110, -147, 111, -147, - 106, -147, 16, -147, 115, -147, -147, 84, -2, -147, - 347, 118, -147, -147, 140, -147, 84, 30, -147, -147, - 84, 8, -147, 117, 0, -147, -147, 69, -147, 122, - -147, -147, -147, 9, -147, -147, -147, -2, -147, 229, - 30, -147, -147, -147, -147 + -2, -26, 38, -149, -14, -149, 232, -149, -149, -149, + -149, -149, -149, -149, -149, -149, -149, -149, -149, -149, + -149, -149, 24, -149, -149, 366, -12, 25, 2, -149, + -149, -1, 3, 11, 43, 48, -23, 15, 1, 219, + 58, 232, 89, 89, 78, 76, 328, 102, -149, -149, + -3, 64, 65, 66, 67, 69, 70, 74, 75, 77, + 80, 81, 82, 102, 83, 219, -149, -149, 87, 87, + 87, 87, 90, 268, 91, 232, 108, -149, -149, -149, + -149, -149, -149, -149, -149, -149, -149, -149, -149, -149, + -149, -149, -149, -149, -149, -149, -149, -149, -149, -149, + -149, -149, -149, -149, 328, -149, 92, -149, -149, -149, + -149, -149, -149, -149, 93, 94, 95, 96, 328, 89, + 76, 76, 78, 89, 78, 78, 89, 89, 76, 328, + 97, -149, 140, -149, -149, -149, -149, -149, -149, 102, + 99, -149, 232, 105, 106, -149, 104, -149, 107, 232, + 136, 62, 328, 242, -149, 328, 328, 92, -149, 109, + -149, -149, -149, -149, -149, -149, -149, -149, -149, 92, + 366, 110, 113, 114, 119, -149, 102, 68, 232, 120, + -149, 366, -149, 366, -149, -149, -149, -41, -149, 232, + 92, 92, 76, 304, 121, 102, -149, 102, 102, 102, + -149, -149, -149, -149, -149, 122, -149, 123, -149, -36, + 124, -149, 366, 125, 242, -149, -149, -149, -149, 128, + -149, 129, -149, 130, -149, 63, -149, 132, -149, -149, + 102, -5, -149, 328, 135, -149, -149, 145, -149, 102, + -7, -149, -149, 102, 76, -149, 133, 8, -149, -149, + 92, -149, 138, -149, -149, -149, 19, -149, -149, -149, + -5, -149, 232, -7, -149, -149, -149, -149 }; /* YYDEFACT[STATE-NUM] -- Default reduction number in state STATE-NUM. @@ -891,55 +892,55 @@ static const yytype_uint8 yydefact[] = { 0, 0, 0, 3, 0, 1, 88, 2, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, - 154, 109, 0, 6, 87, 0, 85, 11, 0, 86, - 108, 0, 0, 0, 0, 0, 0, 0, 0, 12, - 47, 88, 0, 0, 0, 0, 118, 0, 4, 7, + 155, 110, 0, 6, 87, 0, 85, 11, 0, 86, + 109, 0, 0, 0, 0, 0, 0, 0, 0, 12, + 47, 88, 0, 0, 0, 0, 119, 0, 4, 7, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 13, 14, 17, 23, 23, 23, - 23, 87, 0, 0, 48, 59, 89, 151, 107, 90, - 147, 149, 148, 150, 153, 152, 91, 92, 144, 133, + 0, 0, 0, 0, 0, 13, 14, 17, 23, 23, + 23, 23, 87, 0, 0, 48, 59, 89, 152, 108, + 90, 148, 150, 149, 151, 154, 153, 91, 92, 145, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, - 124, 125, 126, 118, 129, 93, 116, 117, 119, 121, - 127, 128, 123, 108, 0, 0, 0, 118, 0, 0, - 0, 0, 0, 0, 0, 0, 0, 118, 0, 16, - 0, 15, 24, 19, 22, 21, 20, 0, 0, 18, - 49, 0, 52, 54, 0, 53, 108, 60, 110, 0, - 0, 0, 8, 118, 118, 96, 98, 99, 145, 101, - 102, 103, 106, 100, 104, 105, 95, 0, 0, 0, - 0, 0, 50, 0, 0, 61, 0, 64, 0, 65, - 111, 5, 122, 120, 0, 131, 88, 97, 94, 0, - 0, 0, 0, 85, 0, 0, 0, 51, 55, 58, - 57, 56, 0, 62, 66, 67, 70, 0, 84, 112, - 0, 0, 130, 6, 146, 31, 0, 32, 34, 75, - 78, 29, 0, 26, 0, 30, 63, 0, 0, 69, - 118, 0, 113, 132, 9, 33, 0, 0, 77, 25, - 0, 0, 68, 70, 0, 72, 74, 115, 114, 0, - 76, 83, 82, 0, 80, 27, 28, 0, 71, 88, - 0, 79, 73, 10, 81 + 144, 125, 126, 127, 119, 130, 93, 117, 118, 120, + 122, 128, 129, 124, 109, 0, 0, 0, 119, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 119, + 0, 16, 0, 15, 24, 19, 22, 21, 20, 0, + 0, 18, 49, 0, 52, 54, 0, 53, 109, 60, + 111, 0, 0, 0, 8, 119, 119, 96, 98, 99, + 146, 101, 102, 103, 107, 100, 104, 105, 106, 95, + 0, 0, 0, 0, 0, 50, 0, 0, 61, 0, + 64, 0, 65, 112, 5, 123, 121, 0, 132, 88, + 97, 94, 0, 0, 0, 0, 85, 0, 0, 0, + 51, 55, 58, 57, 56, 0, 62, 66, 67, 70, + 0, 84, 113, 0, 0, 131, 6, 147, 31, 0, + 32, 34, 75, 78, 29, 0, 26, 0, 30, 63, + 0, 0, 69, 119, 0, 114, 133, 9, 33, 0, + 0, 77, 25, 0, 0, 68, 70, 0, 72, 74, + 116, 115, 0, 76, 83, 82, 0, 80, 27, 28, + 0, 71, 88, 0, 79, 73, 10, 81 }; /* YYPGOTO[NTERM-NUM]. */ static const yytype_int16 yypgoto[] = { - -147, -147, -147, -147, -5, -31, -147, -147, -147, -147, - -147, -130, 131, -147, -25, -147, -147, -57, -147, -147, - -147, -147, 6, -26, -147, -147, 66, -147, 29, -147, - -147, -147, 24, -147, -147, -24, -147, -147, -56, -147, - -32, -147, -147, -53, -147, -33, -15, -40, -28, -44, - -147, -147, 2, -100, -147, -147, 65, -147, -147, -147, - -147, -146, -147, -41, -34, -103, -22 + -149, -149, -149, -149, 9, -24, -149, -149, -149, -149, + -149, -129, 134, -149, 44, -149, -149, -43, -149, -149, + -149, -149, 10, -6, -149, -149, 59, -149, 28, -149, + -149, -149, 27, -149, -149, -20, -149, -149, -54, -149, + -32, -149, -149, -51, -149, -30, -21, -40, -33, -44, + -149, -149, 6, -92, -149, -149, 57, -149, -149, -149, + -149, -148, -149, -42, -35, -103, -22 }; /* YYDEFGOTO[NTERM-NUM]. */ -static const yytype_uint8 yydefgoto[] = +static const yytype_int16 yydefgoto[] = { - 0, 2, 4, 7, 23, 36, 49, 186, 249, 40, - 64, 128, 65, 66, 133, 67, 222, 223, 68, 69, - 70, 190, 191, 24, 75, 140, 141, 142, 143, 144, - 148, 175, 176, 177, 204, 205, 229, 244, 245, 218, - 219, 238, 253, 254, 207, 25, 26, 27, 28, 29, - 181, 209, 210, 105, 106, 107, 108, 109, 110, 111, - 184, 112, 157, 84, 85, 86, 30 + 0, 2, 4, 7, 23, 36, 49, 189, 252, 40, + 65, 130, 66, 67, 135, 68, 225, 226, 69, 70, + 71, 193, 194, 24, 76, 142, 143, 144, 145, 146, + 150, 178, 179, 180, 207, 208, 232, 247, 248, 221, + 222, 241, 256, 257, 210, 25, 26, 27, 28, 29, + 184, 212, 213, 106, 107, 108, 109, 110, 111, 112, + 187, 113, 159, 85, 86, 87, 30 }; /* YYTABLE[YYPACT[STATE-NUM]] -- What to do in state STATE-NUM. If @@ -947,150 +948,146 @@ static const yytype_uint8 yydefgoto[] = number is the opposite. If YYTABLE_NINF, syntax error. */ static const yytype_int16 yytable[] = { - 35, 76, 104, 149, 87, 185, 72, 169, 78, 79, - 37, 73, 20, 71, 20, 20, 61, 155, 160, 1, - 162, 163, 47, 211, 113, 114, 212, 166, 116, 80, - 81, 72, 228, 82, 83, -108, 73, 21, 71, 31, - 129, 115, 48, 134, 135, 136, 145, 32, 33, 34, - 3, 251, 146, 187, 188, 252, 5, 37, 6, 104, - 257, 39, 38, 258, 221, 233, 225, 200, 182, 260, - 150, 41, 261, 104, 239, 42, 240, 74, 158, 159, - 43, 113, 44, 104, 156, 80, 81, 45, 161, 82, - 83, 164, 165, 77, 46, 113, 50, 80, 81, 77, - 20, 82, 83, 117, 118, 113, 104, 119, 120, 104, - 104, 121, 145, 122, 178, 129, 123, 132, 146, 179, - 124, 125, 126, 127, 137, 130, 139, 147, 113, 150, - 247, 113, 113, 201, 192, 151, 152, 167, 153, 154, - 168, 180, 178, 173, 172, 170, 189, 179, 214, 174, - 194, 199, 193, -58, 196, 195, 206, 192, 197, 20, - 203, 217, 226, 37, 227, 208, 230, 232, 237, 235, - 220, 236, 129, 224, 129, 193, 241, 248, 47, 228, - 259, 213, 234, 255, 246, 51, 104, 52, 53, 54, - 55, 56, 57, 58, 208, 131, 216, 59, 60, 202, - 256, 262, 198, 242, 250, 243, 171, 264, 113, 0, - 0, 231, 0, 246, 220, 183, 0, 0, 224, 263, - 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, - 18, 19, 20, 8, 9, 10, 11, 12, 13, 14, - 15, 16, 17, 18, 19, 20, 0, 0, 0, 0, - 62, 0, 63, 0, 0, 21, 88, 89, 90, 91, - 92, 93, 94, 95, 96, 97, 98, 99, 21, 0, - 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 22, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 22, 8, 9, 10, 11, 12, - 13, 14, 15, 16, 17, 18, 19, 20, 8, 9, + 35, 77, 105, 88, 37, 188, 74, 79, 80, 73, + 172, 20, 151, 20, 254, 47, 62, 20, 255, 162, + 214, 164, 165, 215, 114, 115, 157, 231, 117, 1, + -109, 3, 74, 72, 21, 73, 48, 169, 5, 116, + 20, 131, 147, 51, 6, 52, 53, 54, 55, 56, + 57, 58, 37, 148, 38, 59, 60, 61, 39, 72, + 105, 42, 41, 190, 191, 43, 236, 31, 224, 260, + 228, 203, 261, 44, 105, 32, 33, 34, 160, 161, + 263, 50, 114, 264, 158, 105, 168, 75, 163, 81, + 82, 166, 167, 83, 84, 78, 114, 81, 82, 81, + 82, 83, 84, 83, 84, 45, 78, 114, 105, 147, + 46, 105, 105, 136, 137, 138, 182, 131, 20, 181, + 148, 185, 242, 152, 243, 139, 118, 119, 120, 121, + 114, 122, 123, 114, 114, 204, 124, 125, 149, 126, + 195, 250, 127, 128, 129, 182, 132, 134, 181, 196, + 217, 141, 154, 152, 202, 170, 153, 155, 156, 209, + 37, 171, 211, 195, 173, 175, 177, 176, 183, -58, + 192, 198, 196, 223, 197, 131, 227, 131, 199, 200, + 206, 220, 229, 47, 230, 235, 233, 249, 238, 105, + 239, 211, 237, 240, 244, 251, 231, 262, 216, 133, + 258, 174, 259, 219, 201, 205, 265, 253, 246, 186, + 245, 114, 267, 0, 0, 0, 249, 223, 234, 0, + 0, 227, 266, 8, 9, 10, 11, 12, 13, 14, + 15, 16, 17, 18, 19, 20, 8, 9, 10, 11, + 12, 13, 14, 15, 16, 17, 18, 19, 20, 0, + 0, 0, 0, 63, 0, 64, 0, 0, 21, 89, + 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, + 100, 21, 8, 9, 10, 11, 12, 13, 14, 15, + 16, 17, 18, 19, 20, 22, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 22, 0, + 0, 0, 0, 0, 0, 0, 0, 21, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 21, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 21, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 138, 0, 0, 0, 0, 0, 0, - 0, 0, 215, 20, 88, 89, 90, 91, 92, 93, - 94, 95, 96, 97, 98, 99, 0, 0, 0, 0, - 0, 0, 0, 0, 100, 0, 21, 101, 102, 8, - 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, - 19, 20, 0, 0, 103, 0, 0, 0, 0, 0, + 0, 140, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 21, 20, 89, 90, 91, 92, 93, + 94, 95, 96, 97, 98, 99, 100, 0, 0, 0, + 0, 0, 0, 218, 0, 101, 0, 21, 102, 103, + 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, + 18, 19, 20, 0, 0, 0, 104, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, - 0, 0, 0, 0, 21 + 0, 0, 0, 0, 0, 21 }; static const yytype_int16 yycheck[] = { - 22, 41, 46, 103, 45, 151, 39, 137, 42, 43, - 25, 39, 16, 39, 16, 16, 38, 117, 121, 31, - 123, 124, 38, 60, 46, 47, 63, 127, 50, 21, - 22, 64, 62, 25, 26, 65, 64, 39, 64, 43, - 62, 42, 58, 68, 69, 70, 74, 51, 52, 53, - 56, 21, 74, 153, 154, 25, 0, 72, 57, 103, - 60, 33, 65, 63, 194, 211, 196, 3, 58, 60, - 60, 59, 63, 117, 58, 61, 60, 29, 119, 120, - 61, 103, 61, 127, 118, 21, 22, 61, 122, 25, - 26, 125, 126, 17, 61, 117, 65, 21, 22, 17, - 16, 25, 26, 61, 61, 127, 150, 61, 61, 153, - 154, 61, 140, 61, 147, 137, 61, 59, 140, 147, - 61, 61, 61, 61, 35, 62, 59, 30, 150, 60, - 230, 153, 154, 174, 167, 62, 57, 57, 61, 61, - 21, 32, 175, 60, 59, 64, 60, 175, 189, 61, - 63, 173, 167, 61, 63, 57, 178, 190, 59, 16, - 59, 59, 59, 178, 60, 180, 61, 59, 62, 59, - 192, 60, 194, 195, 196, 190, 61, 59, 38, 62, - 58, 186, 213, 240, 228, 42, 230, 44, 45, 46, - 47, 48, 49, 50, 209, 64, 190, 54, 55, 175, - 241, 257, 173, 227, 236, 227, 140, 260, 230, -1, - -1, 209, -1, 257, 236, 150, -1, -1, 240, 259, - 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, - 14, 15, 16, 4, 5, 6, 7, 8, 9, 10, - 11, 12, 13, 14, 15, 16, -1, -1, -1, -1, - 34, -1, 36, -1, -1, 39, 17, 18, 19, 20, - 21, 22, 23, 24, 25, 26, 27, 28, 39, -1, - -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, 65, -1, -1, -1, -1, -1, -1, -1, -1, - -1, -1, -1, -1, 65, 4, 5, 6, 7, 8, - 9, 10, 11, 12, 13, 14, 15, 16, 4, 5, + 22, 41, 46, 45, 25, 153, 39, 42, 43, 39, + 139, 16, 104, 16, 21, 38, 38, 16, 25, 122, + 61, 124, 125, 64, 46, 47, 118, 63, 50, 31, + 66, 57, 65, 39, 39, 65, 59, 129, 0, 42, + 16, 63, 75, 42, 58, 44, 45, 46, 47, 48, + 49, 50, 73, 75, 66, 54, 55, 56, 33, 65, + 104, 62, 60, 155, 156, 62, 214, 43, 197, 61, + 199, 3, 64, 62, 118, 51, 52, 53, 120, 121, + 61, 66, 104, 64, 119, 129, 128, 29, 123, 21, + 22, 126, 127, 25, 26, 17, 118, 21, 22, 21, + 22, 25, 26, 25, 26, 62, 17, 129, 152, 142, + 62, 155, 156, 69, 70, 71, 149, 139, 16, 149, + 142, 59, 59, 61, 61, 35, 62, 62, 62, 62, + 152, 62, 62, 155, 156, 177, 62, 62, 30, 62, + 170, 233, 62, 62, 62, 178, 63, 60, 178, 170, + 192, 60, 58, 61, 176, 58, 63, 62, 62, 181, + 181, 21, 183, 193, 65, 60, 62, 61, 32, 62, + 61, 58, 193, 195, 64, 197, 198, 199, 64, 60, + 60, 60, 60, 38, 61, 60, 62, 231, 60, 233, + 61, 212, 216, 63, 62, 60, 63, 59, 189, 65, + 243, 142, 244, 193, 176, 178, 260, 239, 230, 152, + 230, 233, 263, -1, -1, -1, 260, 239, 212, -1, + -1, 243, 262, 4, 5, 6, 7, 8, 9, 10, + 11, 12, 13, 14, 15, 16, 4, 5, 6, 7, + 8, 9, 10, 11, 12, 13, 14, 15, 16, -1, + -1, -1, -1, 34, -1, 36, -1, -1, 39, 17, + 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, + 28, 39, 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 16, 66, -1, -1, -1, -1, + -1, -1, -1, -1, -1, -1, -1, -1, 66, -1, + -1, -1, -1, -1, -1, -1, -1, 39, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, -1, -1, -1, -1, -1, -1, -1, -1, -1, - 39, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, -1, -1, 39, -1, -1, -1, -1, -1, -1, - -1, -1, -1, 62, -1, -1, -1, -1, -1, -1, - -1, -1, 58, 16, 17, 18, 19, 20, 21, 22, - 23, 24, 25, 26, 27, 28, -1, -1, -1, -1, - -1, -1, -1, -1, 37, -1, 39, 40, 41, 4, - 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, - 15, 16, -1, -1, 57, -1, -1, -1, -1, -1, + -1, 63, -1, -1, -1, -1, -1, -1, -1, -1, + -1, -1, -1, 39, 16, 17, 18, 19, 20, 21, + 22, 23, 24, 25, 26, 27, 28, -1, -1, -1, + -1, -1, -1, 59, -1, 37, -1, 39, 40, 41, + 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, + 14, 15, 16, -1, -1, -1, 58, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, - -1, -1, -1, -1, 39 + -1, -1, -1, -1, -1, 39 }; /* YYSTOS[STATE-NUM] -- The (internal number of the) accessing symbol of state STATE-NUM. */ static const yytype_uint8 yystos[] = { - 0, 31, 67, 56, 68, 0, 57, 69, 4, 5, + 0, 31, 68, 57, 69, 0, 58, 70, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, - 16, 39, 65, 70, 89, 111, 112, 113, 114, 115, - 132, 43, 51, 52, 53, 132, 71, 112, 65, 33, - 75, 59, 61, 61, 61, 61, 61, 38, 58, 72, - 65, 42, 44, 45, 46, 47, 48, 49, 50, 54, - 55, 132, 34, 36, 76, 78, 79, 81, 84, 85, - 86, 89, 111, 114, 29, 90, 113, 17, 130, 130, - 21, 22, 25, 26, 129, 130, 131, 129, 17, 18, - 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, - 37, 40, 41, 57, 115, 119, 120, 121, 122, 123, - 124, 125, 127, 132, 132, 42, 132, 61, 61, 61, - 61, 61, 61, 61, 61, 61, 61, 61, 77, 132, - 62, 78, 59, 80, 80, 80, 80, 35, 62, 59, - 91, 92, 93, 94, 95, 114, 132, 30, 96, 119, - 60, 62, 57, 61, 61, 119, 130, 128, 129, 129, - 131, 130, 131, 131, 130, 130, 119, 57, 21, 77, - 64, 92, 59, 60, 61, 97, 98, 99, 111, 114, - 32, 116, 58, 122, 126, 127, 73, 119, 119, 60, - 87, 88, 111, 112, 63, 57, 63, 59, 94, 132, - 3, 129, 98, 59, 100, 101, 132, 110, 112, 117, - 118, 60, 63, 70, 129, 58, 88, 59, 105, 106, - 132, 77, 82, 83, 132, 77, 59, 60, 62, 102, - 61, 118, 59, 127, 71, 59, 60, 62, 107, 58, - 60, 61, 101, 132, 103, 104, 115, 119, 59, 74, - 106, 21, 25, 108, 109, 83, 129, 60, 63, 58, - 60, 63, 104, 113, 109 + 16, 39, 66, 71, 90, 112, 113, 114, 115, 116, + 133, 43, 51, 52, 53, 133, 72, 113, 66, 33, + 76, 60, 62, 62, 62, 62, 62, 38, 59, 73, + 66, 42, 44, 45, 46, 47, 48, 49, 50, 54, + 55, 56, 133, 34, 36, 77, 79, 80, 82, 85, + 86, 87, 90, 112, 115, 29, 91, 114, 17, 131, + 131, 21, 22, 25, 26, 130, 131, 132, 130, 17, + 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, + 28, 37, 40, 41, 58, 116, 120, 121, 122, 123, + 124, 125, 126, 128, 133, 133, 42, 133, 62, 62, + 62, 62, 62, 62, 62, 62, 62, 62, 62, 62, + 78, 133, 63, 79, 60, 81, 81, 81, 81, 35, + 63, 60, 92, 93, 94, 95, 96, 115, 133, 30, + 97, 120, 61, 63, 58, 62, 62, 120, 131, 129, + 130, 130, 132, 131, 132, 132, 131, 131, 130, 120, + 58, 21, 78, 65, 93, 60, 61, 62, 98, 99, + 100, 112, 115, 32, 117, 59, 123, 127, 128, 74, + 120, 120, 61, 88, 89, 112, 113, 64, 58, 64, + 60, 95, 133, 3, 130, 99, 60, 101, 102, 133, + 111, 113, 118, 119, 61, 64, 71, 130, 59, 89, + 60, 106, 107, 133, 78, 83, 84, 133, 78, 60, + 61, 63, 103, 62, 119, 60, 128, 72, 60, 61, + 63, 108, 59, 61, 62, 102, 133, 104, 105, 116, + 120, 60, 75, 107, 21, 25, 109, 110, 84, 130, + 61, 64, 59, 61, 64, 105, 114, 110 }; /* YYR1[YYN] -- Symbol number of symbol that rule YYN derives. */ static const yytype_uint8 yyr1[] = { - 0, 66, 67, 68, 69, 70, 71, 71, 73, 74, - 72, 75, 75, 75, 76, 76, 77, 78, 78, 79, - 79, 79, 79, 80, 80, 81, 82, 82, 83, 84, - 85, 86, 87, 87, 88, 89, 89, 89, 89, 89, - 89, 89, 89, 89, 89, 89, 89, 90, 90, 90, - 91, 91, 92, 92, 93, 93, 94, 94, 95, 96, - 96, 96, 97, 97, 98, 98, 99, 100, 100, 101, - 102, 102, 103, 103, 104, 105, 105, 106, 107, 107, - 108, 108, 109, 109, 110, 111, 112, 112, 113, 113, - 114, 114, 114, 114, 114, 114, 114, 114, 114, 114, - 114, 114, 114, 114, 114, 114, 114, 114, 115, 115, - 116, 116, 116, 117, 117, 118, 119, 119, 120, 121, - 121, 122, 122, 123, 123, 123, 123, 123, 123, 124, - 125, 126, 126, 127, 127, 127, 127, 127, 127, 127, - 127, 127, 127, 127, 127, 128, 128, 129, 129, 129, - 129, 130, 131, 131, 132 + 0, 67, 68, 69, 70, 71, 72, 72, 74, 75, + 73, 76, 76, 76, 77, 77, 78, 79, 79, 80, + 80, 80, 80, 81, 81, 82, 83, 83, 84, 85, + 86, 87, 88, 88, 89, 90, 90, 90, 90, 90, + 90, 90, 90, 90, 90, 90, 90, 91, 91, 91, + 92, 92, 93, 93, 94, 94, 95, 95, 96, 97, + 97, 97, 98, 98, 99, 99, 100, 101, 101, 102, + 103, 103, 104, 104, 105, 106, 106, 107, 108, 108, + 109, 109, 110, 110, 111, 112, 113, 113, 114, 114, + 115, 115, 115, 115, 115, 115, 115, 115, 115, 115, + 115, 115, 115, 115, 115, 115, 115, 115, 115, 116, + 116, 117, 117, 117, 118, 118, 119, 120, 120, 121, + 122, 122, 123, 123, 124, 124, 124, 124, 124, 124, + 125, 126, 127, 127, 128, 128, 128, 128, 128, 128, + 128, 128, 128, 128, 128, 128, 129, 129, 130, 130, + 130, 130, 131, 132, 132, 133 }; /* YYR2[YYN] -- Number of symbols on the right hand side of rule YYN. */ @@ -1106,12 +1103,12 @@ static const yytype_int8 yyr2[] = 0, 3, 1, 3, 1, 1, 3, 2, 0, 3, 1, 3, 1, 1, 1, 1, 1, 1, 0, 3, 4, 4, 4, 4, 6, 5, 5, 6, 5, 5, - 5, 5, 5, 5, 5, 5, 5, 4, 1, 1, - 0, 1, 2, 2, 3, 3, 1, 1, 0, 1, - 3, 1, 3, 1, 1, 1, 1, 1, 1, 1, - 4, 1, 3, 1, 1, 1, 1, 1, 1, 1, - 1, 1, 1, 1, 1, 1, 3, 1, 1, 1, - 1, 1, 1, 1, 1 + 5, 5, 5, 5, 5, 5, 5, 5, 4, 1, + 1, 0, 1, 2, 2, 3, 3, 1, 1, 0, + 1, 3, 1, 3, 1, 1, 1, 1, 1, 1, + 1, 4, 1, 3, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 3, 1, 1, + 1, 1, 1, 1, 1, 1 }; @@ -1849,19 +1846,19 @@ yyparse (void) switch (yyn) { case 2: /* ncdesc: NETCDF datasetid rootgroup */ -#line 240 "ncgen.y" +#line 241 "ncgen.y" {if (error_count > 0) YYABORT;} -#line 1855 "ncgeny.c" +#line 1852 "ncgeny.c" break; case 3: /* datasetid: DATASETID */ -#line 243 "ncgen.y" +#line 244 "ncgen.y" {createrootgroup(datasetname);} -#line 1861 "ncgeny.c" +#line 1858 "ncgeny.c" break; case 8: /* $@1: %empty */ -#line 262 "ncgen.y" +#line 263 "ncgen.y" { Symbol* id = (yyvsp[-1].sym); markcdf4("Group specification"); @@ -1869,29 +1866,29 @@ yyparse (void) yyerror("duplicate group declaration within parent group for %s", id->name); } -#line 1873 "ncgeny.c" +#line 1870 "ncgeny.c" break; case 9: /* $@2: %empty */ -#line 271 "ncgen.y" +#line 272 "ncgen.y" {listpop(groupstack);} -#line 1879 "ncgeny.c" +#line 1876 "ncgeny.c" break; case 12: /* typesection: TYPES */ -#line 277 "ncgen.y" +#line 278 "ncgen.y" {} -#line 1885 "ncgeny.c" +#line 1882 "ncgeny.c" break; case 13: /* typesection: TYPES typedecls */ -#line 279 "ncgen.y" +#line 280 "ncgen.y" {markcdf4("Type specification");} -#line 1891 "ncgeny.c" +#line 1888 "ncgeny.c" break; case 16: /* typename: ident */ -#line 285 "ncgen.y" +#line 286 "ncgen.y" { /* Use when defining a type */ (yyvsp[0].sym)->objectclass = NC_TYPE; if(dupobjectcheck(NC_TYPE,(yyvsp[0].sym))) @@ -1899,23 +1896,23 @@ yyparse (void) (yyvsp[0].sym)->name); listpush(typdefs,(void*)(yyvsp[0].sym)); } -#line 1903 "ncgeny.c" +#line 1900 "ncgeny.c" break; case 17: /* type_or_attr_decl: typedecl */ -#line 294 "ncgen.y" +#line 295 "ncgen.y" {} -#line 1909 "ncgeny.c" +#line 1906 "ncgeny.c" break; case 18: /* type_or_attr_decl: attrdecl ';' */ -#line 294 "ncgen.y" +#line 295 "ncgen.y" {} -#line 1915 "ncgeny.c" +#line 1912 "ncgeny.c" break; case 25: /* enumdecl: primtype ENUM typename '{' enumidlist '}' */ -#line 308 "ncgen.y" +#line 309 "ncgen.y" { int i; addtogroup((yyvsp[-3].sym)); /* sets prefix*/ @@ -1942,17 +1939,17 @@ yyparse (void) } listsetlength(stack,stackbase);/* remove stack nodes*/ } -#line 1946 "ncgeny.c" +#line 1943 "ncgeny.c" break; case 26: /* enumidlist: enumid */ -#line 337 "ncgen.y" +#line 338 "ncgen.y" {(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym));} -#line 1952 "ncgeny.c" +#line 1949 "ncgeny.c" break; case 27: /* enumidlist: enumidlist ',' enumid */ -#line 339 "ncgen.y" +#line 340 "ncgen.y" { int i; (yyval.mark)=(yyvsp[-2].mark); @@ -1967,22 +1964,22 @@ yyparse (void) } listpush(stack,(void*)(yyvsp[0].sym)); } -#line 1971 "ncgeny.c" +#line 1968 "ncgeny.c" break; case 28: /* enumid: ident '=' constint */ -#line 356 "ncgen.y" +#line 357 "ncgen.y" { (yyvsp[-2].sym)->objectclass=NC_TYPE; (yyvsp[-2].sym)->subclass=NC_ECONST; (yyvsp[-2].sym)->typ.econst=(yyvsp[0].constant); (yyval.sym)=(yyvsp[-2].sym); } -#line 1982 "ncgeny.c" +#line 1979 "ncgeny.c" break; case 29: /* opaquedecl: OPAQUE_ '(' INT_CONST ')' typename */ -#line 365 "ncgen.y" +#line 366 "ncgen.y" { vercheck(NC_OPAQUE); addtogroup((yyvsp[0].sym)); /*sets prefix*/ @@ -1992,11 +1989,11 @@ yyparse (void) (yyvsp[0].sym)->typ.size=int32_val; (yyvsp[0].sym)->typ.alignment=ncaux_class_alignment(NC_OPAQUE); } -#line 1996 "ncgeny.c" +#line 1993 "ncgeny.c" break; case 30: /* vlendecl: typeref '(' '*' ')' typename */ -#line 377 "ncgen.y" +#line 378 "ncgen.y" { Symbol* basetype = (yyvsp[-4].sym); vercheck(NC_VLEN); @@ -2008,11 +2005,11 @@ yyparse (void) (yyvsp[0].sym)->typ.size=VLENSIZE; (yyvsp[0].sym)->typ.alignment=ncaux_class_alignment(NC_VLEN); } -#line 2012 "ncgeny.c" +#line 2009 "ncgeny.c" break; case 31: /* compounddecl: COMPOUND typename '{' fields '}' */ -#line 391 "ncgen.y" +#line 392 "ncgen.y" { int i,j; vercheck(NC_COMPOUND); @@ -2042,23 +2039,23 @@ yyparse (void) } listsetlength(stack,stackbase);/* remove stack nodes*/ } -#line 2046 "ncgeny.c" +#line 2043 "ncgeny.c" break; case 32: /* fields: field ';' */ -#line 423 "ncgen.y" +#line 424 "ncgen.y" {(yyval.mark)=(yyvsp[-1].mark);} -#line 2052 "ncgeny.c" +#line 2049 "ncgeny.c" break; case 33: /* fields: fields field ';' */ -#line 424 "ncgen.y" +#line 425 "ncgen.y" {(yyval.mark)=(yyvsp[-2].mark);} -#line 2058 "ncgeny.c" +#line 2055 "ncgeny.c" break; case 34: /* field: typeref fieldlist */ -#line 428 "ncgen.y" +#line 429 "ncgen.y" { int i; (yyval.mark)=(yyvsp[0].mark); @@ -2070,107 +2067,107 @@ yyparse (void) f->typ.basetype = (yyvsp[-1].sym); } } -#line 2074 "ncgeny.c" +#line 2071 "ncgeny.c" break; case 35: /* primtype: CHAR_K */ -#line 441 "ncgen.y" +#line 442 "ncgen.y" { (yyval.sym) = primsymbols[NC_CHAR]; } -#line 2080 "ncgeny.c" +#line 2077 "ncgeny.c" break; case 36: /* primtype: BYTE_K */ -#line 442 "ncgen.y" +#line 443 "ncgen.y" { (yyval.sym) = primsymbols[NC_BYTE]; } -#line 2086 "ncgeny.c" +#line 2083 "ncgeny.c" break; case 37: /* primtype: SHORT_K */ -#line 443 "ncgen.y" +#line 444 "ncgen.y" { (yyval.sym) = primsymbols[NC_SHORT]; } -#line 2092 "ncgeny.c" +#line 2089 "ncgeny.c" break; case 38: /* primtype: INT_K */ -#line 444 "ncgen.y" +#line 445 "ncgen.y" { (yyval.sym) = primsymbols[NC_INT]; } -#line 2098 "ncgeny.c" +#line 2095 "ncgeny.c" break; case 39: /* primtype: FLOAT_K */ -#line 445 "ncgen.y" +#line 446 "ncgen.y" { (yyval.sym) = primsymbols[NC_FLOAT]; } -#line 2104 "ncgeny.c" +#line 2101 "ncgeny.c" break; case 40: /* primtype: DOUBLE_K */ -#line 446 "ncgen.y" +#line 447 "ncgen.y" { (yyval.sym) = primsymbols[NC_DOUBLE]; } -#line 2110 "ncgeny.c" +#line 2107 "ncgeny.c" break; case 41: /* primtype: UBYTE_K */ -#line 447 "ncgen.y" +#line 448 "ncgen.y" { vercheck(NC_UBYTE); (yyval.sym) = primsymbols[NC_UBYTE]; } -#line 2116 "ncgeny.c" +#line 2113 "ncgeny.c" break; case 42: /* primtype: USHORT_K */ -#line 448 "ncgen.y" +#line 449 "ncgen.y" { vercheck(NC_USHORT); (yyval.sym) = primsymbols[NC_USHORT]; } -#line 2122 "ncgeny.c" +#line 2119 "ncgeny.c" break; case 43: /* primtype: UINT_K */ -#line 449 "ncgen.y" +#line 450 "ncgen.y" { vercheck(NC_UINT); (yyval.sym) = primsymbols[NC_UINT]; } -#line 2128 "ncgeny.c" +#line 2125 "ncgeny.c" break; case 44: /* primtype: INT64_K */ -#line 450 "ncgen.y" +#line 451 "ncgen.y" { vercheck(NC_INT64); (yyval.sym) = primsymbols[NC_INT64]; } -#line 2134 "ncgeny.c" +#line 2131 "ncgeny.c" break; case 45: /* primtype: UINT64_K */ -#line 451 "ncgen.y" +#line 452 "ncgen.y" { vercheck(NC_UINT64); (yyval.sym) = primsymbols[NC_UINT64]; } -#line 2140 "ncgeny.c" +#line 2137 "ncgeny.c" break; case 46: /* primtype: STRING_K */ -#line 452 "ncgen.y" +#line 453 "ncgen.y" { vercheck(NC_STRING); (yyval.sym) = primsymbols[NC_STRING]; } -#line 2146 "ncgeny.c" +#line 2143 "ncgeny.c" break; case 48: /* dimsection: DIMENSIONS */ -#line 456 "ncgen.y" +#line 457 "ncgen.y" {} -#line 2152 "ncgeny.c" +#line 2149 "ncgeny.c" break; case 49: /* dimsection: DIMENSIONS dimdecls */ -#line 457 "ncgen.y" +#line 458 "ncgen.y" {} -#line 2158 "ncgeny.c" +#line 2155 "ncgeny.c" break; case 52: /* dim_or_attr_decl: dimdeclist */ -#line 464 "ncgen.y" +#line 465 "ncgen.y" {} -#line 2164 "ncgeny.c" +#line 2161 "ncgeny.c" break; case 53: /* dim_or_attr_decl: attrdecl */ -#line 464 "ncgen.y" +#line 465 "ncgen.y" {} -#line 2170 "ncgeny.c" +#line 2167 "ncgeny.c" break; case 56: /* dimdecl: dimd '=' constint */ -#line 472 "ncgen.y" +#line 473 "ncgen.y" { (yyvsp[-2].sym)->dim.declsize = (size_t)extractint((yyvsp[0].constant)); #ifdef GENDEBUG1 @@ -2178,11 +2175,11 @@ fprintf(stderr,"dimension: %s = %llu\n",(yyvsp[-2].sym)->name,(unsigned long lon #endif reclaimconstant((yyvsp[0].constant)); } -#line 2182 "ncgeny.c" +#line 2179 "ncgeny.c" break; case 57: /* dimdecl: dimd '=' NC_UNLIMITED_K */ -#line 480 "ncgen.y" +#line 481 "ncgen.y" { (yyvsp[-2].sym)->dim.declsize = NC_UNLIMITED; (yyvsp[-2].sym)->dim.isunlimited = 1; @@ -2190,11 +2187,11 @@ fprintf(stderr,"dimension: %s = %llu\n",(yyvsp[-2].sym)->name,(unsigned long lon fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); #endif } -#line 2194 "ncgeny.c" +#line 2191 "ncgeny.c" break; case 58: /* dimd: ident */ -#line 490 "ncgen.y" +#line 491 "ncgen.y" { (yyvsp[0].sym)->objectclass=NC_DIM; if(dupobjectcheck(NC_DIM,(yyvsp[0].sym))) @@ -2204,35 +2201,35 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); (yyval.sym)=(yyvsp[0].sym); listpush(dimdefs,(void*)(yyvsp[0].sym)); } -#line 2208 "ncgeny.c" +#line 2205 "ncgeny.c" break; case 60: /* vasection: VARIABLES */ -#line 502 "ncgen.y" +#line 503 "ncgen.y" {} -#line 2214 "ncgeny.c" +#line 2211 "ncgeny.c" break; case 61: /* vasection: VARIABLES vadecls */ -#line 503 "ncgen.y" +#line 504 "ncgen.y" {} -#line 2220 "ncgeny.c" +#line 2217 "ncgeny.c" break; case 64: /* vadecl_or_attr: vardecl */ -#line 510 "ncgen.y" +#line 511 "ncgen.y" {} -#line 2226 "ncgeny.c" +#line 2223 "ncgeny.c" break; case 65: /* vadecl_or_attr: attrdecl */ -#line 510 "ncgen.y" +#line 511 "ncgen.y" {} -#line 2232 "ncgeny.c" +#line 2229 "ncgeny.c" break; case 66: /* vardecl: typeref varlist */ -#line 513 "ncgen.y" +#line 514 "ncgen.y" { int i; stackbase=(yyvsp[0].mark); @@ -2252,25 +2249,25 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); } listsetlength(stack,stackbase);/* remove stack nodes*/ } -#line 2256 "ncgeny.c" +#line 2253 "ncgeny.c" break; case 67: /* varlist: varspec */ -#line 535 "ncgen.y" +#line 536 "ncgen.y" {(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym)); } -#line 2264 "ncgeny.c" +#line 2261 "ncgeny.c" break; case 68: /* varlist: varlist ',' varspec */ -#line 539 "ncgen.y" +#line 540 "ncgen.y" {(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));} -#line 2270 "ncgeny.c" +#line 2267 "ncgeny.c" break; case 69: /* varspec: ident dimspec */ -#line 543 "ncgen.y" +#line 544 "ncgen.y" { int i; Dimset dimset; @@ -2297,35 +2294,35 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); listsetlength(stack,stackbase);/* remove stack nodes*/ (yyval.sym) = var; } -#line 2301 "ncgeny.c" +#line 2298 "ncgeny.c" break; case 70: /* dimspec: %empty */ -#line 571 "ncgen.y" +#line 572 "ncgen.y" {(yyval.mark)=listlength(stack);} -#line 2307 "ncgeny.c" +#line 2304 "ncgeny.c" break; case 71: /* dimspec: '(' dimlist ')' */ -#line 572 "ncgen.y" +#line 573 "ncgen.y" {(yyval.mark)=(yyvsp[-1].mark);} -#line 2313 "ncgeny.c" +#line 2310 "ncgeny.c" break; case 72: /* dimlist: dimref */ -#line 575 "ncgen.y" +#line 576 "ncgen.y" {(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym));} -#line 2319 "ncgeny.c" +#line 2316 "ncgeny.c" break; case 73: /* dimlist: dimlist ',' dimref */ -#line 577 "ncgen.y" +#line 578 "ncgen.y" {(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));} -#line 2325 "ncgeny.c" +#line 2322 "ncgeny.c" break; case 74: /* dimref: path */ -#line 581 "ncgen.y" +#line 582 "ncgen.y" {Symbol* dimsym = (yyvsp[0].sym); dimsym->objectclass = NC_DIM; /* Find the actual dimension*/ @@ -2336,25 +2333,25 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); } (yyval.sym)=dimsym; } -#line 2340 "ncgeny.c" +#line 2337 "ncgeny.c" break; case 75: /* fieldlist: fieldspec */ -#line 595 "ncgen.y" +#line 596 "ncgen.y" {(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym)); } -#line 2348 "ncgeny.c" +#line 2345 "ncgeny.c" break; case 76: /* fieldlist: fieldlist ',' fieldspec */ -#line 599 "ncgen.y" +#line 600 "ncgen.y" {(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));} -#line 2354 "ncgeny.c" +#line 2351 "ncgeny.c" break; case 77: /* fieldspec: ident fielddimspec */ -#line 604 "ncgen.y" +#line 605 "ncgen.y" { int i; Dimset dimset; @@ -2381,35 +2378,35 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); listsetlength(stack,stackbase);/* remove stack nodes*/ (yyval.sym) = (yyvsp[-1].sym); } -#line 2385 "ncgeny.c" +#line 2382 "ncgeny.c" break; case 78: /* fielddimspec: %empty */ -#line 632 "ncgen.y" +#line 633 "ncgen.y" {(yyval.mark)=listlength(stack);} -#line 2391 "ncgeny.c" +#line 2388 "ncgeny.c" break; case 79: /* fielddimspec: '(' fielddimlist ')' */ -#line 633 "ncgen.y" +#line 634 "ncgen.y" {(yyval.mark)=(yyvsp[-1].mark);} -#line 2397 "ncgeny.c" +#line 2394 "ncgeny.c" break; case 80: /* fielddimlist: fielddim */ -#line 637 "ncgen.y" +#line 638 "ncgen.y" {(yyval.mark)=listlength(stack); listpush(stack,(void*)(yyvsp[0].sym));} -#line 2403 "ncgeny.c" +#line 2400 "ncgeny.c" break; case 81: /* fielddimlist: fielddimlist ',' fielddim */ -#line 639 "ncgen.y" +#line 640 "ncgen.y" {(yyval.mark)=(yyvsp[-2].mark); listpush(stack,(void*)(yyvsp[0].sym));} -#line 2409 "ncgeny.c" +#line 2406 "ncgeny.c" break; case 82: /* fielddim: UINT_CONST */ -#line 644 "ncgen.y" +#line 645 "ncgen.y" { /* Anonymous integer dimension. Can only occur in type definitions*/ char anon[32]; @@ -2419,11 +2416,11 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); (yyval.sym)->dim.isconstant = 1; (yyval.sym)->dim.declsize = uint32_val; } -#line 2423 "ncgeny.c" +#line 2420 "ncgeny.c" break; case 83: /* fielddim: INT_CONST */ -#line 654 "ncgen.y" +#line 655 "ncgen.y" { /* Anonymous integer dimension. Can only occur in type definitions*/ char anon[32]; @@ -2437,11 +2434,11 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); (yyval.sym)->dim.isconstant = 1; (yyval.sym)->dim.declsize = int32_val; } -#line 2441 "ncgeny.c" +#line 2438 "ncgeny.c" break; case 84: /* varref: ambiguous_ref */ -#line 674 "ncgen.y" +#line 675 "ncgen.y" {Symbol* vsym = (yyvsp[0].sym); if(vsym->objectclass != NC_VAR) { derror("Undefined or forward referenced variable: %s",vsym->name); @@ -2449,11 +2446,11 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); } (yyval.sym)=vsym; } -#line 2453 "ncgeny.c" +#line 2450 "ncgeny.c" break; case 85: /* typeref: ambiguous_ref */ -#line 685 "ncgen.y" +#line 686 "ncgen.y" {Symbol* tsym = (yyvsp[0].sym); if(tsym->objectclass != NC_TYPE) { derror("Undefined or forward referenced type: %s",tsym->name); @@ -2461,11 +2458,11 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); } (yyval.sym)=tsym; } -#line 2465 "ncgeny.c" +#line 2462 "ncgeny.c" break; case 86: /* ambiguous_ref: path */ -#line 696 "ncgen.y" +#line 697 "ncgen.y" {Symbol* tvsym = (yyvsp[0].sym); Symbol* sym; /* disambiguate*/ tvsym->objectclass = NC_VAR; @@ -2484,53 +2481,53 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); } (yyval.sym)=tvsym; } -#line 2488 "ncgeny.c" +#line 2485 "ncgeny.c" break; case 87: /* ambiguous_ref: primtype */ -#line 714 "ncgen.y" +#line 715 "ncgen.y" {(yyval.sym)=(yyvsp[0].sym);} -#line 2494 "ncgeny.c" +#line 2491 "ncgeny.c" break; case 88: /* attrdecllist: %empty */ -#line 721 "ncgen.y" +#line 722 "ncgen.y" {} -#line 2500 "ncgeny.c" +#line 2497 "ncgeny.c" break; case 89: /* attrdecllist: attrdecl ';' attrdecllist */ -#line 721 "ncgen.y" +#line 722 "ncgen.y" {} -#line 2506 "ncgeny.c" +#line 2503 "ncgeny.c" break; case 90: /* attrdecl: ':' _NCPROPS '=' conststring */ -#line 725 "ncgen.y" +#line 726 "ncgen.y" {(yyval.sym) = makespecial(_NCPROPS_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2512 "ncgeny.c" +#line 2509 "ncgeny.c" break; case 91: /* attrdecl: ':' _ISNETCDF4 '=' constbool */ -#line 727 "ncgen.y" +#line 728 "ncgen.y" {(yyval.sym) = makespecial(_ISNETCDF4_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2518 "ncgeny.c" +#line 2515 "ncgeny.c" break; case 92: /* attrdecl: ':' _SUPERBLOCK '=' constint */ -#line 729 "ncgen.y" +#line 730 "ncgen.y" {(yyval.sym) = makespecial(_SUPERBLOCK_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2524 "ncgeny.c" +#line 2521 "ncgeny.c" break; case 93: /* attrdecl: ':' ident '=' datalist */ -#line 731 "ncgen.y" +#line 732 "ncgen.y" { (yyval.sym)=makeattribute((yyvsp[-2].sym),NULL,NULL,(yyvsp[0].datalist),ATTRGLOBAL);} -#line 2530 "ncgeny.c" +#line 2527 "ncgeny.c" break; case 94: /* attrdecl: typeref ambiguous_ref ':' ident '=' datalist */ -#line 733 "ncgen.y" +#line 734 "ncgen.y" {Symbol* tsym = (yyvsp[-5].sym); Symbol* vsym = (yyvsp[-4].sym); Symbol* asym = (yyvsp[-2].sym); if(vsym->objectclass == NC_VAR) { (yyval.sym)=makeattribute(asym,vsym,tsym,(yyvsp[0].datalist),ATTRVAR); @@ -2539,11 +2536,11 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); YYABORT; } } -#line 2543 "ncgeny.c" +#line 2540 "ncgeny.c" break; case 95: /* attrdecl: ambiguous_ref ':' ident '=' datalist */ -#line 742 "ncgen.y" +#line 743 "ncgen.y" {Symbol* sym = (yyvsp[-4].sym); Symbol* asym = (yyvsp[-2].sym); if(sym->objectclass == NC_VAR) { (yyval.sym)=makeattribute(asym,sym,NULL,(yyvsp[0].datalist),ATTRVAR); @@ -2554,351 +2551,357 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); YYABORT; } } -#line 2558 "ncgeny.c" +#line 2555 "ncgeny.c" break; case 96: /* attrdecl: ambiguous_ref ':' _FILLVALUE '=' datalist */ -#line 753 "ncgen.y" +#line 754 "ncgen.y" {(yyval.sym) = makespecial(_FILLVALUE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].datalist),ISLIST);} -#line 2564 "ncgeny.c" +#line 2561 "ncgeny.c" break; case 97: /* attrdecl: typeref ambiguous_ref ':' _FILLVALUE '=' datalist */ -#line 755 "ncgen.y" +#line 756 "ncgen.y" {(yyval.sym) = makespecial(_FILLVALUE_FLAG,(yyvsp[-4].sym),(yyvsp[-5].sym),(void*)(yyvsp[0].datalist),ISLIST);} -#line 2570 "ncgeny.c" +#line 2567 "ncgeny.c" break; case 98: /* attrdecl: ambiguous_ref ':' _STORAGE '=' conststring */ -#line 757 "ncgen.y" +#line 758 "ncgen.y" {(yyval.sym) = makespecial(_STORAGE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2576 "ncgeny.c" +#line 2573 "ncgeny.c" break; case 99: /* attrdecl: ambiguous_ref ':' _CHUNKSIZES '=' intlist */ -#line 759 "ncgen.y" +#line 760 "ncgen.y" {(yyval.sym) = makespecial(_CHUNKSIZES_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].datalist),ISLIST);} -#line 2582 "ncgeny.c" +#line 2579 "ncgeny.c" break; case 100: /* attrdecl: ambiguous_ref ':' _FLETCHER32 '=' constbool */ -#line 761 "ncgen.y" +#line 762 "ncgen.y" {(yyval.sym) = makespecial(_FLETCHER32_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2588 "ncgeny.c" +#line 2585 "ncgeny.c" break; case 101: /* attrdecl: ambiguous_ref ':' _DEFLATELEVEL '=' constint */ -#line 763 "ncgen.y" +#line 764 "ncgen.y" {(yyval.sym) = makespecial(_DEFLATE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2594 "ncgeny.c" +#line 2591 "ncgeny.c" break; case 102: /* attrdecl: ambiguous_ref ':' _SHUFFLE '=' constbool */ -#line 765 "ncgen.y" +#line 766 "ncgen.y" {(yyval.sym) = makespecial(_SHUFFLE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2600 "ncgeny.c" +#line 2597 "ncgeny.c" break; case 103: /* attrdecl: ambiguous_ref ':' _ENDIANNESS '=' conststring */ -#line 767 "ncgen.y" +#line 768 "ncgen.y" {(yyval.sym) = makespecial(_ENDIAN_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2606 "ncgeny.c" +#line 2603 "ncgeny.c" break; case 104: /* attrdecl: ambiguous_ref ':' _FILTER '=' conststring */ -#line 769 "ncgen.y" +#line 770 "ncgen.y" {(yyval.sym) = makespecial(_FILTER_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2612 "ncgeny.c" +#line 2609 "ncgeny.c" break; case 105: /* attrdecl: ambiguous_ref ':' _CODECS '=' conststring */ -#line 771 "ncgen.y" +#line 772 "ncgen.y" {(yyval.sym) = makespecial(_CODECS_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2618 "ncgeny.c" +#line 2615 "ncgeny.c" break; - case 106: /* attrdecl: ambiguous_ref ':' _NOFILL '=' constbool */ -#line 773 "ncgen.y" + case 106: /* attrdecl: ambiguous_ref ':' _QUANTIZE '=' constint */ +#line 774 "ncgen.y" + {(yyval.sym) = makespecial(_QUANTIZE_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} +#line 2621 "ncgeny.c" + break; + + case 107: /* attrdecl: ambiguous_ref ':' _NOFILL '=' constbool */ +#line 776 "ncgen.y" {(yyval.sym) = makespecial(_NOFILL_FLAG,(yyvsp[-4].sym),NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2624 "ncgeny.c" +#line 2627 "ncgeny.c" break; - case 107: /* attrdecl: ':' _FORMAT '=' conststring */ -#line 775 "ncgen.y" + case 108: /* attrdecl: ':' _FORMAT '=' conststring */ +#line 778 "ncgen.y" {(yyval.sym) = makespecial(_FORMAT_FLAG,NULL,NULL,(void*)(yyvsp[0].constant),ISCONST);} -#line 2630 "ncgeny.c" +#line 2633 "ncgeny.c" break; - case 108: /* path: ident */ -#line 780 "ncgen.y" + case 109: /* path: ident */ +#line 783 "ncgen.y" { (yyval.sym)=(yyvsp[0].sym); (yyvsp[0].sym)->ref.is_ref=1; (yyvsp[0].sym)->is_prefixed=0; setpathcurrent((yyvsp[0].sym)); } -#line 2641 "ncgeny.c" +#line 2644 "ncgeny.c" break; - case 109: /* path: PATH */ -#line 787 "ncgen.y" + case 110: /* path: PATH */ +#line 790 "ncgen.y" { (yyval.sym)=(yyvsp[0].sym); (yyvsp[0].sym)->ref.is_ref=1; (yyvsp[0].sym)->is_prefixed=1; /* path is set in ncgen.l*/ } -#line 2652 "ncgeny.c" +#line 2655 "ncgeny.c" break; - case 111: /* datasection: DATA */ -#line 796 "ncgen.y" + case 112: /* datasection: DATA */ +#line 799 "ncgen.y" {} -#line 2658 "ncgeny.c" +#line 2661 "ncgeny.c" break; - case 112: /* datasection: DATA datadecls */ -#line 797 "ncgen.y" + case 113: /* datasection: DATA datadecls */ +#line 800 "ncgen.y" {} -#line 2664 "ncgeny.c" +#line 2667 "ncgeny.c" break; - case 115: /* datadecl: varref '=' datalist */ -#line 805 "ncgen.y" + case 116: /* datadecl: varref '=' datalist */ +#line 808 "ncgen.y" {(yyvsp[-2].sym)->data = (yyvsp[0].datalist);} -#line 2670 "ncgeny.c" +#line 2673 "ncgeny.c" break; - case 116: /* datalist: datalist0 */ -#line 808 "ncgen.y" + case 117: /* datalist: datalist0 */ +#line 811 "ncgen.y" {(yyval.datalist) = (yyvsp[0].datalist);} -#line 2676 "ncgeny.c" +#line 2679 "ncgeny.c" break; - case 117: /* datalist: datalist1 */ -#line 809 "ncgen.y" + case 118: /* datalist: datalist1 */ +#line 812 "ncgen.y" {(yyval.datalist) = (yyvsp[0].datalist);} -#line 2682 "ncgeny.c" +#line 2685 "ncgeny.c" break; - case 118: /* datalist0: %empty */ -#line 813 "ncgen.y" + case 119: /* datalist0: %empty */ +#line 816 "ncgen.y" {(yyval.datalist) = builddatalist(0);} -#line 2688 "ncgeny.c" +#line 2691 "ncgeny.c" break; - case 119: /* datalist1: dataitem */ -#line 817 "ncgen.y" + case 120: /* datalist1: dataitem */ +#line 820 "ncgen.y" {(yyval.datalist) = const2list((yyvsp[0].constant));} -#line 2694 "ncgeny.c" +#line 2697 "ncgeny.c" break; - case 120: /* datalist1: datalist ',' dataitem */ -#line 819 "ncgen.y" + case 121: /* datalist1: datalist ',' dataitem */ +#line 822 "ncgen.y" {dlappend((yyvsp[-2].datalist),((yyvsp[0].constant))); (yyval.datalist)=(yyvsp[-2].datalist); } -#line 2700 "ncgeny.c" +#line 2703 "ncgeny.c" break; - case 121: /* dataitem: constdata */ -#line 823 "ncgen.y" + case 122: /* dataitem: constdata */ +#line 826 "ncgen.y" {(yyval.constant)=(yyvsp[0].constant);} -#line 2706 "ncgeny.c" +#line 2709 "ncgeny.c" break; - case 122: /* dataitem: '{' datalist '}' */ -#line 824 "ncgen.y" + case 123: /* dataitem: '{' datalist '}' */ +#line 827 "ncgen.y" {(yyval.constant)=builddatasublist((yyvsp[-1].datalist));} -#line 2712 "ncgeny.c" +#line 2715 "ncgeny.c" break; - case 123: /* constdata: simpleconstant */ -#line 828 "ncgen.y" + case 124: /* constdata: simpleconstant */ +#line 831 "ncgen.y" {(yyval.constant)=(yyvsp[0].constant);} -#line 2718 "ncgeny.c" +#line 2721 "ncgeny.c" break; - case 124: /* constdata: OPAQUESTRING */ -#line 829 "ncgen.y" + case 125: /* constdata: OPAQUESTRING */ +#line 832 "ncgen.y" {(yyval.constant)=makeconstdata(NC_OPAQUE);} -#line 2724 "ncgeny.c" +#line 2727 "ncgeny.c" break; - case 125: /* constdata: FILLMARKER */ -#line 830 "ncgen.y" + case 126: /* constdata: FILLMARKER */ +#line 833 "ncgen.y" {(yyval.constant)=makeconstdata(NC_FILLVALUE);} -#line 2730 "ncgeny.c" +#line 2733 "ncgeny.c" break; - case 126: /* constdata: NIL */ -#line 831 "ncgen.y" + case 127: /* constdata: NIL */ +#line 834 "ncgen.y" {(yyval.constant)=makeconstdata(NC_NIL);} -#line 2736 "ncgeny.c" +#line 2739 "ncgeny.c" break; - case 127: /* constdata: econstref */ -#line 832 "ncgen.y" + case 128: /* constdata: econstref */ +#line 835 "ncgen.y" {(yyval.constant)=(yyvsp[0].constant);} -#line 2742 "ncgeny.c" +#line 2745 "ncgeny.c" break; - case 129: /* econstref: path */ -#line 837 "ncgen.y" + case 130: /* econstref: path */ +#line 840 "ncgen.y" {(yyval.constant) = makeenumconstref((yyvsp[0].sym));} -#line 2748 "ncgeny.c" +#line 2751 "ncgeny.c" break; - case 130: /* function: ident '(' arglist ')' */ -#line 841 "ncgen.y" + case 131: /* function: ident '(' arglist ')' */ +#line 844 "ncgen.y" {(yyval.constant)=evaluate((yyvsp[-3].sym),(yyvsp[-1].datalist));} -#line 2754 "ncgeny.c" +#line 2757 "ncgeny.c" break; - case 131: /* arglist: simpleconstant */ -#line 846 "ncgen.y" + case 132: /* arglist: simpleconstant */ +#line 849 "ncgen.y" {(yyval.datalist) = const2list((yyvsp[0].constant));} -#line 2760 "ncgeny.c" +#line 2763 "ncgeny.c" break; - case 132: /* arglist: arglist ',' simpleconstant */ -#line 848 "ncgen.y" + case 133: /* arglist: arglist ',' simpleconstant */ +#line 851 "ncgen.y" {dlappend((yyvsp[-2].datalist),((yyvsp[0].constant))); (yyval.datalist)=(yyvsp[-2].datalist);} -#line 2766 "ncgeny.c" +#line 2769 "ncgeny.c" break; - case 133: /* simpleconstant: CHAR_CONST */ -#line 852 "ncgen.y" + case 134: /* simpleconstant: CHAR_CONST */ +#line 855 "ncgen.y" {(yyval.constant)=makeconstdata(NC_CHAR);} -#line 2772 "ncgeny.c" +#line 2775 "ncgeny.c" break; - case 134: /* simpleconstant: BYTE_CONST */ -#line 853 "ncgen.y" + case 135: /* simpleconstant: BYTE_CONST */ +#line 856 "ncgen.y" {(yyval.constant)=makeconstdata(NC_BYTE);} -#line 2778 "ncgeny.c" +#line 2781 "ncgeny.c" break; - case 135: /* simpleconstant: SHORT_CONST */ -#line 854 "ncgen.y" + case 136: /* simpleconstant: SHORT_CONST */ +#line 857 "ncgen.y" {(yyval.constant)=makeconstdata(NC_SHORT);} -#line 2784 "ncgeny.c" +#line 2787 "ncgeny.c" break; - case 136: /* simpleconstant: INT_CONST */ -#line 855 "ncgen.y" + case 137: /* simpleconstant: INT_CONST */ +#line 858 "ncgen.y" {(yyval.constant)=makeconstdata(NC_INT);} -#line 2790 "ncgeny.c" +#line 2793 "ncgeny.c" break; - case 137: /* simpleconstant: INT64_CONST */ -#line 856 "ncgen.y" + case 138: /* simpleconstant: INT64_CONST */ +#line 859 "ncgen.y" {(yyval.constant)=makeconstdata(NC_INT64);} -#line 2796 "ncgeny.c" +#line 2799 "ncgeny.c" break; - case 138: /* simpleconstant: UBYTE_CONST */ -#line 857 "ncgen.y" + case 139: /* simpleconstant: UBYTE_CONST */ +#line 860 "ncgen.y" {(yyval.constant)=makeconstdata(NC_UBYTE);} -#line 2802 "ncgeny.c" +#line 2805 "ncgeny.c" break; - case 139: /* simpleconstant: USHORT_CONST */ -#line 858 "ncgen.y" + case 140: /* simpleconstant: USHORT_CONST */ +#line 861 "ncgen.y" {(yyval.constant)=makeconstdata(NC_USHORT);} -#line 2808 "ncgeny.c" +#line 2811 "ncgeny.c" break; - case 140: /* simpleconstant: UINT_CONST */ -#line 859 "ncgen.y" + case 141: /* simpleconstant: UINT_CONST */ +#line 862 "ncgen.y" {(yyval.constant)=makeconstdata(NC_UINT);} -#line 2814 "ncgeny.c" +#line 2817 "ncgeny.c" break; - case 141: /* simpleconstant: UINT64_CONST */ -#line 860 "ncgen.y" + case 142: /* simpleconstant: UINT64_CONST */ +#line 863 "ncgen.y" {(yyval.constant)=makeconstdata(NC_UINT64);} -#line 2820 "ncgeny.c" +#line 2823 "ncgeny.c" break; - case 142: /* simpleconstant: FLOAT_CONST */ -#line 861 "ncgen.y" + case 143: /* simpleconstant: FLOAT_CONST */ +#line 864 "ncgen.y" {(yyval.constant)=makeconstdata(NC_FLOAT);} -#line 2826 "ncgeny.c" +#line 2829 "ncgeny.c" break; - case 143: /* simpleconstant: DOUBLE_CONST */ -#line 862 "ncgen.y" + case 144: /* simpleconstant: DOUBLE_CONST */ +#line 865 "ncgen.y" {(yyval.constant)=makeconstdata(NC_DOUBLE);} -#line 2832 "ncgeny.c" +#line 2835 "ncgeny.c" break; - case 144: /* simpleconstant: TERMSTRING */ -#line 863 "ncgen.y" + case 145: /* simpleconstant: TERMSTRING */ +#line 866 "ncgen.y" {(yyval.constant)=makeconstdata(NC_STRING);} -#line 2838 "ncgeny.c" +#line 2841 "ncgeny.c" break; - case 145: /* intlist: constint */ -#line 867 "ncgen.y" + case 146: /* intlist: constint */ +#line 870 "ncgen.y" {(yyval.datalist) = const2list((yyvsp[0].constant));} -#line 2844 "ncgeny.c" +#line 2847 "ncgeny.c" break; - case 146: /* intlist: intlist ',' constint */ -#line 868 "ncgen.y" + case 147: /* intlist: intlist ',' constint */ +#line 871 "ncgen.y" {(yyval.datalist)=(yyvsp[-2].datalist); dlappend((yyvsp[-2].datalist),((yyvsp[0].constant)));} -#line 2850 "ncgeny.c" +#line 2853 "ncgeny.c" break; - case 147: /* constint: INT_CONST */ -#line 873 "ncgen.y" + case 148: /* constint: INT_CONST */ +#line 876 "ncgen.y" {(yyval.constant)=makeconstdata(NC_INT);} -#line 2856 "ncgeny.c" +#line 2859 "ncgeny.c" break; - case 148: /* constint: UINT_CONST */ -#line 875 "ncgen.y" + case 149: /* constint: UINT_CONST */ +#line 878 "ncgen.y" {(yyval.constant)=makeconstdata(NC_UINT);} -#line 2862 "ncgeny.c" +#line 2865 "ncgeny.c" break; - case 149: /* constint: INT64_CONST */ -#line 877 "ncgen.y" + case 150: /* constint: INT64_CONST */ +#line 880 "ncgen.y" {(yyval.constant)=makeconstdata(NC_INT64);} -#line 2868 "ncgeny.c" +#line 2871 "ncgeny.c" break; - case 150: /* constint: UINT64_CONST */ -#line 879 "ncgen.y" + case 151: /* constint: UINT64_CONST */ +#line 882 "ncgen.y" {(yyval.constant)=makeconstdata(NC_UINT64);} -#line 2874 "ncgeny.c" +#line 2877 "ncgeny.c" break; - case 151: /* conststring: TERMSTRING */ -#line 883 "ncgen.y" + case 152: /* conststring: TERMSTRING */ +#line 886 "ncgen.y" {(yyval.constant)=makeconstdata(NC_STRING);} -#line 2880 "ncgeny.c" +#line 2883 "ncgeny.c" break; - case 152: /* constbool: conststring */ -#line 887 "ncgen.y" + case 153: /* constbool: conststring */ +#line 890 "ncgen.y" {(yyval.constant)=(yyvsp[0].constant);} -#line 2886 "ncgeny.c" +#line 2889 "ncgeny.c" break; - case 153: /* constbool: constint */ -#line 888 "ncgen.y" + case 154: /* constbool: constint */ +#line 891 "ncgen.y" {(yyval.constant)=(yyvsp[0].constant);} -#line 2892 "ncgeny.c" +#line 2895 "ncgeny.c" break; - case 154: /* ident: IDENT */ -#line 894 "ncgen.y" + case 155: /* ident: IDENT */ +#line 897 "ncgen.y" {(yyval.sym)=(yyvsp[0].sym);} -#line 2898 "ncgeny.c" +#line 2901 "ncgeny.c" break; -#line 2902 "ncgeny.c" +#line 2905 "ncgeny.c" default: break; } @@ -3123,7 +3126,7 @@ fprintf(stderr,"dimension: %s = UNLIMITED\n",(yyvsp[-2].sym)->name); return yyresult; } -#line 897 "ncgen.y" +#line 900 "ncgen.y" #ifndef NO_STDARG @@ -3460,6 +3463,7 @@ makespecial(int tag, Symbol* vsym, Symbol* tsym, void* data, int isconst) break; case _SUPERBLOCK_FLAG: case _DEFLATE_FLAG: + case _QUANTIZE_FLAG: tmp = nullconst(); tmp->nctype = NC_INT; convert1(con,tmp); @@ -3554,6 +3558,11 @@ makespecial(int tag, Symbol* vsym, Symbol* tsym, void* data, int isconst) special->_DeflateLevel = idata; special->flags |= _DEFLATE_FLAG; break; + case _QUANTIZE_FLAG: + special->_Quantizer = NC_QUANTIZE_BITGROOM; + special->_NSD = idata; + special->flags |= _QUANTIZE_FLAG; + break; case _SHUFFLE_FLAG: special->_Shuffle = tf; special->flags |= _SHUFFLE_FLAG; diff --git a/ncgen/ncgeny.h b/ncgen/ncgeny.h index 8481e1292d..1587491a2a 100644 --- a/ncgen/ncgeny.h +++ b/ncgen/ncgeny.h @@ -107,7 +107,8 @@ extern int ncgdebug; _SUPERBLOCK = 308, /* _SUPERBLOCK */ _FILTER = 309, /* _FILTER */ _CODECS = 310, /* _CODECS */ - DATASETID = 311 /* DATASETID */ + _QUANTIZE = 311, /* _QUANTIZE */ + DATASETID = 312 /* DATASETID */ }; typedef enum yytokentype yytoken_kind_t; #endif @@ -125,7 +126,7 @@ int nctype; /* for tracking attribute list type*/ Datalist* datalist; NCConstant* constant; -#line 129 "ncgeny.h" +#line 130 "ncgeny.h" }; typedef union YYSTYPE YYSTYPE; diff --git a/nczarr_test/CMakeLists.txt b/nczarr_test/CMakeLists.txt index 3d8662d7fa..312fba9153 100644 --- a/nczarr_test/CMakeLists.txt +++ b/nczarr_test/CMakeLists.txt @@ -92,6 +92,9 @@ IF(ENABLE_TESTS) add_sh_test(nczarr_test run_misc) add_sh_test(nczarr_test run_nczarr_fill) + BUILD_BIN_TEST(test_quantize ${TSTCOMMONSRC}) + add_sh_test(nczarr_test run_quantize) + if(ENABLE_NCZARR_S3) add_sh_test(nczarr_test run_s3_cleanup) ENDIF() diff --git a/nczarr_test/Makefile.am b/nczarr_test/Makefile.am index d2278ae17c..b69b4ebabf 100644 --- a/nczarr_test/Makefile.am +++ b/nczarr_test/Makefile.am @@ -35,7 +35,7 @@ ut_projections_SOURCES = ut_projections.c ${commonsrc} ut_chunking_SOURCES = ut_chunking.c ${commonsrc} tst_fillonlyz_SOURCES = tst_fillonlyz.c ${tstcommonsrc} -check_PROGRAMS += tst_zchunks tst_zchunks2 tst_zchunks3 tst_fillonlyz +check_PROGRAMS += tst_zchunks tst_zchunks2 tst_zchunks3 tst_fillonlyz test_quantize TESTS += run_ut_map.sh TESTS += run_ut_mapapi.sh @@ -55,6 +55,8 @@ check_PROGRAMS += tst_chunkcases tst_chunkcases_SOURCES = tst_chunkcases.c ${tstcommonsrc} TESTS += run_chunkcases.sh +TESTS += run_quantize.sh + TESTS += run_purezarr.sh TESTS += run_interop.sh TESTS += run_misc.sh @@ -126,7 +128,7 @@ run_ut_map.sh run_ut_mapapi.sh run_ut_misc.sh run_ut_chunk.sh run_ncgen4.sh \ run_nccopyz.sh run_fillonlyz.sh run_chunkcases.sh test_nczarr.sh run_perf_chunks1.sh run_s3_cleanup.sh \ run_purezarr.sh run_interop.sh run_misc.sh \ run_filter.sh run_specific_filters.sh \ -run_newformat.sh run_nczarr_fill.sh +run_newformat.sh run_nczarr_fill.sh run_quantize.sh EXTRA_DIST += \ ref_ut_map_create.cdl ref_ut_map_writedata.cdl ref_ut_map_writemeta2.cdl ref_ut_map_writemeta.cdl \ diff --git a/nczarr_test/ref_filtered.cdl b/nczarr_test/ref_filtered.cdl index 6afa77b2e2..cf9e6febb4 100644 --- a/nczarr_test/ref_filtered.cdl +++ b/nczarr_test/ref_filtered.cdl @@ -11,7 +11,6 @@ dimensions: group: g { variables: float var(dim0, dim1, dim2, dim3) ; - var:_FillValue = 9.96921e+36f ; var:_Storage = "chunked" ; var:_ChunkSizes = 4, 4, 4, 4 ; var:_Filter = "307,9" ; diff --git a/nczarr_test/ref_purezarr.cdl b/nczarr_test/ref_purezarr.cdl index f9b9720608..edc00790f7 100644 --- a/nczarr_test/ref_purezarr.cdl +++ b/nczarr_test/ref_purezarr.cdl @@ -4,7 +4,6 @@ dimensions: _zdim_5 = 5 ; variables: int i(_zdim_2, _zdim_5) ; - i:_FillValue = -2147483647 ; data: i = diff --git a/nczarr_test/ref_xarray.cdl b/nczarr_test/ref_xarray.cdl index cd3e3c3874..69ef9f8e8b 100644 --- a/nczarr_test/ref_xarray.cdl +++ b/nczarr_test/ref_xarray.cdl @@ -4,7 +4,6 @@ dimensions: y = 5 ; variables: int i(x, y) ; - i:_FillValue = -2147483647 ; data: i = diff --git a/nczarr_test/run_interop.sh b/nczarr_test/run_interop.sh index 37fd6d6a5f..2ff026155f 100755 --- a/nczarr_test/run_interop.sh +++ b/nczarr_test/run_interop.sh @@ -84,12 +84,8 @@ esac #if test "x$FEATURE_NCZARR_ZIP" = xyes ; then testallcases zip; fi if test "x$FEATURE_S3TESTS" = xyes ; then testallcases s3; fi exit + # Cleanup rm -fr ${execdir}/ref_power_901_constants.file -rm -f ${execdir}/ref_zarr_test_data.cdl -if test "x$srcdir" != "x$execdir" ; then - rm -fr ${execdir}/ref_power_901_constants.zip - rm -fr ${execdir}/ref_quotes.zip -fi exit 0 diff --git a/nczarr_test/run_quantize.sh b/nczarr_test/run_quantize.sh new file mode 100755 index 0000000000..67ab361713 --- /dev/null +++ b/nczarr_test/run_quantize.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +if test "x$srcdir" = x ; then srcdir=`pwd`; fi +. ../test_common.sh + +. "$srcdir/test_nczarr.sh" + +# This shell script runs test_quantize + +set -e + +testcase() { + zext=$1 + fileargs tmp_quantize "mode=$zarr,$zext" + case "$zext" in + file) template="file://${execdir}/%s.zarr#mode=zarr,$zext" ;; + zip) template="file://${execdir}/%s.zip#mode=zarr,$zext" ;; + s3) template="s3://${NCZARR_S3_TEST_BUCKET}/netcdf-c/%s.zarr#mode=zarr,$zext" ;; + *) echo "unknown file type"; exit 1 ;; + esac + ${execdir}/test_quantize "$template" +} + +testcase file +if test "x$FEATURE_NCZARR_ZIP" = xyes ; then testcase zip; fi +if test "x$FEATURE_S3TESTS" = xyes ; then testcase s3; fi diff --git a/nczarr_test/test_quantize.c b/nczarr_test/test_quantize.c new file mode 100644 index 0000000000..23685633c4 --- /dev/null +++ b/nczarr_test/test_quantize.c @@ -0,0 +1,853 @@ +/* This is part of the netCDF package. + Copyright 2021 University Corporation for Atmospheric Research/Unidata + See COPYRIGHT file for conditions of use. + + Test quantization of netcdf-4 variables. Quantization is the + zeroing-out of bits in float or double data beyond a desired + precision. + + Ed Hartnett, 8/19/21 +*/ + +#include +#include "err_macros.h" +#include "netcdf.h" + +#define DEBUG + + +#define TEST "test_quantize" +#define FILE_NAME "tmp_quantize" +#define NDIM1 1 +#define DIM_NAME_1 "meters_along_canal" +#define DIM_LEN_3 3 +#define DIM_LEN_1 1 +#define DIM_LEN_5 5 +#define DIM_LEN_8 8 +#define VAR_NAME_1 "Amsterdam_houseboat_location" +#define VAR_NAME_2 "Amsterdam_street_noise_decibels" +#define NSD_3 3 +#define NSD_9 9 + +/* This var used to help print a float in hex. */ +char pf_str[20]; + +/* This struct allows us to treat float as uint32_t + * types. */ +union FU { + float f; + uint32_t u; +}; + +/* This struct allows us to treat double points as uint64_t + * types. */ +union DU { + double d; + uint64_t u; +}; + +/* This function prints a float as hex. */ +char * +pf(float myf) +{ + union { + float f; + uint32_t u; + } fu; + fu.f = myf; + sprintf(pf_str, "0x%x", fu.u); + return pf_str; +} + +/* This function prints a double as hex. */ +char * +pd(double myd) +{ + union { + double d; + uint64_t u; + } du; + du.d = myd; + sprintf(pf_str, "0x%lx", du.u); + return pf_str; +} + +int +main(int argc, char **argv) +{ + int retval = NC_NOERR; + const char* template = NULL; + char file_name[4096]; + + if(argc == 1) + {fprintf(stderr,"usage: test_quantize \n"); exit(1);} + + template = argv[1]; + + snprintf(file_name,sizeof(file_name),template,FILE_NAME); +#ifdef DEBUG + fprintf(stderr,"file_name=|%s|\n",file_name); +#endif + + + printf("\n*** Testing netcdf-4 variable quantization functions.\n"); + printf("**** testing quantization setting and error conditions..."); + { + int ncid, dimid, varid1, varid2; + int quantize_mode_in, nsd_in; + + /* Create a netcdf-4 file with two vars. Attempt + * quantization. It will work, eventually... */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_dim(ncid, DIM_NAME_1, DIM_LEN_3, &dimid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Bad varid. */ + if (nc_def_var_quantize(ncid, NC_GLOBAL, NC_QUANTIZE_BITGROOM, NSD_3) != NC_EGLOBAL) ERR; + if (nc_def_var_quantize(ncid, varid2 + 1, NC_QUANTIZE_BITGROOM, NSD_3) != NC_ENOTVAR) ERR; + /* Invalid values. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM + 1, NSD_3) != NC_EINVAL) ERR; + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, -1) != NC_EINVAL) ERR; + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NC_QUANTIZE_MAX_FLOAT_NSD + 1) != NC_EINVAL) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM + 1, 3) != NC_EINVAL) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, -1) != NC_EINVAL) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NC_QUANTIZE_MAX_DOUBLE_NSD + 1) != NC_EINVAL) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, 0) != NC_EINVAL) ERR; + + /* This will work. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_inq_var_quantize(ncid, varid1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM) ERR; + if (nsd_in != NSD_3) ERR; + + /* Wait, I changed my mind! Let's turn off quantization. */ + if (nc_def_var_quantize(ncid, varid1, NC_NOQUANTIZE, 0)) ERR; + if (nc_inq_var_quantize(ncid, varid1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_NOQUANTIZE) ERR; + if (nsd_in != 0) ERR; + + /* Changed my mind again, turn it on. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* I changed my mind again! Turn it off! */ + if (nc_def_var_quantize(ncid, varid1, NC_NOQUANTIZE, 0)) ERR; + if (nc_inq_var_quantize(ncid, varid1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_NOQUANTIZE) ERR; + if (nsd_in != 0) ERR; + + /* Changed my mind again, turn it on. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* This also will work for double. */ + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_9)) ERR; + if (nc_inq_var_quantize(ncid, varid2, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM) ERR; + if (nsd_in != NSD_9) ERR; + + /* End define mode. */ + if (nc_enddef(ncid)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + /* Open the file and check. */ + if ((retval=nc_open(file_name, NC_WRITE, &ncid))) + ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM) ERR; + if (nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM) ERR; + if (nsd_in != NSD_9) ERR; + if (nc_close(ncid)) ERR; + } + SUMMARIZE_ERR; + +#define NX_BIG 100 +#define NY_BIG 100 +#define NTYPES 9 +#define VAR_NAME "Amsterdam_coffeeshop_location" +#define X_NAME "distance_from_center" +#define Y_NAME "distance_along_canal" +#define NDIM2 2 + + printf("**** testing quantization handling of non-floats..."); + { + int ncid; + int dimid[NDIM2]; + int varid; + int nsd_in, quantize_mode; + int nsd_out = 3; + int xtype[NTYPES] = {NC_CHAR, NC_SHORT, NC_INT, NC_BYTE, NC_UBYTE, + NC_USHORT, NC_UINT, NC_INT64, NC_UINT64}; + int t; + + for (t = 0; t < NTYPES; t++) + { + char fname[4096]; + char basename[4096]; + + sprintf(basename, "tmp_bitgroom_type_%d", xtype[t]); + sprintf(fname, template, basename); +#ifdef DEBUG + fprintf(stderr,"fname=|%s|\n",fname); +#endif + + /* Create file. */ + if (nc_create(fname, NC_NETCDF4, &ncid)) ERR; + if (nc_def_dim(ncid, X_NAME, NX_BIG, &dimid[0])) ERR; + if (nc_def_dim(ncid, Y_NAME, NY_BIG, &dimid[1])) ERR; + if (nc_def_var(ncid, VAR_NAME, xtype[t], NDIM2, dimid, &varid)) ERR; + + /* Bitgroom filter returns NC_EINVAL because this is not an + * NC_FLOAT or NC_DOULBE. */ + if (nc_def_var_quantize(ncid, varid, NC_QUANTIZE_BITGROOM, nsd_out) != NC_EINVAL) ERR; + if (nc_close(ncid)) ERR; + + /* Check file. */ + { + if (nc_open(fname, NC_NETCDF4, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, varid, &quantize_mode, &nsd_in)) ERR; + if (quantize_mode) ERR; + if (nc_close(ncid)) ERR; + } + } + } + SUMMARIZE_ERR; + printf("**** testing quantization of scalars..."); + { + int ncid, varid1, varid2; + int quantize_mode_in, nsd_in; + float float_data[DIM_LEN_1] = {1.1111111}; + double double_data[DIM_LEN_1] = {1.111111111111}; + + /* Create a netcdf-4 file with two scalar vars. */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, 0, NULL, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, 0, NULL, &varid2)) ERR; + + /* Turn on quantize for both vars. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write some data. */ + if (nc_put_var_float(ncid, varid1, float_data)) ERR; + if (nc_put_var_double(ncid, varid2, double_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_in; + double double_in; + union FU fin; + int nsd_att_in; + /* union FU fout; */ + union DU dfin; + /* union DU dfout; */ + + /* Open the file and check metadata. */ + if (nc_open(file_name, NC_WRITE, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + + /* Each var now has an attribute describing the quantize settings. */ + if (nc_get_att_int(ncid, 0, NC_QUANTIZE_ATT_NAME, &nsd_att_in)) ERR; + if (nsd_att_in != NSD_3) ERR; + if (nc_get_att_int(ncid, 1, NC_QUANTIZE_ATT_NAME, &nsd_att_in)) ERR; + if (nsd_att_in != NSD_3) ERR; + + /* Check the data. */ + if (nc_get_var(ncid, varid1, &float_in)) ERR; + if (nc_get_var(ncid, varid2, &double_in)) ERR; + /* fout.f = float_data[0]; */ + fin.f = float_in; + /* dfout.d = double_data[0]; */ + dfin.d = double_in; + /* printf ("\nfloat_data: %10f : 0x%x float_data_in: %10f : 0x%x\n", */ + /* float_data[0], fout.u, float_data[0], fin.u); */ + if (fin.u != 0x3f8e3000) ERR; + /* printf ("\ndouble_data: %15g : 0x%16lx double_data_in: %15g : 0x%lx\n", */ + /* double_data[0], dfout.u, double_data[0], dfin.u); */ + if (dfin.u != 0x3ff1c60000000000) ERR; + + /* Close the file again. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + printf("**** testing quantization of one value..."); + { + int ncid, dimid, varid1, varid2; + int quantize_mode_in, nsd_in; + float float_data[DIM_LEN_1] = {1.1111111}; + double double_data[DIM_LEN_1] = {1.111111111111}; + + /* Create a netcdf-4 file with two vars. */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_dim(ncid, DIM_NAME_1, DIM_LEN_1, &dimid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Turn on quantize for both vars. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write some data. */ + if (nc_put_var_float(ncid, varid1, float_data)) ERR; + if (nc_put_var_double(ncid, varid2, double_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_in; + double double_in; + union FU fin; + /* union FU fout; */ + union DU dfin; + /* union DU dfout; */ + + /* Open the file and check metadata. */ + if (nc_open(file_name, NC_WRITE, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + + /* Check the data. */ + if (nc_get_var(ncid, varid1, &float_in)) ERR; + if (nc_get_var(ncid, varid2, &double_in)) ERR; + /* fout.f = float_data[0]; */ + fin.f = float_in; + /* dfout.d = double_data[0]; */ + dfin.d = double_in; + /* printf ("\nfloat_data: %10f : 0x%x float_data_in: %10f : 0x%x\n", */ + /* float_data[0], fout.u, float_data[0], fin.u); */ + if (fin.u != 0x3f8e3000) ERR; + /* printf ("\ndouble_data: %15g : 0x%16lx double_data_in: %15g : 0x%lx\n", */ + /* double_data[0], dfout.u, double_data[0], dfin.u); */ + if (dfin.u != 0x3ff1c60000000000) ERR; + + /* Close the file again. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + printf("**** testing more quantization values..."); + { + int ncid, dimid, varid1, varid2; + int quantize_mode_in, nsd_in; + float float_data[DIM_LEN_5] = {1.11111111, 1.0, 9.99999999, 12345.67, .1234567}; + double double_data[DIM_LEN_5] = {1.1111111, 1.0, 9.999999999, 1234567890.12345, 123456789012345.0}; + int x; + + /* Create a netcdf-4 file with two vars. */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_dim(ncid, DIM_NAME_1, DIM_LEN_5, &dimid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Turn on quantize for both vars. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write some data. */ + if (nc_put_var_float(ncid, varid1, float_data)) ERR; + if (nc_put_var_double(ncid, varid2, double_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_in[DIM_LEN_5]; + double double_in[DIM_LEN_5]; + union FU { + float f; + uint32_t u; + }; + + union FU fin; + /* union FU fout; */ + union FU xpect[DIM_LEN_5]; + union DU dfin; + /* union DU dfout; */ + union DU double_xpect[DIM_LEN_5]; + xpect[0].u = 0x3f8e3000; + xpect[1].u = 0x3f800fff; + xpect[2].u = 0x41200000; + xpect[3].u = 0x4640efff; + xpect[4].u = 0x3dfcd000; + double_xpect[0].u = 0x3ff1c60000000000; + double_xpect[1].u = 0x3ff001ffffffffff; + double_xpect[2].u = 0x4023fe0000000000; + double_xpect[3].u = 0x41d265ffffffffff; + double_xpect[4].u = 0x42dc120000000000; + + /* Open the file and check metadata. */ + if (nc_open(file_name, NC_WRITE, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + + /* Check the data. */ + if (nc_get_var(ncid, varid1, float_in)) ERR; + if (nc_get_var(ncid, varid2, double_in)) ERR; + /* printf("\n"); */ + for (x = 0; x < DIM_LEN_5; x++) + { + /* fout.f = float_data[x]; */ + fin.f = float_in[x]; + /* printf ("float_data: %10f : 0x%x float_data_in: %10f : 0x%x\n", */ + /* float_data[x], fout.u, float_data[x], fin.u); */ + if (fin.u != xpect[x].u) ERR; + /* dfout.d = double_data[x]; */ + dfin.d = double_in[x]; + /* printf("double_data: %15g : 0x%16lx double_data_in: %15g : 0x%16lx\n", */ + /* double_data[x], dfout.u, double_data[x], dfin.u); */ + if (dfin.u != double_xpect[x].u) ERR; + } + + /* Close the file again. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + printf("**** testing quantization of one value with type conversion..."); + { + int ncid, dimid, varid1, varid2; + int quantize_mode_in, nsd_in; + float float_data[DIM_LEN_1] = {1.1111111}; + double double_data[DIM_LEN_1] = {1.111111111111}; + + /* Create a netcdf-4 file with two vars. */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_dim(ncid, DIM_NAME_1, DIM_LEN_1, &dimid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Turn on quantize for both vars. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write some double data to float var. */ + if (nc_put_var_double(ncid, varid1, double_data)) ERR; + + /* Write some float data to double var. */ + if (nc_put_var_float(ncid, varid2, float_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_in; + double double_in; + union FU fin; + /* union FU fout; */ + union DU dfin; + /* union DU dfout; */ + int nsd_att_in; + + /* Open the file and check metadata. */ + if (nc_open(file_name, NC_WRITE, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + + /* Each var now has an attribute describing the quantize settings. */ + if (nc_get_att_int(ncid, 0, NC_QUANTIZE_ATT_NAME, &nsd_att_in)) ERR; + if (nsd_att_in != NSD_3) ERR; + if (nc_get_att_int(ncid, 1, NC_QUANTIZE_ATT_NAME, &nsd_att_in)) ERR; + if (nsd_att_in != NSD_3) ERR; + + /* Check the data. */ + if (nc_get_var(ncid, varid1, &float_in)) ERR; + if (nc_get_var(ncid, varid2, &double_in)) ERR; + /* fout.f = (float)double_data[0]; */ + fin.f = float_in; + /* dfout.d = float_data[0]; */ + dfin.d = double_in; + /* printf ("\ndouble_data: %15g : 0x%x float_data_in: %10f : 0x%x\n", */ + /* double_data[0], fout.u, float_in, fin.u); */ + if (fin.u != 0x3f8e3000) ERR; + /* printf ("\nfloat_data: %15g : 0x%16lx double_data_in: %15g : 0x%lx\n", */ + /* float_data[0], dfout.u, double_in, dfin.u); */ + if (dfin.u != 0x3ff1c60000000000) ERR; + + /* Close the file again. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + printf("**** testing more quantization values with type conversion..."); + { + int ncid, dimid, varid1, varid2; + int quantize_mode_in, nsd_in; + float float_data[DIM_LEN_5] = {1.11111111, 1.0, 9.99999999, 12345.67, .1234567}; + double double_data[DIM_LEN_5] = {1.1111111, 1.0, 9.999999999, 1234567890.12345, 123456789012345.0}; + int x; + + /* Create a netcdf-4 file with two vars. */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_dim(ncid, DIM_NAME_1, DIM_LEN_5, &dimid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Turn on quantize for both vars. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write some data. */ + if (nc_put_var_double(ncid, varid1, double_data)) ERR; + if (nc_put_var_float(ncid, varid2, float_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_in[DIM_LEN_5]; + double double_in[DIM_LEN_5]; + union FU { + float f; + uint32_t u; + }; + + union FU fin; + /* union FU fout; */ + union FU xpect[DIM_LEN_5]; + union DU dfin; + /* union DU dfout; */ + union DU double_xpect[DIM_LEN_5]; + xpect[0].u = 0x3f8e3000; + xpect[1].u = 0x3f800fff; + xpect[2].u = 0x41200000; + xpect[3].u = 0x4e932fff; + xpect[4].u = 0x56e09000; + double_xpect[0].u = 0x3ff1c60000000000; + double_xpect[1].u = 0x3ff001ffffffffff; + double_xpect[2].u = 0x4024000000000000; + double_xpect[3].u = 0x40c81dffffffffff; + double_xpect[4].u = 0x3fbf9a0000000000; + + /* Open the file and check metadata. */ + if (nc_open(file_name, NC_WRITE, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + + /* Check the data. */ + if (nc_get_var(ncid, varid1, float_in)) ERR; + if (nc_get_var(ncid, varid2, double_in)) ERR; + /* printf("\n"); */ + for (x = 0; x < DIM_LEN_5; x++) + { + /* fout.f = float_data[x]; */ + fin.f = float_in[x]; + /* printf ("float_data: %10f : 0x%x float_data_in: %10f : 0x%x\n", */ + /* float_data[x], fout.u, float_data[x], fin.u); */ + if (fin.u != xpect[x].u) ERR; + /* dfout.d = double_data[x]; */ + dfin.d = double_in[x]; + /* printf("double_data: %15g : 0x%16lx double_data_in: %15g : 0x%16lx\n", */ + /* double_data[x], dfout.u, double_data[x], dfin.u); */ + if (dfin.u != double_xpect[x].u) ERR; + } + + /* Close the file again. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + printf("**** testing more quantization values with default fill values..."); + { + int ncid, dimid, varid1, varid2; + int quantize_mode_in, nsd_in; + float float_data[DIM_LEN_5] = {1.11111111, NC_FILL_FLOAT, 9.99999999, 12345.67, NC_FILL_FLOAT}; + double double_data[DIM_LEN_5] = {1.1111111, NC_FILL_DOUBLE, 9.999999999, 1234567890.12345, NC_FILL_DOUBLE}; + int x; + + /* Create a netcdf-4 file with two vars. */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_dim(ncid, DIM_NAME_1, DIM_LEN_5, &dimid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Turn on quantize for both vars. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write some data. */ + if (nc_put_var_float(ncid, varid1, float_data)) ERR; + if (nc_put_var_double(ncid, varid2, double_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_in[DIM_LEN_5]; + double double_in[DIM_LEN_5]; + union FU { + float f; + uint32_t u; + }; + + union FU fin; + /* union FU fout; */ + union FU xpect[DIM_LEN_5]; + union DU dfin; + /* union DU dfout; */ + union DU double_xpect[DIM_LEN_5]; + xpect[0].u = 0x3f8e3000; + xpect[1].u = 0x7cf00000; + xpect[2].u = 0x41200000; + xpect[3].u = 0x4640efff; + xpect[4].u = 0x7cf00000; + double_xpect[0].u = 0x3ff1c60000000000; + double_xpect[1].u = 0x479e000000000000; + double_xpect[2].u = 0x4023fe0000000000; + double_xpect[3].u = 0x41d265ffffffffff; + double_xpect[4].u = 0x479e000000000000; + + /* Open the file and check metadata. */ + if (nc_open(file_name, NC_WRITE, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + + /* Check the data. */ + if (nc_get_var(ncid, varid1, float_in)) ERR; + if (nc_get_var(ncid, varid2, double_in)) ERR; + /* printf("\n"); */ + for (x = 0; x < DIM_LEN_5; x++) + { + /* fout.f = float_data[x]; */ + fin.f = float_in[x]; + /* printf ("float_data: %10f : 0x%x float_data_in: %10f : 0x%x\n", */ + /* float_data[x], fout.u, float_data[x], fin.u); */ + if (fin.u != xpect[x].u) ERR; + /* dfout.d = double_data[x]; */ + dfin.d = double_in[x]; + /* printf("double_data: %15g : 0x%16lx double_data_in: %15g : 0x%16lx\n", */ + /* double_data[x], dfout.u, double_data[x], dfin.u); */ + if (dfin.u != double_xpect[x].u) ERR; + } + + /* Close the file again. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + printf("**** testing more quantization values with custom fill values..."); + { + #define CUSTOM_FILL_FLOAT 99.99999 + #define CUSTOM_FILL_DOUBLE -99999.99999 + int ncid, dimid, varid1, varid2; + int quantize_mode_in, nsd_in; + float float_data[DIM_LEN_5] = {1.11111111, CUSTOM_FILL_FLOAT, 9.99999999, 12345.67, CUSTOM_FILL_FLOAT}; + double double_data[DIM_LEN_5] = {1.1111111, CUSTOM_FILL_DOUBLE, 9.999999999, 1234567890.12345, CUSTOM_FILL_DOUBLE}; + float custom_fill_float = CUSTOM_FILL_FLOAT; + double custom_fill_double = CUSTOM_FILL_DOUBLE; + int x; + + /* Create a netcdf-4 file with two vars. */ + if (nc_create(file_name, NC_NETCDF4|NC_CLOBBER, &ncid)) ERR; + if (nc_def_dim(ncid, DIM_NAME_1, DIM_LEN_5, &dimid)) ERR; + if (nc_def_var(ncid, VAR_NAME_1, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_put_att_float(ncid, varid1, _FillValue, NC_FLOAT, 1, &custom_fill_float)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + if (nc_put_att_double(ncid, varid2, _FillValue, NC_DOUBLE, 1, &custom_fill_double)) ERR; + + /* Turn on quantize for both vars. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write some data. */ + if (nc_put_var_float(ncid, varid1, float_data)) ERR; + if (nc_put_var_double(ncid, varid2, double_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_in[DIM_LEN_5]; + double double_in[DIM_LEN_5]; + union FU { + float f; + uint32_t u; + }; + + union FU fin; + /* union FU fout; */ + union FU xpect[DIM_LEN_5]; + union DU dfin; + /* union DU dfout; */ + union DU double_xpect[DIM_LEN_5]; + xpect[0].u = 0x3f8e3000; + xpect[1].u = 0x42c7ffff; + xpect[2].u = 0x41200000; + xpect[3].u = 0x4640efff; + xpect[4].u = 0x42c7ffff; + double_xpect[0].u = 0x3ff1c60000000000; + double_xpect[1].u = 0xc0f869fffff583a5; + double_xpect[2].u = 0x4023fe0000000000; + double_xpect[3].u = 0x41d265ffffffffff; + double_xpect[4].u = 0xc0f869fffff583a5; + + /* Open the file and check metadata. */ + if (nc_open(file_name, NC_WRITE, &ncid)) ERR; + if (nc_inq_var_quantize(ncid, 0, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + if (nc_inq_var_quantize(ncid, 1, &quantize_mode_in, &nsd_in)) ERR; + if (quantize_mode_in != NC_QUANTIZE_BITGROOM || nsd_in != NSD_3) ERR; + + /* Check the data. */ + if (nc_get_var(ncid, varid1, float_in)) ERR; + if (nc_get_var(ncid, varid2, double_in)) ERR; + /* printf("\n"); */ + for (x = 0; x < DIM_LEN_5; x++) + { + /* fout.f = float_data[x]; */ + fin.f = float_in[x]; + /* printf ("float_data: %10f : 0x%x float_data_in: %10f : 0x%x\n", */ + /* float_data[x], fout.u, float_data[x], fin.u); */ + if (fin.u != xpect[x].u) ERR; + /* dfout.d = double_data[x]; */ + dfin.d = double_in[x]; + /* printf("double_data: %15g : 0x%16lx double_data_in: %15g : 0x%16lx\n", */ + /* double_data[x], dfout.u, double_data[x], dfin.u); */ + if (dfin.u != double_xpect[x].u) ERR; + } + + /* Close the file again. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + printf("*** Checking BitGroom values with type conversion between ints and floats..."); + { + int ncid; + int dimid; + int varid1, varid2; + unsigned char uc = 99; + signed char sc = -99; + unsigned short us = 9999; + signed short ss = -9999; + unsigned int ui = 9999999; + signed int si = -9999999; + unsigned long long int ull = 999999999; + signed long long int sll = -999999999; + size_t index; + + /* Create file. */ + if (nc_create(file_name, NC_NETCDF4, &ncid)) ERR; + + /* Create dims. */ + if (nc_def_dim(ncid, X_NAME, DIM_LEN_8, &dimid)) ERR; + + /* Create the variables. */ + if (nc_def_var(ncid, VAR_NAME, NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, VAR_NAME_2, NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Set up quantization. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Write data. */ + index = 0; + if (nc_put_var1_uchar(ncid, varid1, &index, &uc)) ERR; + if (nc_put_var1_uchar(ncid, varid2, &index, &uc)) ERR; + index = 1; + if (nc_put_var1_schar(ncid, varid1, &index, &sc)) ERR; + if (nc_put_var1_schar(ncid, varid2, &index, &sc)) ERR; + index = 2; + if (nc_put_var1_ushort(ncid, varid1, &index, &us)) ERR; + if (nc_put_var1_ushort(ncid, varid2, &index, &us)) ERR; + index = 3; + if (nc_put_var1_short(ncid, varid1, &index, &ss)) ERR; + if (nc_put_var1_short(ncid, varid2, &index, &ss)) ERR; + index = 4; + if (nc_put_var1_uint(ncid, varid1, &index, &ui)) ERR; + if (nc_put_var1_uint(ncid, varid2, &index, &ui)) ERR; + index = 5; + if (nc_put_var1_int(ncid, varid1, &index, &si)) ERR; + if (nc_put_var1_int(ncid, varid2, &index, &si)) ERR; + index = 6; + if (nc_put_var1_ulonglong(ncid, varid1, &index, &ull)) ERR; + if (nc_put_var1_ulonglong(ncid, varid2, &index, &ull)) ERR; + index = 7; + if (nc_put_var1_longlong(ncid, varid1, &index, &sll)) ERR; + if (nc_put_var1_longlong(ncid, varid2, &index, &sll)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + { + float float_data_in[DIM_LEN_8]; + double double_data_in[DIM_LEN_8]; + int x; + + /* Now reopen the file and check. */ + if (nc_open(file_name, NC_NETCDF4, &ncid)) ERR; + + /* Read the data. */ + if (nc_get_var_float(ncid, varid1, float_data_in)) ERR; + if (nc_get_var_double(ncid, varid2, double_data_in)) ERR; + + union FU xpect[DIM_LEN_8]; + union DU double_xpect[DIM_LEN_8]; + /* This test comes up with different answers to this than + * the corresponding bitgroom filter test, but that's + * OK. In netcdf-c quantization is applied as the data are + * written by the user, but in HDF5 filters, the bitgroom + * filter is applied to all data values as they are + * written to disk. See + * https://github.com/ccr/ccr/issues/194 for a full + * explanation. */ + xpect[0].u = 0x42c60000; + xpect[1].u = 0xc2c60000; + xpect[2].u = 0x461c3000; + xpect[3].u = 0xc61c3000; + xpect[4].u = 0x4b189000; + xpect[5].u = 0xcb189000; + xpect[6].u = 0x4e6e6b28; + xpect[6].u = 0x4e6e6000; + xpect[7].u = 0xce6e6000; + double_xpect[0].u = 0x4058c00000000000; + double_xpect[1].u = 0xc058c00000000000; + double_xpect[2].u = 0x40c3860000000000; + double_xpect[3].u = 0xc0c3860000000000; + double_xpect[4].u = 0x4163120000000000; + double_xpect[5].u = 0xc163120000000000; + double_xpect[6].u = 0x41cdcc0000000000; + double_xpect[7].u = 0xc1cdcc0000000000; + + for (x = 0; x < DIM_LEN_8; x++) + { + union FU fin; + union DU dfin; + fin.f = float_data_in[x]; + dfin.d = double_data_in[x]; + /* printf ("%d float_data_in : %08.8f : 0x%x expected %08.8f : 0x%x\n", */ + /* x, float_data_in[x], fin.u, xpect[x].f, xpect[x].u); */ + /* printf ("%d double_data_in : %15g : 0x%lx expected %15g : 0x%lx\n", */ + /* x, double_data_in[x], dfin.u, double_xpect[x].d, double_xpect[x].u); */ + if (fin.u != xpect[x].u) + ERR; + if (dfin.u != double_xpect[x].u) + ERR; + } + + /* Close the file. */ + if (nc_close(ncid)) ERR; + } + } + SUMMARIZE_ERR; + FINAL_RESULTS; +} diff --git a/plugins/Makefile.am b/plugins/Makefile.am index 00422a8f8e..599aca37a5 100644 --- a/plugins/Makefile.am +++ b/plugins/Makefile.am @@ -52,7 +52,7 @@ libh5noop1_la_SOURCES = H5Znoop1.c H5Zutil.c h5noop.h endif #ENABLE_FILTER_TESTING BUILT_SOURCES = H5Znoop1.c -DISTCLEANFILES = H5Znoop1.c ncjson.h +DISTCLEANFILES = H5Znoop1.c H5Znoop1.c: Makefile H5Znoop.c echo '#define NOOP_INSTANCE 1' > $@ cat ${srcdir}/H5Znoop.c >> $@ From 2543a8ed41b0387343b455ac1926d065fd0b8aeb Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Wed, 3 Nov 2021 15:49:15 -0600 Subject: [PATCH 02/12] Update Release Notes --- RELEASE_NOTES.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 0134a2e83d..3b119d4467 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -7,7 +7,7 @@ This file contains a high-level description of this package's evolution. Release ## 4.8.2 - TBD -* [Enhancement] Add bitgroom support to NCZarr. See [Github #2???](https://github.com/Unidata/netcdf-c/pull/2???) +* [Enhancement] Add bitgroom support to NCZarr. See [Github #2140](https://github.com/Unidata/netcdf-c/pull/2140) * [Enhancement] Support byte-range reading of netcdf-3 files stored in private buckets in S3. See [Github #2134](https://github.com/Unidata/netcdf-c/pull/2134) * [Enhancement] Support Amazon S3 access for NCZarr. Also support use of the existing Amazon SDK credentials system. See [Github #2114](https://github.com/Unidata/netcdf-c/pull/2114) * [Bug Fix] Fix string allocation error in H5FDhttp.c. See [Github #2127](https://github.com/Unidata/netcdf-c/pull/2127). From c2c3ce3a7d4992e5e2e310f2aa9045c2c4b380e8 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Wed, 3 Nov 2021 16:03:38 -0600 Subject: [PATCH 03/12] Fix lgtm alert --- libnczarr/zvar.c | 1 + 1 file changed, 1 insertion(+) diff --git a/libnczarr/zvar.c b/libnczarr/zvar.c index 8021d98614..d23124343e 100644 --- a/libnczarr/zvar.c +++ b/libnczarr/zvar.c @@ -536,6 +536,7 @@ ncz_def_var_extra(int ncid, int varid, int *shuffle, int *unused1, (fletcher32?*fletcher32:-1), (no_fill?*no_fill:-1), fill_value, + (endianness?*endianness:-1), (quantize_mode?*quantize_mode:-1), (nsd?*nsd:-1) ); From 5e3ea0e49e87dd572ec7b552f134e09ec9ca2565 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Thu, 4 Nov 2021 21:06:45 -0600 Subject: [PATCH 04/12] Fix libxml2 check --- configure.ac | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/configure.ac b/configure.ac index 707fbdc58b..6cd69b9f54 100644 --- a/configure.ac +++ b/configure.ac @@ -1062,7 +1062,7 @@ AC_MSG_CHECKING([whether to search for and use external libxml2]) AC_ARG_ENABLE([libxml2], [AS_HELP_STRING([--disable-libxml2], [disable detection and use of libxml2 in favor of the bundled ezxml interpreter])]) -test "x$disable_libxml2" = xyes && enable_libxml2=no +test "x$enable_libxml2" = xno || enable_libxml2=yes AC_MSG_RESULT($enable_libxml2) AC_SUBST([XMLPARSER],"ezxml") From d065a2890c2451213526435b50b454d843e28d15 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 5 Nov 2021 12:59:56 -0600 Subject: [PATCH 05/12] Fix bitgroom support to NCZarr re: PR https://github.com/Unidata/netcdf-c/pull/2140 Fix some outstanding problems with the above PR. * Minor consistency change to --enable-libxml2 * Simplify the makelib construction rule in libncxml/Makefile.am by eliminating the need for unistd and the file apis. --- configure.ac | 2 +- libncxml/Makefile.am | 27 +++++---- libncxml/ezxml.c | 2 + libncxml/ezxml.h | 133 ++++++++++++++++++++----------------------- 4 files changed, 81 insertions(+), 83 deletions(-) diff --git a/configure.ac b/configure.ac index 707fbdc58b..6cd69b9f54 100644 --- a/configure.ac +++ b/configure.ac @@ -1062,7 +1062,7 @@ AC_MSG_CHECKING([whether to search for and use external libxml2]) AC_ARG_ENABLE([libxml2], [AS_HELP_STRING([--disable-libxml2], [disable detection and use of libxml2 in favor of the bundled ezxml interpreter])]) -test "x$disable_libxml2" = xyes && enable_libxml2=no +test "x$enable_libxml2" = xno || enable_libxml2=yes AC_MSG_RESULT($enable_libxml2) AC_SUBST([XMLPARSER],"ezxml") diff --git a/libncxml/Makefile.am b/libncxml/Makefile.am index 7ff656932e..8d88e5d98d 100644 --- a/libncxml/Makefile.am +++ b/libncxml/Makefile.am @@ -32,19 +32,24 @@ EXTRA_DIST = CMakeLists.txt license.txt REPO=https://downloads.sourceforge.net/project/ezxml/ EZXML=ezxml-0.8.6.tar.gz makelib:: - rm -fr ./ezxml.[ch] ./license.txt ./ezxml - tar -zxf ./${EZXML} - echo '#define EZXML_NOMMAP 1' > ezxml.c - cat /d' | \ - sed -e 's|//\(.*\)|/*\1*/|' \ - sed -e 's|//\(.*\)|/*\1*/|' \ - cat >./ezxml.c - sed -e 's|//\(.*\)|/*\1*/|' ./ezxml.h - sed -i .bak 's//\n#ifdef HAVE_UNISTD_H\n#include \n#endif/g' ezxml.h - rm ezxml.h.bak + echo "WARNING DO NOT RUN THIS since the patches are not in the tar file" + exit 1 + rm -fr ./ezxml.[ch] ./license.txt ./ezxml ./ezxml.c.? ./ezxml.h.? + tar -zxf ${EZXML} + cat ezxml/ezxml.h > ./ezxml.h + sed -i.1 -e '/ezxml_parse_fd/d' ezxml.h + sed -i.2 -e '/ezxml_parse_file/d' ezxml.h + sed -i.3 -e '/ezxml_parse_fp/d' ezxml.h + sed -i.4 -e 's|//\(.*\)|/*\1*/|' ezxml.h + echo "#define EZXML_NOMMAP 1" > ./ezxml.c + cat ezxml/ezxml.c >> ./ezxml.c + sed -i.1 -e '//d' ezxml.c + sed -i.2 -e '/ezxml_parse_fp(FILE/i#if 0' ./ezxml.c + sed -i.3 -e '/ezxml_ampencode(const/i#endif //0' ./ezxml.c + sed -i.4 -e 's|//\(.*\)|/*\1*/|' ezxml.c cp ezxml/license.txt . rm -fr ezxml + rm -f ezxml.c.? ezxml.h.? # Define path to the xml github dir; this value assumes it is in a parallel directory to netcdf-c (YMMV) GITSRC=${top_srcdir}/../tinyxml2 diff --git a/libncxml/ezxml.c b/libncxml/ezxml.c index 96af31f97d..36b265885f 100644 --- a/libncxml/ezxml.c +++ b/libncxml/ezxml.c @@ -623,6 +623,7 @@ ezxml_t ezxml_parse_str(char *s, size_t len) /* Wrapper for ezxml_parse_str() that accepts a file stream. Reads the entire*/ /* stream into memory and then parses it. For xml files, use ezxml_parse_file()*/ /* or ezxml_parse_fd()*/ +#if 0 ezxml_t ezxml_parse_fp(FILE *fp) { ezxml_root_t root; @@ -688,6 +689,7 @@ ezxml_t ezxml_parse_file(const char *file) /* Encodes ampersand sequences appending the results to *dst, reallocating *dst*/ /* if length excedes max. a is non-zero for attribute encoding. Returns *dst*/ +#endif /*0*/ char *ezxml_ampencode(const char *s, size_t len, char **dst, size_t *dlen, size_t *max, short a) { diff --git a/libncxml/ezxml.h b/libncxml/ezxml.h index a39334acee..c68c9b1666 100644 --- a/libncxml/ezxml.h +++ b/libncxml/ezxml.h @@ -29,142 +29,133 @@ #include #include #include -#ifdef HAVE_UNISTD_H -#include -#endif #ifdef __cplusplus extern "C" { #endif -#define EZXML_BUFSIZE 1024 /* size of internal memory buffers*/ -#define EZXML_NAMEM 0x80 /* name is malloced*/ -#define EZXML_TXTM 0x40 /* txt is malloced*/ -#define EZXML_DUP 0x20 /* attribute name and value are strduped*/ +#define EZXML_BUFSIZE 1024 /* size of internal memory buffers */ +#define EZXML_NAMEM 0x80 /* name is malloced */ +#define EZXML_TXTM 0x40 /* txt is malloced */ +#define EZXML_DUP 0x20 /* attribute name and value are strduped */ typedef struct ezxml *ezxml_t; struct ezxml { - char *name; /* tag name*/ - char **attr; /* tag attributes { name, value, name, value, ... NULL }*/ - char *txt; /* tag character content, empty string if none*/ - size_t off; /* tag offset from start of parent tag character content*/ - ezxml_t next; /* next tag with same name in this section at this depth*/ - ezxml_t sibling; /* next tag with different name in same section and depth*/ - ezxml_t ordered; /* next tag, same section and depth, in original order*/ - ezxml_t child; /* head of sub tag list, NULL if none*/ - ezxml_t parent; /* parent tag, NULL if current tag is root tag*/ - short flags; /* additional information*/ + char *name; /* tag name */ + char **attr; /* tag attributes { name, value, name, value, ... NULL } */ + char *txt; /* tag character content, empty string if none */ + size_t off; /* tag offset from start of parent tag character content */ + ezxml_t next; /* next tag with same name in this section at this depth */ + ezxml_t sibling; /* next tag with different name in same section and depth */ + ezxml_t ordered; /* next tag, same section and depth, in original order */ + ezxml_t child; /* head of sub tag list, NULL if none */ + ezxml_t parent; /* parent tag, NULL if current tag is root tag */ + short flags; /* additional information */ }; -/* Given a string of xml data and its length, parses it and creates an ezxml*/ -/* structure. For efficiency, modifies the data by adding null terminators*/ -/* and decoding ampersand sequences. If you don't want this, copy the data and*/ -/* pass in the copy. Returns NULL on failure.*/ +/* Given a string of xml data and its length, parses it and creates an ezxml */ +/* structure. For efficiency, modifies the data by adding null terminators */ +/* and decoding ampersand sequences. If you don't want this, copy the data and */ +/* pass in the copy. Returns NULL on failure. */ ezxml_t ezxml_parse_str(char *s, size_t len); -/* A wrapper for ezxml_parse_str() that accepts a file descriptor. First*/ -/* attempts to mem map the file. Failing that, reads the file into memory.*/ -/* Returns NULL on failure.*/ -ezxml_t ezxml_parse_fd(int fd); - -/* a wrapper for ezxml_parse_fd() that accepts a file name*/ -ezxml_t ezxml_parse_file(const char *file); +/* A wrapper for ezxml_parse_str() that accepts a file descriptor. First */ +/* attempts to mem map the file. Failing that, reads the file into memory. */ +/* Returns NULL on failure. */ -/* Wrapper for ezxml_parse_str() that accepts a file stream. Reads the entire*/ -/* stream into memory and then parses it. For xml files, use ezxml_parse_file()*/ -/* or ezxml_parse_fd()*/ -ezxml_t ezxml_parse_fp(FILE *fp); + +/* Wrapper for ezxml_parse_str() that accepts a file stream. Reads the entire */ -/* returns the first child tag (one level deeper) with the given name or NULL*/ -/* if not found*/ +/* returns the first child tag (one level deeper) with the given name or NULL */ +/* if not found */ ezxml_t ezxml_child(ezxml_t xml, const char *name); -/* returns the next tag of the same name in the same section and depth or NULL*/ -/* if not found*/ +/* returns the next tag of the same name in the same section and depth or NULL */ +/* if not found */ #define ezxml_next(xml) ((xml) ? xml->next : NULL) -/* Returns the Nth tag with the same name in the same section at the same depth*/ -/* or NULL if not found. An index of 0 returns the tag given.*/ +/* Returns the Nth tag with the same name in the same section at the same depth */ +/* or NULL if not found. An index of 0 returns the tag given. */ ezxml_t ezxml_idx(ezxml_t xml, int idx); -/* returns the name of the given tag*/ +/* returns the name of the given tag */ #define ezxml_name(xml) ((xml) ? xml->name : NULL) -/* returns the given tag's character content or empty string if none*/ +/* returns the given tag's character content or empty string if none */ #define ezxml_txt(xml) ((xml) ? xml->txt : "") -/* returns the value of the requested tag attribute, or NULL if not found*/ +/* returns the value of the requested tag attribute, or NULL if not found */ const char *ezxml_attr(ezxml_t xml, const char *attr); -/* Traverses the ezxml sturcture to retrieve a specific subtag. Takes a*/ -/* variable length list of tag names and indexes. The argument list must be*/ -/* terminated by either an index of -1 or an empty string tag name. Example: */ -/* title = ezxml_get(library, "shelf", 0, "book", 2, "title", -1);*/ -/* This retrieves the title of the 3rd book on the 1st shelf of library.*/ -/* Returns NULL if not found.*/ +/* Traverses the ezxml sturcture to retrieve a specific subtag. Takes a */ +/* variable length list of tag names and indexes. The argument list must be */ +/* terminated by either an index of -1 or an empty string tag name. Example: */ +/* title = ezxml_get(library, "shelf", 0, "book", 2, "title", -1); */ +/* This retrieves the title of the 3rd book on the 1st shelf of library. */ +/* Returns NULL if not found. */ ezxml_t ezxml_get(ezxml_t xml, ...); -/* Converts an ezxml structure back to xml. Returns a string of xml data that*/ -/* must be freed.*/ +/* Converts an ezxml structure back to xml. Returns a string of xml data that */ +/* must be freed. */ char *ezxml_toxml(ezxml_t xml); -/* returns a NULL terminated array of processing instructions for the given*/ -/* target*/ +/* returns a NULL terminated array of processing instructions for the given */ +/* target */ const char **ezxml_pi(ezxml_t xml, const char *target); -/* frees the memory allocated for an ezxml structure*/ +/* frees the memory allocated for an ezxml structure */ void ezxml_free(ezxml_t xml); - -/* returns parser error message or empty string if none*/ + +/* returns parser error message or empty string if none */ const char *ezxml_error(ezxml_t xml); -/* returns a new empty ezxml structure with the given root tag name*/ +/* returns a new empty ezxml structure with the given root tag name */ ezxml_t ezxml_new(const char *name); -/* wrapper for ezxml_new() that strdup()s name*/ +/* wrapper for ezxml_new() that strdup()s name */ #define ezxml_new_d(name) ezxml_set_flag(ezxml_new(strdup(name)), EZXML_NAMEM) -/* Adds a child tag. off is the offset of the child tag relative to the start*/ -/* of the parent tag's character content. Returns the child tag.*/ +/* Adds a child tag. off is the offset of the child tag relative to the start */ +/* of the parent tag's character content. Returns the child tag. */ ezxml_t ezxml_add_child(ezxml_t xml, const char *name, size_t off); -/* wrapper for ezxml_add_child() that strdup()s name*/ +/* wrapper for ezxml_add_child() that strdup()s name */ #define ezxml_add_child_d(xml, name, off) \ ezxml_set_flag(ezxml_add_child(xml, strdup(name), off), EZXML_NAMEM) -/* sets the character content for the given tag and returns the tag*/ +/* sets the character content for the given tag and returns the tag */ ezxml_t ezxml_set_txt(ezxml_t xml, const char *txt); -/* wrapper for ezxml_set_txt() that strdup()s txt*/ +/* wrapper for ezxml_set_txt() that strdup()s txt */ #define ezxml_set_txt_d(xml, txt) \ ezxml_set_flag(ezxml_set_txt(xml, strdup(txt)), EZXML_TXTM) -/* Sets the given tag attribute or adds a new attribute if not found. A value*/ -/* of NULL will remove the specified attribute. Returns the tag given.*/ +/* Sets the given tag attribute or adds a new attribute if not found. A value */ +/* of NULL will remove the specified attribute. Returns the tag given. */ ezxml_t ezxml_set_attr(ezxml_t xml, const char *name, const char *value); -/* Wrapper for ezxml_set_attr() that strdup()s name/value. Value cannot be NULL*/ +/* Wrapper for ezxml_set_attr() that strdup()s name/value. Value cannot be NULL */ #define ezxml_set_attr_d(xml, name, value) \ ezxml_set_attr(ezxml_set_flag(xml, EZXML_DUP), strdup(name), strdup(value)) -/* sets a flag for the given tag and returns the tag*/ +/* sets a flag for the given tag and returns the tag */ ezxml_t ezxml_set_flag(ezxml_t xml, short flag); -/* removes a tag along with its subtags without freeing its memory*/ +/* removes a tag along with its subtags without freeing its memory */ ezxml_t ezxml_cut(ezxml_t xml); -/* inserts an existing tag into an ezxml structure*/ +/* inserts an existing tag into an ezxml structure */ ezxml_t ezxml_insert(ezxml_t xml, ezxml_t dest, size_t off); -/* Moves an existing tag to become a subtag of dest at the given offset from*/ -/* the start of dest's character content. Returns the moved tag.*/ +/* Moves an existing tag to become a subtag of dest at the given offset from */ +/* the start of dest's character content. Returns the moved tag. */ #define ezxml_move(xml, dest, off) ezxml_insert(ezxml_cut(xml), dest, off) -/* removes a tag along with all its subtags*/ +/* removes a tag along with all its subtags */ #define ezxml_remove(xml) ezxml_free(ezxml_cut(xml)) #ifdef __cplusplus } #endif -#endif /* _EZXML_H*/ +#endif /* _EZXML_H */ From 9f1ce6e912928c91dfe46e315dd42de95abdb0c9 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 5 Nov 2021 13:13:22 -0600 Subject: [PATCH 06/12] debug mode --- ncdap_test/Makefile.am | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ncdap_test/Makefile.am b/ncdap_test/Makefile.am index 875c9fa5e2..0cd7726284 100644 --- a/ncdap_test/Makefile.am +++ b/ncdap_test/Makefile.am @@ -7,9 +7,9 @@ include $(top_srcdir)/lib_flags.am # Un comment to use a more verbose test driver -#SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -#LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -#TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose # Note which tests depend on other tests. Necessary for make -j check. TEST_EXTENSIONS = .sh From 6f5f0654659d29529a1207b2e2ee63600ad06af3 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 5 Nov 2021 13:32:59 -0600 Subject: [PATCH 07/12] debug off --- ncdap_test/Makefile.am | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/ncdap_test/Makefile.am b/ncdap_test/Makefile.am index 0cd7726284..875c9fa5e2 100644 --- a/ncdap_test/Makefile.am +++ b/ncdap_test/Makefile.am @@ -7,9 +7,9 @@ include $(top_srcdir)/lib_flags.am # Un comment to use a more verbose test driver -SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +#SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +#LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +#TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose # Note which tests depend on other tests. Necessary for make -j check. TEST_EXTENSIONS = .sh From 0e7e87aef357f2b9355f5d3180c3481e27ec0d5b Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 26 Nov 2021 19:57:48 -0700 Subject: [PATCH 08/12] Add extra test (see PR https://github.com/Unidata/netcdf-c/pull/2154) --- .github/workflows/run_tests.yml | 2 +- libnczarr/zfilter.c | 8 ++-- libnczarr/ztracedispatch.h | 9 ++-- ncdap_test/findtestserver.c.in | 2 +- nczarr_test/test_quantize.c | 82 +++++++++++++++++++++++++++++++++ 5 files changed, 92 insertions(+), 11 deletions(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index b1b723a99f..ddbbf03173 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -4,7 +4,7 @@ name: Run netCDF Tests -on: [pull_request] +on: [pull_request,push] jobs: diff --git a/libnczarr/zfilter.c b/libnczarr/zfilter.c index 9039d45c29..257f00cfdd 100644 --- a/libnczarr/zfilter.c +++ b/libnczarr/zfilter.c @@ -833,9 +833,11 @@ NCZ_filter_jsonize(const NC_FILE_INFO_T* file, const NC_VAR_INFO_T* var, NCZ_Fil ZTRACE(6,"var=%s filter=%s",var->hdr.name,(filter != NULL && filter->codec.id != NULL?filter->codec.id:"null")); - /* assumptions */ - assert(filter->flags & FLAG_WORKING); - + /* Ensure working parameters */ + if((filter->flags & FLAG_WORKING)==0) { + if((stat = ensure_working(var,filter))) goto done; + } + /* Convert the HDF5 id + parameters to the codec form */ /* We need to ensure the the current visible parameters are defined and had the opportunity to come diff --git a/libnczarr/ztracedispatch.h b/libnczarr/ztracedispatch.h index 29e9bc457a..1051c66f6a 100644 --- a/libnczarr/ztracedispatch.h +++ b/libnczarr/ztracedispatch.h @@ -451,7 +451,6 @@ static int NCZTR_put_vars(int ncid, int varid, const size_t *startp, const size_ return ZUNTRACE(stat); } -#if 0 static int NCZTR_def_var_deflate(int ncid, int varid, int shuffle, int deflate, int level) { int stat = NC_NOERR; @@ -475,8 +474,6 @@ static int NCZTR_def_var_filter(int ncid, int varid, unsigned int id, size_t npa return (stat); } -#endif - static const NC_Dispatch NCZ_dispatcher_trace = { NC_FORMATX_NCZARR, @@ -557,11 +554,11 @@ static const NC_Dispatch NCZ_dispatcher_trace = { NC_NOTNC4_inq_enum_member, NC_NOTNC4_inq_enum_ident, NC_NOTNC4_def_opaque, - NC_NOTNC4_def_var_deflate, - NC_NOTNC4_def_var_fletcher32, + NCZTR_def_var_deflate, + NCZTR_def_var_fletcher32, NCZTR_def_var_chunking, NCZTR_def_var_endian, - NCZ_def_var_filter, + NCZTR_def_var_filter, NCZTR_set_var_chunk_cache, NCZTR_get_var_chunk_cache, NCZTR_inq_var_filter_ids, diff --git a/ncdap_test/findtestserver.c.in b/ncdap_test/findtestserver.c.in index cbe3fdb993..5a2c0c4b3e 100644 --- a/ncdap_test/findtestserver.c.in +++ b/ncdap_test/findtestserver.c.in @@ -87,7 +87,7 @@ main(int argc, char** argv) } url = nc_findtestserver(servlet,serverlist); if(url == NULL) { - url = ""; + url = strdup(""); fprintf(stderr,"not found: %s\n",servlet); } printf("%s",url); diff --git a/nczarr_test/test_quantize.c b/nczarr_test/test_quantize.c index 23685633c4..9e17938c18 100644 --- a/nczarr_test/test_quantize.c +++ b/nczarr_test/test_quantize.c @@ -849,5 +849,87 @@ main(int argc, char **argv) } } SUMMARIZE_ERR; + printf("*** Nice, simple example of using BitGroom plus zlib..."); + { +#define DIM_LEN_SIMPLE 100 +#define EPSILON .1 + int ncid; + int dimid; + int varid1, varid2; + float *float_data; + double *double_data; + int i; + + /* Set up some data to write. */ + if (!(float_data = malloc(DIM_LEN_SIMPLE * sizeof(float)))) + ERR; + if (!(double_data = malloc(DIM_LEN_SIMPLE * sizeof(double)))) + ERR; + for (i = 0; i < DIM_LEN_SIMPLE; i++) + { + float_data[i] = 1.5 * i; + double_data[i] = 1.5 * i; + } + + /* Create the file. */ + if (nc_create(file_name, NC_NETCDF4, &ncid)) ERR; + + /* Add one dimension. */ + if (nc_def_dim(ncid, "dim1", DIM_LEN_SIMPLE, &dimid)) ERR; + + /* Create two variables, one float, one double. Quantization + * may only be applied to floating point data. */ + if (nc_def_var(ncid, "var1", NC_FLOAT, NDIM1, &dimid, &varid1)) ERR; + if (nc_def_var(ncid, "var2", NC_DOUBLE, NDIM1, &dimid, &varid2)) ERR; + + /* Set up quantization. This will not make the data any + * smaller, unless compression is also turned on. In this + * case, we will set 3 significant digits. */ + if (nc_def_var_quantize(ncid, varid1, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + if (nc_def_var_quantize(ncid, varid2, NC_QUANTIZE_BITGROOM, NSD_3)) ERR; + + /* Set up zlib compression. This will work better because the + * data are quantized, yielding a smaller output file. We will + * set compression level to 1, which is usually the best + * choice. */ + if (nc_def_var_deflate(ncid, varid1, 0, 1, 1)) ERR; + if (nc_def_var_deflate(ncid, varid2, 0, 1, 1)) ERR; + + /* Write the data. */ + if (nc_put_var_float(ncid, varid1, float_data)) ERR; + if (nc_put_var_double(ncid, varid2, double_data)) ERR; + + /* Close the file. */ + if (nc_close(ncid)) ERR; + + /* Check the resulting file for correctness. */ + { + float float_data_in[DIM_LEN_SIMPLE]; + double double_data_in[DIM_LEN_SIMPLE]; + + /* Now reopen the file and check. */ + if (nc_open(file_name, NC_NETCDF4, &ncid)) ERR; + + /* Read the data. */ + if (nc_get_var_float(ncid, varid1, float_data_in)) ERR; + if (nc_get_var_double(ncid, varid2, double_data_in)) ERR; + + for (i = 0; i < DIM_LEN_SIMPLE; i++) + { + if (abs(float_data_in[i] - float_data[i]) > EPSILON) + ERR; + if (abs(double_data_in[i] - double_data[i]) > EPSILON) + ERR; + } + + /* Close the file. */ + if (nc_close(ncid)) ERR; + } + + /* Free resources. */ + free(float_data); + free(double_data); + } + SUMMARIZE_ERR; FINAL_RESULTS; } From d2f5de872605a63fc0fe8f92bd58755bea88384c Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 26 Nov 2021 19:58:23 -0700 Subject: [PATCH 09/12] remove debug --- .github/workflows/run_tests.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/run_tests.yml b/.github/workflows/run_tests.yml index ddbbf03173..b1b723a99f 100644 --- a/.github/workflows/run_tests.yml +++ b/.github/workflows/run_tests.yml @@ -4,7 +4,7 @@ name: Run netCDF Tests -on: [pull_request,push] +on: [pull_request] jobs: From b89147386a07868d05c83e8fbc2ec92881efc8f0 Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 26 Nov 2021 19:59:51 -0700 Subject: [PATCH 10/12] remove some more debug code --- nczarr_test/run_interop.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nczarr_test/run_interop.sh b/nczarr_test/run_interop.sh index 2ff026155f..4782934ed6 100755 --- a/nczarr_test/run_interop.sh +++ b/nczarr_test/run_interop.sh @@ -80,8 +80,8 @@ case "$zext" in esac } -#testallcases file -#if test "x$FEATURE_NCZARR_ZIP" = xyes ; then testallcases zip; fi +testallcases file +if test "x$FEATURE_NCZARR_ZIP" = xyes ; then testallcases zip; fi if test "x$FEATURE_S3TESTS" = xyes ; then testallcases s3; fi exit From a49e875d507f168696744522a0c099edd3ec06bf Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 26 Nov 2021 20:30:05 -0700 Subject: [PATCH 11/12] verify failure --- ncdap_test/Makefile.am | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/ncdap_test/Makefile.am b/ncdap_test/Makefile.am index 875c9fa5e2..ec63e0fd62 100644 --- a/ncdap_test/Makefile.am +++ b/ncdap_test/Makefile.am @@ -7,9 +7,10 @@ include $(top_srcdir)/lib_flags.am # Un comment to use a more verbose test driver -#SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -#LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -#TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +TESTS_ENVIRONMENT += export SETX=1; # Note which tests depend on other tests. Necessary for make -j check. TEST_EXTENSIONS = .sh From b19106093dcd07ebb0a5d59094edbb52b4cb06ab Mon Sep 17 00:00:00 2001 From: Dennis Heimbigner Date: Fri, 26 Nov 2021 21:28:43 -0700 Subject: [PATCH 12/12] could not verify --- ncdap_test/Makefile.am | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/ncdap_test/Makefile.am b/ncdap_test/Makefile.am index ec63e0fd62..25b19cde26 100644 --- a/ncdap_test/Makefile.am +++ b/ncdap_test/Makefile.am @@ -7,10 +7,10 @@ include $(top_srcdir)/lib_flags.am # Un comment to use a more verbose test driver -SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose -TESTS_ENVIRONMENT += export SETX=1; +#SH_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +#LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +#TEST_LOG_DRIVER = $(SHELL) $(top_srcdir)/test-driver-verbose +#TESTS_ENVIRONMENT = export SETX=1; # Note which tests depend on other tests. Necessary for make -j check. TEST_EXTENSIONS = .sh