Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix nccopy chunking to use default chunking #1764

Merged
merged 2 commits into from
Jun 25, 2020
Merged

Fix nccopy chunking to use default chunking #1764

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c195e24
Fix nccopy chunking to use default chunking
DennisHeimbigner Jun 24, 2020
04dbf30
Update release notes
DennisHeimbigner Jun 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions RELEASE_NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ This file contains a high-level description of this package's evolution. Release

## 4.8.0 - TBD

* [Bug Fix] Fix nccopy to properly set default chunking parameters when not otherwise specified. This can significantly improve performance in selected cases. Note that if seeing slow performance with nccopy, then, as a work-around, specifically set the chunking parameters. [https://github.com/Unidata/netcdf-c/issues/1763].
* [Bug Fix] Fix some protocol bugs/differences between the netcdf-c library and the OPeNDAP Hyrax server. Also cleanup checksum handling [https://github.com/Unidata/netcdf-c/issues/1712].
* [Bug Fix] Add necessary __declspec declarations to allow compilation
of netcdf library without causing errors or (_declspec related)
Expand Down
26 changes: 23 additions & 3 deletions ncdump/nccopy.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -905,6 +905,7 @@ copy_chunking(int igrp, int i_varid, int ogrp, int o_varid, int ndims, int inkin
size_t ichunkp[NC_MAX_VAR_DIMS];
size_t ochunkp[NC_MAX_VAR_DIMS];
size_t dimlens[NC_MAX_VAR_DIMS];
size_t dfaltchunkp[NC_MAX_VAR_DIMS]; /* default chunking for ovarid */
int is_unlimited = 0;

/* First, check the file kinds */
Expand All @@ -914,6 +915,7 @@ copy_chunking(int igrp, int i_varid, int ogrp, int o_varid, int ndims, int inkin
memset(ichunkp,0,sizeof(ichunkp));
memset(ochunkp,0,sizeof(ochunkp));
memset(dimlens,0,sizeof(dimlens));
memset(dfaltchunkp,0,sizeof(dfaltchunkp));

/* Get the chunking, if any, on the current input variable */
if(innc4) {
Expand Down Expand Up @@ -1000,12 +1002,30 @@ copy_chunking(int igrp, int i_varid, int ogrp, int o_varid, int ndims, int inkin
dimlens[idim] = mb4dimsize;
}
}
/* compute the final ochunksizes: precedence is output, input, dimeln */

/* Get the current default chunking on the output variable */
/* Unfortunately, there is no way to get this info except by
forcing chunking */
if(ocontig == NC_CHUNKED) {
/* this may fail if chunking is not possible, in which case ignore */
int ret = nc_def_var_chunking(ogrp, o_varid, NC_CHUNKED, dfaltchunkp);
if(ret == NC_NOERR) {
int storage;
NC_CHECK(nc_inq_var_chunking(ogrp, o_varid, &storage, dfaltchunkp));
if(storage != NC_CHUNKED) return NC_EINTERNAL;
}
}

/* compute the final ochunksizes: precedence is output, input, defaults, dimelen */
for(idim = 0; idim < ndims; idim++) {
if(ochunkp[idim] == 0) {
if(ichunkp[idim] != 0)
ochunkp[idim] = ichunkp[idim];
}
if(ochunkp[idim] == 0) {
if(dfaltchunkp[idim] != 0)
ochunkp[idim] = dfaltchunkp[idim];
}
if(ochunkp[idim] == 0) {
if(dimlens[idim] != 0)
ochunkp[idim] = dimlens[idim];
Expand Down Expand Up @@ -1462,6 +1482,7 @@ copy_vars(int igrp, int ogrp)
return stat;
}

#if 0
static void
report(int rank, size_t* start, size_t* count, void* buf)
{
Expand All @@ -1482,7 +1503,7 @@ report(int rank, size_t* start, size_t* count, void* buf)
fprintf(stderr,"\n");
fflush(stderr);
}

#endif

/* Copy the schema in a group and all its subgroups, recursively, from
* group igrp in input to parent group ogrp in destination. Use
Expand Down Expand Up @@ -1636,7 +1657,6 @@ copy_var_data(int igrp, int varid, int ogrp) {
* subsequent calls. */
while((ntoget = nc_next_iter(iterp, start, count)) > 0) {
NC_CHECK(nc_get_vara(igrp, varid, start, count, buf));
report(iterp->rank,start,count,buf);
NC_CHECK(nc_put_vara(ogrp, ovarid, start, count, buf));
#ifdef USE_NETCDF4
/* we have to explicitly free values for strings and vlens */
Expand Down