nco_cmip5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quick view and manipulations on data using NetCDF Operators (NCO)\n",
    "\n",
    "\n",
    "In this notebook we demonstrate how to look at the metadata and content of the files in the CMIP5 datasets using NetCDF operators (NCO).\n",
    "\n",
    "* ncdump\n",
    "* ncks\n",
    "* ncview \n",
    "\n",
    "This example uses Coupled Model Intercomparison Project (CMIP5) collections. For more information, please visit [data catalogue](https://geonetwork.nci.org.au/geonetwork/srv/eng/catalog.search#/metadata/f3525_9322_8600_7716) and note the terms of use.\n",
    "\n",
    "---------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Authors: NCI Virtual Research Environment Team\n",
    "- Keywords: NCO, CMIP, NetCDF\n",
    "- Create Date: 2020-Apr\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is NCO?\n",
    "\n",
    "The netCDF Operators (NCO) comprise about a dozen standalone, command line programs that take netCDF, HDF files, or DAP services as input, then operate (e.g., derive new fields, compute statistics, print, hyperslab, manipulate metadata, regrid) and output the results to screen or files in text, binary, or netCDF formats. NCO aids analysis of gridded and unstructured scientific data. The Unix shell commands of NCO allows users to manipulate and analyze files interactively, or with expressive scripts that avoid some overhead of higher-level programming environments."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ncdump\n",
    "\n",
    "The NetCDF command `ncdump` is used to output a text representation of a NetCDF file's contents. More detailed information can be found from [Unidata website](http://www.unidata.ucar.edu/software/netcdf/netcdf-4/newdocs/ncdump-man-1.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, test that `ncdump` can be run from the command line. This will display a standard help list of its options:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ncdump [-c|-h] [-v ...] [[-b|-f] [c|f]] [-l len] [-n name] [-p n[,n]] [-k] [-x] [-s] [-t|-i] [-g ...] [-w] [-Ln] file\n",
      "  [-c]             Coordinate variable data and header information\n",
      "  [-h]             Header information only, no data\n",
      "  [-v var1[,...]]  Data for variable(s) <var1>,... only\n",
      "  [-b [c|f]]       Brief annotations for C or Fortran indices in data\n",
      "  [-f [c|f]]       Full annotations for C or Fortran indices in data\n",
      "  [-l len]         Line length maximum in data section (default 80)\n",
      "  [-n name]        Name for netCDF (default derived from file name)\n",
      "  [-p n[,n]]       Display floating-point values with less precision\n",
      "  [-k]             Output kind of netCDF file\n",
      "  [-s]             Output special (virtual) attributes\n",
      "  [-t]             Output time data as date-time strings\n",
      "  [-i]             Output time data as date-time strings with ISO-8601 'T' separator\n",
      "  [-g grp1[,...]]  Data and metadata for group(s) <grp1>,... only\n",
      "  [-w]             With client-side caching of variables for DAP URLs\n",
      "  [-x]             Output XML (NcML) instead of CDL\n",
      "  [-Xp]            Unconditionally suppress output of the properties attribute\n",
      "  [-Ln]            Set log level to n (>= 0); ignore if logging not enabled.\n",
      "  file             Name of netCDF file (or URL if DAP access enabled)\n",
      "netcdf library version 4.7.0 of Aug 15 2019 22:03:10 $\n"
     ]
    }
   ],
   "source": [
    "!ncdump"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Basic Usage:**\n",
    "\n",
    "```\n",
    "ncdump <option> <file_path>/<url>\n",
    "```\n",
    "or \n",
    "```\n",
    "ncdump <option> <url>\n",
    "```\n",
    "\n",
    "**View basic file header (or metadata) information**\n",
    "The `-h` option (for 'header')\n",
    "\n",
    "Use the `-h` option to view the netcdf file header information.\n",
    "\n",
    "This will display\n",
    "1. file dimensions\n",
    "2. variables and variable metadata\n",
    "3. global metadata. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netcdf pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012 {\n",
      "dimensions:\n",
      "\ttime = UNLIMITED ; // (1140 currently)\n",
      "\tj = 300 ;\n",
      "\ti = 360 ;\n",
      "\tbnds = 2 ;\n",
      "\tvertices = 4 ;\n",
      "variables:\n",
      "\tdouble time(time) ;\n",
      "\t\ttime:bounds = \"time_bnds\" ;\n",
      "\t\ttime:units = \"days since 0001-01-01\" ;\n",
      "\t\ttime:calendar = \"proleptic_gregorian\" ;\n",
      "\t\ttime:axis = \"T\" ;\n",
      "\t\ttime:long_name = \"time\" ;\n",
      "\t\ttime:standard_name = \"time\" ;\n",
      "\tdouble time_bnds(time, bnds) ;\n",
      "\tint j(j) ;\n",
      "\t\tj:units = \"1\" ;\n",
      "\t\tj:long_name = \"cell index along second dimension\" ;\n",
      "\tint i(i) ;\n",
      "\t\ti:units = \"1\" ;\n",
      "\t\ti:long_name = \"cell index along first dimension\" ;\n",
      "\tfloat lat(j, i) ;\n",
      "\t\tlat:standard_name = \"latitude\" ;\n",
      "\t\tlat:long_name = \"latitude coordinate\" ;\n",
      "\t\tlat:units = \"degrees_north\" ;\n",
      "\t\tlat:bounds = \"lat_vertices\" ;\n",
      "\tfloat lon(j, i) ;\n",
      "\t\tlon:standard_name = \"longitude\" ;\n",
      "\t\tlon:long_name = \"longitude coordinate\" ;\n",
      "\t\tlon:units = \"degrees_east\" ;\n",
      "\t\tlon:bounds = \"lon_vertices\" ;\n",
      "\tfloat lat_vertices(j, i, vertices) ;\n",
      "\t\tlat_vertices:units = \"degrees_north\" ;\n",
      "\tfloat lon_vertices(j, i, vertices) ;\n",
      "\t\tlon_vertices:units = \"degrees_east\" ;\n",
      "\tfloat pr(time, j, i) ;\n",
      "\t\tpr:standard_name = \"rainfall_flux\" ;\n",
      "\t\tpr:long_name = \"Rainfall Flux where Ice Free Ocean over Sea\" ;\n",
      "\t\tpr:comment = \"This is not the rainfall flux just over the ice-free portion of the ocean. It also contains the flux of water between the sea-ice and ocean.\" ;\n",
      "\t\tpr:units = \"kg m-2 s-1\" ;\n",
      "\t\tpr:original_units = \"(kg/m^3)*(m/sec)\" ;\n",
      "\t\tpr:history = \"2012-10-26T00:27:19Z altered by CMOR: Converted units from \\'(kg/m^3)*(m/sec)\\' to \\'kg m-2 s-1\\'. 2012-10-26T00:27:19Z altered by CMOR: replaced missing value flag (-1e+20) with standard missing value (1e+20).\" ;\n",
      "\t\tpr:cell_methods = \"time: mean area: mean where ice_free_sea over sea\" ;\n",
      "\t\tpr:cell_measures = \"area: areacello\" ;\n",
      "\t\tpr:missing_value = 1.e+20f ;\n",
      "\t\tpr:_FillValue = 1.e+20f ;\n",
      "\t\tpr:associated_files = \"baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_ocean_fx_ACCESS1-0_rcp85_r0i0p0.nc areacello: areacello_fx_ACCESS1-0_rcp85_r0i0p0.nc\" ;\n",
      "\t\tpr:coordinates = \"lat lon\" ;\n",
      "\n",
      "// global attributes:\n",
      "\t\t:institution = \"CSIRO (Commonwealth Scientific and Industrial Research Organisation, Australia), and BOM (Bureau of Meteorology, Australia)\" ;\n",
      "\t\t:institute_id = \"CSIRO-BOM\" ;\n",
      "\t\t:experiment_id = \"rcp85\" ;\n",
      "\t\t:source = \"ACCESS1-0 2011. Atmosphere: AGCM v1.0 (N96 grid-point, 1.875 degrees EW x approx 1.25 degree NS, 38 levels); ocean: NOAA/GFDL MOM4p1 (nominal 1.0 degree EW x 1.0 degrees NS, tripolar north of 65N, equatorial refinement to 1/3 degree from 10S to 10 N, cosine dependent NS south of 25S, 50 levels); sea ice: CICE4.1 (nominal 1.0 degree EW x 1.0 degrees NS, tripolar north of 65N, equatorial refinement to 1/3 degree from 10S to 10 N, cosine dependent NS south of 25S); land: MOSES2 (1.875 degree EW x 1.25 degree NS, 4 levels\" ;\n",
      "\t\t:model_id = \"ACCESS1-0\" ;\n",
      "\t\t:forcing = \"GHG, Oz, SA, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFC11, CFC12, CFC113, HCFC22, HFC125, HFC134a)\" ;\n",
      "\t\t:parent_experiment_id = \"historical\" ;\n",
      "\t\t:parent_experiment_rip = \"r1i1p1\" ;\n",
      "\t\t:branch_time = 732311. ;\n",
      "\t\t:contact = \"The ACCESS wiki: http://wiki.csiro.au/confluence/display/ACCESS/Home. Contact Tony.Hirst@csiro.au regarding the ACCESS coupled climate model. Contact Peter.Uhe@csiro.au regarding ACCESS coupled climate model CMIP5 datasets.\" ;\n",
      "\t\t:history = \"CMIP5 compliant file produced from raw ACCESS model output using the ACCESS Post-Processor and CMOR2. 2012-10-26T00:27:19Z CMOR rewrote data to comply with CF standards and CMIP5 requirements. Mon Dec  3 14:21:15 2012: updated version number to v20121203.\" ;\n",
      "\t\t:references = \"See http://wiki.csiro.au/confluence/display/ACCESS/ACCESS+Publications\" ;\n",
      "\t\t:initialization_method = 1 ;\n",
      "\t\t:physics_version = 1 ;\n",
      "\t\t:tracking_id = \"2caae357-ac9d-4bb3-81fc-87a06ed3c824\" ;\n",
      "\t\t:version_number = \"v20121203\" ;\n",
      "\t\t:product = \"output\" ;\n",
      "\t\t:experiment = \"RCP8.5\" ;\n",
      "\t\t:frequency = \"mon\" ;\n",
      "\t\t:creation_date = \"2012-10-26T00:27:19Z\" ;\n",
      "\t\t:Conventions = \"CF-1.4\" ;\n",
      "\t\t:project_id = \"CMIP5\" ;\n",
      "\t\t:table_id = \"Table Omon (01 February 2012) 3cc40dbb8d2b9162e4c4c8fd6b2312f2\" ;\n",
      "\t\t:title = \"ACCESS1-0 model output prepared for CMIP5 RCP8.5\" ;\n",
      "\t\t:parent_experiment = \"historical\" ;\n",
      "\t\t:modeling_realm = \"ocean\" ;\n",
      "\t\t:realization = 1 ;\n",
      "\t\t:cmor_version = \"2.8.0\" ;\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!ncdump -h /g/data/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/rcp85/mon/ocean/Omon/r1i1p1/latest/pr/pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012.nc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or Use remote OPeNDAP URL to look at file header. **Note the following command of accessing a url only works on local computer, VDI and Gadi login node, not on the compute node due to restricted external access.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ncdump [-c|-h] [-v ...] [[-b|-f] [c|f]] [-l len] [-n name] [-p n[,n]] [-k] [-x] [-s] [-t|-i] [-g ...] [-w] [-Ln] file\n",
      "  [-c]             Coordinate variable data and header information\n",
      "  [-h]             Header information only, no data\n",
      "  [-v var1[,...]]  Data for variable(s) <var1>,... only\n",
      "  [-b [c|f]]       Brief annotations for C or Fortran indices in data\n",
      "  [-f [c|f]]       Full annotations for C or Fortran indices in data\n",
      "  [-l len]         Line length maximum in data section (default 80)\n",
      "  [-n name]        Name for netCDF (default derived from file name)\n",
      "  [-p n[,n]]       Display floating-point values with less precision\n",
      "  [-k]             Output kind of netCDF file\n",
      "  [-s]             Output special (virtual) attributes\n",
      "  [-t]             Output time data as date-time strings\n",
      "  [-i]             Output time data as date-time strings with ISO-8601 'T' separator\n",
      "  [-g grp1[,...]]  Data and metadata for group(s) <grp1>,... only\n",
      "  [-w]             With client-side caching of variables for DAP URLs\n",
      "  [-x]             Output XML (NcML) instead of CDL\n",
      "  [-Xp]            Unconditionally suppress output of the properties attribute\n",
      "  [-Ln]            Set log level to n (>= 0); ignore if logging not enabled.\n",
      "  file             Name of netCDF file (or URL if DAP access enabled)\n",
      "netcdf library version 4.7.0 of Aug 15 2019 22:03:10 $\n"
     ]
    }
   ],
   "source": [
    "!ncdump -h http://dapds00.nci.org.au/thredds/dodsC/rr3/CMIP5/output1/CSIRO-BOM/ACCESS1-3/historical/mon/atmos/\\\n",
    "Amon/r1i1p1/v20130325/tasmax/tasmax_Amon_ACCESS1-3_historical_r1i1p1_185001-200512.nc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can show variables using the `-v` option. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netcdf pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012 {\n",
      "dimensions:\n",
      "\ttime = UNLIMITED ; // (1140 currently)\n",
      "\tj = 300 ;\n",
      "\ti = 360 ;\n",
      "\tbnds = 2 ;\n",
      "\tvertices = 4 ;\n",
      "variables:\n",
      "\tdouble time(time) ;\n",
      "\t\ttime:bounds = \"time_bnds\" ;\n",
      "\t\ttime:units = \"days since 0001-01-01\" ;\n",
      "\t\ttime:calendar = \"proleptic_gregorian\" ;\n",
      "\t\ttime:axis = \"T\" ;\n",
      "\t\ttime:long_name = \"time\" ;\n",
      "\t\ttime:standard_name = \"time\" ;\n",
      "\tdouble time_bnds(time, bnds) ;\n",
      "\tint j(j) ;\n",
      "\t\tj:units = \"1\" ;\n",
      "\t\tj:long_name = \"cell index along second dimension\" ;\n",
      "\tint i(i) ;\n",
      "\t\ti:units = \"1\" ;\n",
      "\t\ti:long_name = \"cell index along first dimension\" ;\n",
      "\tfloat lat(j, i) ;\n",
      "\t\tlat:standard_name = \"latitude\" ;\n",
      "\t\tlat:long_name = \"latitude coordinate\" ;\n",
      "\t\tlat:units = \"degrees_north\" ;\n",
      "\t\tlat:bounds = \"lat_vertices\" ;\n",
      "\tfloat lon(j, i) ;\n",
      "\t\tlon:standard_name = \"longitude\" ;\n",
      "\t\tlon:long_name = \"longitude coordinate\" ;\n",
      "\t\tlon:units = \"degrees_east\" ;\n",
      "\t\tlon:bounds = \"lon_vertices\" ;\n",
      "\tfloat lat_vertices(j, i, vertices) ;\n",
      "\t\tlat_vertices:units = \"degrees_north\" ;\n",
      "\tfloat lon_vertices(j, i, vertices) ;\n",
      "\t\tlon_vertices:units = \"degrees_east\" ;\n",
      "\tfloat pr(time, j, i) ;\n",
      "\t\tpr:standard_name = \"rainfall_flux\" ;\n",
      "\t\tpr:long_name = \"Rainfall Flux where Ice Free Ocean over Sea\" ;\n",
      "\t\tpr:comment = \"This is not the rainfall flux just over the ice-free portion of the ocean. It also contains the flux of water between the sea-ice and ocean.\" ;\n",
      "\t\tpr:units = \"kg m-2 s-1\" ;\n",
      "\t\tpr:original_units = \"(kg/m^3)*(m/sec)\" ;\n",
      "\t\tpr:history = \"2012-10-26T00:27:19Z altered by CMOR: Converted units from \\'(kg/m^3)*(m/sec)\\' to \\'kg m-2 s-1\\'. 2012-10-26T00:27:19Z altered by CMOR: replaced missing value flag (-1e+20) with standard missing value (1e+20).\" ;\n",
      "\t\tpr:cell_methods = \"time: mean area: mean where ice_free_sea over sea\" ;\n",
      "\t\tpr:cell_measures = \"area: areacello\" ;\n",
      "\t\tpr:missing_value = 1.e+20f ;\n",
      "\t\tpr:_FillValue = 1.e+20f ;\n",
      "\t\tpr:associated_files = \"baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_ocean_fx_ACCESS1-0_rcp85_r0i0p0.nc areacello: areacello_fx_ACCESS1-0_rcp85_r0i0p0.nc\" ;\n",
      "\t\tpr:coordinates = \"lat lon\" ;\n",
      "\n",
      "// global attributes:\n",
      "\t\t:institution = \"CSIRO (Commonwealth Scientific and Industrial Research Organisation, Australia), and BOM (Bureau of Meteorology, Australia)\" ;\n",
      "\t\t:institute_id = \"CSIRO-BOM\" ;\n",
      "\t\t:experiment_id = \"rcp85\" ;\n",
      "\t\t:source = \"ACCESS1-0 2011. Atmosphere: AGCM v1.0 (N96 grid-point, 1.875 degrees EW x approx 1.25 degree NS, 38 levels); ocean: NOAA/GFDL MOM4p1 (nominal 1.0 degree EW x 1.0 degrees NS, tripolar north of 65N, equatorial refinement to 1/3 degree from 10S to 10 N, cosine dependent NS south of 25S, 50 levels); sea ice: CICE4.1 (nominal 1.0 degree EW x 1.0 degrees NS, tripolar north of 65N, equatorial refinement to 1/3 degree from 10S to 10 N, cosine dependent NS south of 25S); land: MOSES2 (1.875 degree EW x 1.25 degree NS, 4 levels\" ;\n",
      "\t\t:model_id = \"ACCESS1-0\" ;\n",
      "\t\t:forcing = \"GHG, Oz, SA, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFC11, CFC12, CFC113, HCFC22, HFC125, HFC134a)\" ;\n",
      "\t\t:parent_experiment_id = \"historical\" ;\n",
      "\t\t:parent_experiment_rip = \"r1i1p1\" ;\n",
      "\t\t:branch_time = 732311. ;\n",
      "\t\t:contact = \"The ACCESS wiki: http://wiki.csiro.au/confluence/display/ACCESS/Home. Contact Tony.Hirst@csiro.au regarding the ACCESS coupled climate model. Contact Peter.Uhe@csiro.au regarding ACCESS coupled climate model CMIP5 datasets.\" ;\n",
      "\t\t:history = \"CMIP5 compliant file produced from raw ACCESS model output using the ACCESS Post-Processor and CMOR2. 2012-10-26T00:27:19Z CMOR rewrote data to comply with CF standards and CMIP5 requirements. Mon Dec  3 14:21:15 2012: updated version number to v20121203.\" ;\n",
      "\t\t:references = \"See http://wiki.csiro.au/confluence/display/ACCESS/ACCESS+Publications\" ;\n",
      "\t\t:initialization_method = 1 ;\n",
      "\t\t:physics_version = 1 ;\n",
      "\t\t:tracking_id = \"2caae357-ac9d-4bb3-81fc-87a06ed3c824\" ;\n",
      "\t\t:version_number = \"v20121203\" ;\n",
      "\t\t:product = \"output\" ;\n",
      "\t\t:experiment = \"RCP8.5\" ;\n",
      "\t\t:frequency = \"mon\" ;\n",
      "\t\t:creation_date = \"2012-10-26T00:27:19Z\" ;\n",
      "\t\t:Conventions = \"CF-1.4\" ;\n",
      "\t\t:project_id = \"CMIP5\" ;\n",
      "\t\t:table_id = \"Table Omon (01 February 2012) 3cc40dbb8d2b9162e4c4c8fd6b2312f2\" ;\n",
      "\t\t:title = \"ACCESS1-0 model output prepared for CMIP5 RCP8.5\" ;\n",
      "\t\t:parent_experiment = \"historical\" ;\n",
      "\t\t:modeling_realm = \"ocean\" ;\n",
      "\t\t:realization = 1 ;\n",
      "\t\t:cmor_version = \"2.8.0\" ;\n",
      "data:\n",
      "\n",
      " time = 732326.5, 732356, 732385.5, 732416, 732446.5, 732477, 732507.5, \n",
      "    732538.5, 732569, 732599.5, 732630, 732660.5, 732691.5, 732721, 732750.5, \n",
      "    732781, 732811.5, 732842, 732872.5, 732903.5, 732934, 732964.5, 732995, \n",
      "    733025.5, 733056.5, 733086.5, 733116.5, 733147, 733177.5, 733208, \n",
      "    733238.5, 733269.5, 733300, 733330.5, 733361, 733391.5, 733422.5, 733452, \n",
      "    733481.5, 733512, 733542.5, 733573, 733603.5, 733634.5, 733665, 733695.5, \n",
      "    733726, 733756.5, 733787.5, 733817, 733846.5, 733877, 733907.5, 733938, \n",
      "    733968.5, 733999.5, 734030, 734060.5, 734091, 734121.5, 734152.5, 734182, \n",
      "    734211.5, 734242, 734272.5, 734303, 734333.5, 734364.5, 734395, 734425.5, \n",
      "    734456, 734486.5, 734517.5, 734547.5, 734577.5, 734608, 734638.5, 734669, \n",
      "    734699.5, 734730.5, 734761, 734791.5, 734822, 734852.5, 734883.5, 734913, \n",
      "    734942.5, 734973, 735003.5, 735034, 735064.5, 735095.5, 735126, 735156.5, \n",
      "    735187, 735217.5, 735248.5, 735278, 735307.5, 735338, 735368.5, 735399, \n",
      "    735429.5, 735460.5, 735491, 735521.5, 735552, 735582.5, 735613.5, 735643, \n",
      "    735672.5, 735703, 735733.5, 735764, 735794.5, 735825.5, 735856, 735886.5, \n",
      "    735917, 735947.5, 735978.5, 736008.5, 736038.5, 736069, 736099.5, 736130, \n",
      "    736160.5, 736191.5, 736222, 736252.5, 736283, 736313.5, 736344.5, 736374, \n",
      "    736403.5, 736434, 736464.5, 736495, 736525.5, 736556.5, 736587, 736617.5, \n",
      "    736648, 736678.5, 736709.5, 736739, 736768.5, 736799, 736829.5, 736860, \n",
      "    736890.5, 736921.5, 736952, 736982.5, 737013, 737043.5, 737074.5, 737104, \n",
      "    737133.5, 737164, 737194.5, 737225, 737255.5, 737286.5, 737317, 737347.5, \n",
      "    737378, 737408.5, 737439.5, 737469.5, 737499.5, 737530, 737560.5, 737591, \n",
      "    737621.5, 737652.5, 737683, 737713.5, 737744, 737774.5, 737805.5, 737835, \n",
      "    737864.5, 737895, 737925.5, 737956, 737986.5, 738017.5, 738048, 738078.5, \n",
      "    738109, 738139.5, 738170.5, 738200, 738229.5, 738260, 738290.5, 738321, \n",
      "    738351.5, 738382.5, 738413, 738443.5, 738474, 738504.5, 738535.5, 738565, \n",
      "    738594.5, 738625, 738655.5, 738686, 738716.5, 738747.5, 738778, 738808.5, \n",
      "    738839, 738869.5, 738900.5, 738930.5, 738960.5, 738991, 739021.5, 739052, \n",
      "    739082.5, 739113.5, 739144, 739174.5, 739205, 739235.5, 739266.5, 739296, \n",
      "    739325.5, 739356, 739386.5, 739417, 739447.5, 739478.5, 739509, 739539.5, \n",
      "    739570, 739600.5, 739631.5, 739661, 739690.5, 739721, 739751.5, 739782, \n",
      "    739812.5, 739843.5, 739874, 739904.5, 739935, 739965.5, 739996.5, 740026, \n",
      "    740055.5, 740086, 740116.5, 740147, 740177.5, 740208.5, 740239, 740269.5, \n",
      "    740300, 740330.5, 740361.5, 740391.5, 740421.5, 740452, 740482.5, 740513, \n",
      "    740543.5, 740574.5, 740605, 740635.5, 740666, 740696.5, 740727.5, 740757, \n",
      "    740786.5, 740817, 740847.5, 740878, 740908.5, 740939.5, 740970, 741000.5, \n",
      "    741031, 741061.5, 741092.5, 741122, 741151.5, 741182, 741212.5, 741243, \n",
      "    741273.5, 741304.5, 741335, 741365.5, 741396, 741426.5, 741457.5, 741487, \n",
      "    741516.5, 741547, 741577.5, 741608, 741638.5, 741669.5, 741700, 741730.5, \n",
      "    741761, 741791.5, 741822.5, 741852.5, 741882.5, 741913, 741943.5, 741974, \n",
      "    742004.5, 742035.5, 742066, 742096.5, 742127, 742157.5, 742188.5, 742218, \n",
      "    742247.5, 742278, 742308.5, 742339, 742369.5, 742400.5, 742431, 742461.5, \n",
      "    742492, 742522.5, 742553.5, 742583, 742612.5, 742643, 742673.5, 742704, \n",
      "    742734.5, 742765.5, 742796, 742826.5, 742857, 742887.5, 742918.5, 742948, \n",
      "    742977.5, 743008, 743038.5, 743069, 743099.5, 743130.5, 743161, 743191.5, \n",
      "    743222, 743252.5, 743283.5, 743313.5, 743343.5, 743374, 743404.5, 743435, \n",
      "    743465.5, 743496.5, 743527, 743557.5, 743588, 743618.5, 743649.5, 743679, \n",
      "    743708.5, 743739, 743769.5, 743800, 743830.5, 743861.5, 743892, 743922.5, \n",
      "    743953, 743983.5, 744014.5, 744044, 744073.5, 744104, 744134.5, 744165, \n",
      "    744195.5, 744226.5, 744257, 744287.5, 744318, 744348.5, 744379.5, 744409, \n",
      "    744438.5, 744469, 744499.5, 744530, 744560.5, 744591.5, 744622, 744652.5, \n",
      "    744683, 744713.5, 744744.5, 744774.5, 744804.5, 744835, 744865.5, 744896, \n",
      "    744926.5, 744957.5, 744988, 745018.5, 745049, 745079.5, 745110.5, 745140, \n",
      "    745169.5, 745200, 745230.5, 745261, 745291.5, 745322.5, 745353, 745383.5, \n",
      "    745414, 745444.5, 745475.5, 745505, 745534.5, 745565, 745595.5, 745626, \n",
      "    745656.5, 745687.5, 745718, 745748.5, 745779, 745809.5, 745840.5, 745870, \n",
      "    745899.5, 745930, 745960.5, 745991, 746021.5, 746052.5, 746083, 746113.5, \n",
      "    746144, 746174.5, 746205.5, 746235.5, 746265.5, 746296, 746326.5, 746357, \n",
      "    746387.5, 746418.5, 746449, 746479.5, 746510, 746540.5, 746571.5, 746601, \n",
      "    746630.5, 746661, 746691.5, 746722, 746752.5, 746783.5, 746814, 746844.5, \n",
      "    746875, 746905.5, 746936.5, 746966, 746995.5, 747026, 747056.5, 747087, \n",
      "    747117.5, 747148.5, 747179, 747209.5, 747240, 747270.5, 747301.5, 747331, \n",
      "    747360.5, 747391, 747421.5, 747452, 747482.5, 747513.5, 747544, 747574.5, \n",
      "    747605, 747635.5, 747666.5, 747696.5, 747726.5, 747757, 747787.5, 747818, \n",
      "    747848.5, 747879.5, 747910, 747940.5, 747971, 748001.5, 748032.5, 748062, \n",
      "    748091.5, 748122, 748152.5, 748183, 748213.5, 748244.5, 748275, 748305.5, \n",
      "    748336, 748366.5, 748397.5, 748427, 748456.5, 748487, 748517.5, 748548, \n",
      "    748578.5, 748609.5, 748640, 748670.5, 748701, 748731.5, 748762.5, 748792, \n",
      "    748821.5, 748852, 748882.5, 748913, 748943.5, 748974.5, 749005, 749035.5, \n",
      "    749066, 749096.5, 749127.5, 749157.5, 749187.5, 749218, 749248.5, 749279, \n",
      "    749309.5, 749340.5, 749371, 749401.5, 749432, 749462.5, 749493.5, 749523, \n",
      "    749552.5, 749583, 749613.5, 749644, 749674.5, 749705.5, 749736, 749766.5, \n",
      "    749797, 749827.5, 749858.5, 749888, 749917.5, 749948, 749978.5, 750009, \n",
      "    750039.5, 750070.5, 750101, 750131.5, 750162, 750192.5, 750223.5, 750253, \n",
      "    750282.5, 750313, 750343.5, 750374, 750404.5, 750435.5, 750466, 750496.5, \n",
      "    750527, 750557.5, 750588.5, 750618.5, 750648.5, 750679, 750709.5, 750740, \n",
      "    750770.5, 750801.5, 750832, 750862.5, 750893, 750923.5, 750954.5, 750984, \n",
      "    751013.5, 751044, 751074.5, 751105, 751135.5, 751166.5, 751197, 751227.5, \n",
      "    751258, 751288.5, 751319.5, 751349, 751378.5, 751409, 751439.5, 751470, \n",
      "    751500.5, 751531.5, 751562, 751592.5, 751623, 751653.5, 751684.5, 751714, \n",
      "    751743.5, 751774, 751804.5, 751835, 751865.5, 751896.5, 751927, 751957.5, \n",
      "    751988, 752018.5, 752049.5, 752079.5, 752109.5, 752140, 752170.5, 752201, \n",
      "    752231.5, 752262.5, 752293, 752323.5, 752354, 752384.5, 752415.5, 752445, \n",
      "    752474.5, 752505, 752535.5, 752566, 752596.5, 752627.5, 752658, 752688.5, \n",
      "    752719, 752749.5, 752780.5, 752810, 752839.5, 752870, 752900.5, 752931, \n",
      "    752961.5, 752992.5, 753023, 753053.5, 753084, 753114.5, 753145.5, 753175, \n",
      "    753204.5, 753235, 753265.5, 753296, 753326.5, 753357.5, 753388, 753418.5, \n",
      "    753449, 753479.5, 753510.5, 753540.5, 753570.5, 753601, 753631.5, 753662, \n",
      "    753692.5, 753723.5, 753754, 753784.5, 753815, 753845.5, 753876.5, 753906, \n",
      "    753935.5, 753966, 753996.5, 754027, 754057.5, 754088.5, 754119, 754149.5, \n",
      "    754180, 754210.5, 754241.5, 754271, 754300.5, 754331, 754361.5, 754392, \n",
      "    754422.5, 754453.5, 754484, 754514.5, 754545, 754575.5, 754606.5, 754636, \n",
      "    754665.5, 754696, 754726.5, 754757, 754787.5, 754818.5, 754849, 754879.5, \n",
      "    754910, 754940.5, 754971.5, 755001.5, 755031.5, 755062, 755092.5, 755123, \n",
      "    755153.5, 755184.5, 755215, 755245.5, 755276, 755306.5, 755337.5, 755367, \n",
      "    755396.5, 755427, 755457.5, 755488, 755518.5, 755549.5, 755580, 755610.5, \n",
      "    755641, 755671.5, 755702.5, 755732, 755761.5, 755792, 755822.5, 755853, \n",
      "    755883.5, 755914.5, 755945, 755975.5, 756006, 756036.5, 756067.5, 756097, \n",
      "    756126.5, 756157, 756187.5, 756218, 756248.5, 756279.5, 756310, 756340.5, \n",
      "    756371, 756401.5, 756432.5, 756462.5, 756492.5, 756523, 756553.5, 756584, \n",
      "    756614.5, 756645.5, 756676, 756706.5, 756737, 756767.5, 756798.5, 756828, \n",
      "    756857.5, 756888, 756918.5, 756949, 756979.5, 757010.5, 757041, 757071.5, \n",
      "    757102, 757132.5, 757163.5, 757193, 757222.5, 757253, 757283.5, 757314, \n",
      "    757344.5, 757375.5, 757406, 757436.5, 757467, 757497.5, 757528.5, 757558, \n",
      "    757587.5, 757618, 757648.5, 757679, 757709.5, 757740.5, 757771, 757801.5, \n",
      "    757832, 757862.5, 757893.5, 757923.5, 757953.5, 757984, 758014.5, 758045, \n",
      "    758075.5, 758106.5, 758137, 758167.5, 758198, 758228.5, 758259.5, 758289, \n",
      "    758318.5, 758349, 758379.5, 758410, 758440.5, 758471.5, 758502, 758532.5, \n",
      "    758563, 758593.5, 758624.5, 758654, 758683.5, 758714, 758744.5, 758775, \n",
      "    758805.5, 758836.5, 758867, 758897.5, 758928, 758958.5, 758989.5, 759019, \n",
      "    759048.5, 759079, 759109.5, 759140, 759170.5, 759201.5, 759232, 759262.5, \n",
      "    759293, 759323.5, 759354.5, 759384.5, 759414.5, 759445, 759475.5, 759506, \n",
      "    759536.5, 759567.5, 759598, 759628.5, 759659, 759689.5, 759720.5, 759750, \n",
      "    759779.5, 759810, 759840.5, 759871, 759901.5, 759932.5, 759963, 759993.5, \n",
      "    760024, 760054.5, 760085.5, 760115, 760144.5, 760175, 760205.5, 760236, \n",
      "    760266.5, 760297.5, 760328, 760358.5, 760389, 760419.5, 760450.5, 760480, \n",
      "    760509.5, 760540, 760570.5, 760601, 760631.5, 760662.5, 760693, 760723.5, \n",
      "    760754, 760784.5, 760815.5, 760845.5, 760875.5, 760906, 760936.5, 760967, \n",
      "    760997.5, 761028.5, 761059, 761089.5, 761120, 761150.5, 761181.5, 761211, \n",
      "    761240.5, 761271, 761301.5, 761332, 761362.5, 761393.5, 761424, 761454.5, \n",
      "    761485, 761515.5, 761546.5, 761576, 761605.5, 761636, 761666.5, 761697, \n",
      "    761727.5, 761758.5, 761789, 761819.5, 761850, 761880.5, 761911.5, 761941, \n",
      "    761970.5, 762001, 762031.5, 762062, 762092.5, 762123.5, 762154, 762184.5, \n",
      "    762215, 762245.5, 762276.5, 762306.5, 762336.5, 762367, 762397.5, 762428, \n",
      "    762458.5, 762489.5, 762520, 762550.5, 762581, 762611.5, 762642.5, 762672, \n",
      "    762701.5, 762732, 762762.5, 762793, 762823.5, 762854.5, 762885, 762915.5, \n",
      "    762946, 762976.5, 763007.5, 763037, 763066.5, 763097, 763127.5, 763158, \n",
      "    763188.5, 763219.5, 763250, 763280.5, 763311, 763341.5, 763372.5, 763402, \n",
      "    763431.5, 763462, 763492.5, 763523, 763553.5, 763584.5, 763615, 763645.5, \n",
      "    763676, 763706.5, 763737.5, 763767.5, 763797.5, 763828, 763858.5, 763889, \n",
      "    763919.5, 763950.5, 763981, 764011.5, 764042, 764072.5, 764103.5, 764133, \n",
      "    764162.5, 764193, 764223.5, 764254, 764284.5, 764315.5, 764346, 764376.5, \n",
      "    764407, 764437.5, 764468.5, 764498, 764527.5, 764558, 764588.5, 764619, \n",
      "    764649.5, 764680.5, 764711, 764741.5, 764772, 764802.5, 764833.5, 764863, \n",
      "    764892.5, 764923, 764953.5, 764984, 765014.5, 765045.5, 765076, 765106.5, \n",
      "    765137, 765167.5, 765198.5, 765228.5, 765258.5, 765289, 765319.5, 765350, \n",
      "    765380.5, 765411.5, 765442, 765472.5, 765503, 765533.5, 765564.5, 765594, \n",
      "    765623.5, 765654, 765684.5, 765715, 765745.5, 765776.5, 765807, 765837.5, \n",
      "    765868, 765898.5, 765929.5, 765959, 765988.5, 766019, 766049.5, 766080, \n",
      "    766110.5, 766141.5, 766172, 766202.5, 766233, 766263.5, 766294.5, 766324, \n",
      "    766353.5, 766384, 766414.5, 766445, 766475.5, 766506.5, 766537, 766567.5, \n",
      "    766598, 766628.5, 766659.5, 766689, 766718.5, 766749, 766779.5, 766810, \n",
      "    766840.5, 766871.5, 766902, 766932.5, 766963, 766993.5 ;\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!ncdump -v time /g/data/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/rcp85/mon/ocean/Omon/r1i1p1/latest/pr/pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012.nc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View special attributes such as chunking and compression information Use the combination  -hs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netcdf pr_day_CNRM-CM6-1_historical_r1i1p1f2_gr_18500101-19491231 {\n",
      "dimensions:\n",
      "\tlat = 128 ;\n",
      "\tlon = 256 ;\n",
      "\ttime = UNLIMITED ; // (36524 currently)\n",
      "\taxis_nbounds = 2 ;\n",
      "variables:\n",
      "\tdouble lat(lat) ;\n",
      "\t\tlat:axis = \"Y\" ;\n",
      "\t\tlat:standard_name = \"latitude\" ;\n",
      "\t\tlat:long_name = \"Latitude\" ;\n",
      "\t\tlat:units = \"degrees_north\" ;\n",
      "\t\tlat:_Storage = \"contiguous\" ;\n",
      "\t\tlat:_Endianness = \"little\" ;\n",
      "\tdouble lon(lon) ;\n",
      "\t\tlon:axis = \"X\" ;\n",
      "\t\tlon:standard_name = \"longitude\" ;\n",
      "\t\tlon:long_name = \"Longitude\" ;\n",
      "\t\tlon:units = \"degrees_east\" ;\n",
      "\t\tlon:_Storage = \"contiguous\" ;\n",
      "\t\tlon:_Endianness = \"little\" ;\n",
      "\tdouble time(time) ;\n",
      "\t\ttime:axis = \"T\" ;\n",
      "\t\ttime:standard_name = \"time\" ;\n",
      "\t\ttime:long_name = \"Time axis\" ;\n",
      "\t\ttime:calendar = \"gregorian\" ;\n",
      "\t\ttime:units = \"days since 1850-01-01 00:00:00\" ;\n",
      "\t\ttime:time_origin = \"1850-01-01 00:00:00\" ;\n",
      "\t\ttime:bounds = \"time_bounds\" ;\n",
      "\t\ttime:_Storage = \"chunked\" ;\n",
      "\t\ttime:_ChunkSizes = 1 ;\n",
      "\t\ttime:_Endianness = \"little\" ;\n",
      "\tdouble time_bounds(time, axis_nbounds) ;\n",
      "\t\ttime_bounds:_Storage = \"chunked\" ;\n",
      "\t\ttime_bounds:_ChunkSizes = 1, 2 ;\n",
      "\t\ttime_bounds:_Endianness = \"little\" ;\n",
      "\tfloat pr(time, lat, lon) ;\n",
      "\t\tpr:long_name = \"Precipitation\" ;\n",
      "\t\tpr:units = \"kg m-2 s-1\" ;\n",
      "\t\tpr:online_operation = \"average\" ;\n",
      "\t\tpr:cell_methods = \"area: time: mean\" ;\n",
      "\t\tpr:interval_operation = \"900 s\" ;\n",
      "\t\tpr:interval_write = \"1 d\" ;\n",
      "\t\tpr:_FillValue = 1.e+20f ;\n",
      "\t\tpr:missing_value = 1.e+20f ;\n",
      "\t\tpr:coordinates = \"\" ;\n",
      "\t\tpr:standard_name = \"precipitation_flux\" ;\n",
      "\t\tpr:description = \"at surface; includes both liquid and solid phases from all types of clouds (both large-scale and convective)\" ;\n",
      "\t\tpr:history = \"none\" ;\n",
      "\t\tpr:cell_measures = \"area: areacella\" ;\n",
      "\t\tpr:_Storage = \"chunked\" ;\n",
      "\t\tpr:_ChunkSizes = 1, 128, 256 ;\n",
      "\t\tpr:_DeflateLevel = 4 ;\n",
      "\t\tpr:_Shuffle = \"true\" ;\n",
      "\t\tpr:_Endianness = \"little\" ;\n",
      "\n",
      "// global attributes:\n",
      "\t\t:name = \"/scratch/work/voldoire/outputs/CMIP6/DECK/CNRM-CM6-1_historical_r1i1p1f2/18500101/pr_day_CNRM-CM6-1_historical_r1i1p1f2_gr_%start_date%-%end_date%\" ;\n",
      "\t\t:Conventions = \"CF-1.7 CMIP-6.2\" ;\n",
      "\t\t:creation_date = \"2018-06-20T08:19:01Z\" ;\n",
      "\t\t:description = \"CMIP6 historical\" ;\n",
      "\t\t:title = \"CNRM-CM6-1 model output prepared for CMIP6 / CMIP historical\" ;\n",
      "\t\t:activity_id = \"CMIP\" ;\n",
      "\t\t:contact = \"contact.cmip@meteo.fr\" ;\n",
      "\t\t:data_specs_version = \"01.00.21\" ;\n",
      "\t\t:dr2xml_version = \"1.10\" ;\n",
      "\t\t:experiment_id = \"historical\" ;\n",
      "\t\t:experiment = \"all-forcing simulation of the recent past\" ;\n",
      "\t\t:external_variables = \"areacella\" ;\n",
      "\t\t:forcing_index = 2 ;\n",
      "\t\t:frequency = \"day\" ;\n",
      "\t\t:further_info_url = \"https://furtherinfo.es-doc.org/CMIP6.CNRM-CERFACS.CNRM-CM6-1.historical.none.r1i1p1f2\" ;\n",
      "\t\t:grid = \"data regridded to a T127 gaussian grid (128x256 latlon) from a native atmosphere T127l reduced gaussian grid\" ;\n",
      "\t\t:grid_label = \"gr\" ;\n",
      "\t\t:nominal_resolution = \"250 km\" ;\n",
      "\t\t:initialization_index = 1 ;\n",
      "\t\t:institution_id = \"CNRM-CERFACS\" ;\n",
      "\t\t:institution = \"CNRM (Centre National de Recherches Meteorologiques, Toulouse 31057, France), CERFACS (Centre Europeen de Recherche et de Formation Avancee en Calcul Scientifique, Toulouse 31057, France)\" ;\n",
      "\t\t:license = \"CMIP6 model data produced by CNRM-CERFACS is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and at http://www.umr-cnrm.fr/cmip6/. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law.\" ;\n",
      "\t\t:mip_era = \"CMIP6\" ;\n",
      "\t\t:parent_experiment_id = \"piControl\" ;\n",
      "\t\t:parent_mip_era = \"CMIP6\" ;\n",
      "\t\t:parent_activity_id = \"CMIP\" ;\n",
      "\t\t:parent_source_id = \"CNRM-CM6-1\" ;\n",
      "\t\t:parent_time_units = \"days since 1850-01-01 00:00:00\" ;\n",
      "\t\t:parent_variant_label = \"r1i1p1f2\" ;\n",
      "\t\t:branch_method = \"standard\" ;\n",
      "\t\t:branch_time_in_parent = 0. ;\n",
      "\t\t:branch_time_in_child = 0. ;\n",
      "\t\t:physics_index = 1 ;\n",
      "\t\t:product = \"model-output\" ;\n",
      "\t\t:realization_index = 1 ;\n",
      "\t\t:realm = \"atmos\" ;\n",
      "\t\t:references = \"http://www.umr-cnrm.fr/cmip6/references\" ;\n",
      "\t\t:source = \"CNRM-CM6-1 (2017):  aerosol: prescribed monthly fields computed by TACTIC_v2 scheme atmos: Arpege 6.3 (T127; Gaussian Reduced with 24572 grid points in total distributed over 128 latitude circles (with 256 grid points per latitude circle between 30degN and 30degS reducing to 20 grid points per latitude circle at 88.9degN and 88.9degS); 91 levels; top level 78.4 km) atmosChem: OZL_v2 land: Surfex 8.0c ocean: Nemo 3.6 (eORCA1, tripolar primarily 1deg; 362 x 294 longitude/latitude; 75 levels; top grid cell 0-1 m) seaIce: Gelato 6.1\" ;\n",
      "\t\t:source_id = \"CNRM-CM6-1\" ;\n",
      "\t\t:source_type = \"AOGCM\" ;\n",
      "\t\t:sub_experiment_id = \"none\" ;\n",
      "\t\t:sub_experiment = \"none\" ;\n",
      "\t\t:table_id = \"day\" ;\n",
      "\t\t:variable_id = \"pr\" ;\n",
      "\t\t:variant_label = \"r1i1p1f2\" ;\n",
      "\t\t:EXPID = \"CNRM-CM6-1_historical_r1i1p1f2\" ;\n",
      "\t\t:CMIP6_CV_version = \"cv=6.2.3.0-7-g2019642\" ;\n",
      "\t\t:dr2xml_md5sum = \"d6225e658d7de0912fca2a4293dbe2a7\" ;\n",
      "\t\t:xios_commit = \"1442-shuffle\" ;\n",
      "\t\t:nemo_gelato_commit = \"49095b3accd5d4c_6524fe19b00467a\" ;\n",
      "\t\t:arpege_minor_version = \"6.3.2\" ;\n",
      "\t\t:tracking_id = \"hdl:21.14100/6790a7ef-5cd8-4efa-ac60-9600029612b3\" ;\n",
      "\t\t:history = \"Mon Jul 30 09:11:13 2018: ncatted -O -a tracking_id,global,m,c,hdl:21.14100/6790a7ef-5cd8-4efa-ac60-9600029612b3 /scratch/work/voldoire/outputs/CMIP6/DECK/CNRM-CM6-1_historical_r1i1p1f2/assembled/pr_day_CNRM-CM6-1_historical_r1i1p1f2_gr_18500101-19491231.nc\\nnone\" ;\n",
      "\t\t:NCO = \"\\\"4.5.5\\\"\" ;\n",
      "\t\t:_SuperblockVersion = 2 ;\n",
      "\t\t:_IsNetcdf4 = 1 ;\n",
      "\t\t:_Format = \"netCDF-4\" ;\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!ncdump -hs /g/data/oi10/replicas/CMIP6/CMIP/CNRM-CERFACS/CNRM-CM6-1/historical/r1i1p1f2/day/pr/gr/v20180917/pr_day_CNRM-CM6-1_historical_r1i1p1f2_gr_18500101-19491231.nc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ncks\n",
    "\n",
    "**ncks** is netCDF kitchen sink operator. Similar to ncdump, this command can give an overview of a netCDF file, extract certain variables, extract certain dimensions and manipulate record dimensions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view the contents of a netCDF file from either a local file or a remote data service link. **Note the following command of accessing a url only works on local computer, VDI and Gadi login node, not on the compute node due to restricted external access.**\n",
    "\n",
    "```\n",
    "$ ncks <file> | more\n",
    "$ ncks <url> | more\n",
    "\n",
    "$ ncks http://dapds00.nci.org.au/thredds/dodsC/rr3/CMIP5/output1/CSIRO-BOM/ACCESS1-3/historical/mon/atmos/\\\n",
    "Amon/r1i1p1/v20130325/tasmax/tasmax_Amon_ACCESS1-3_historical_r1i1p1_185001-200512.nc | more\n",
    "```\n",
    "\n",
    "See more [ncks examples](http://nco.sourceforge.net/nco.html#xmp_ncks) here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netcdf pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012 {\n",
      "  dimensions:\n",
      "    bnds = 2 ;\n",
      "    i = 360 ;\n",
      "    j = 300 ;\n",
      "    time = UNLIMITED ; // (1140 currently)\n",
      "    vertices = 4 ;\n",
      "\n",
      "  variables:\n",
      "    int i(i) ;\n",
      "      i:units = \"1\" ;\n",
      "      i:long_name = \"cell index along first dimension\" ;\n",
      "\n",
      "    int j(j) ;\n",
      "      j:units = \"1\" ;\n",
      "      j:long_name = \"cell index along second dimension\" ;\n",
      "\n",
      "    float lat(j,i) ;\n",
      "      lat:standard_name = \"latitude\" ;\n",
      "      lat:long_name = \"latitude coordinate\" ;\n",
      "      lat:units = \"degrees_north\" ;\n",
      "      lat:bounds = \"lat_vertices\" ;\n",
      "\n",
      "\u001b[Km--More--\u001b[m"
     ]
    }
   ],
   "source": [
    "!ncks /g/data/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/rcp85/mon/ocean/Omon/r1i1p1/latest/pr/pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012.nc | more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can view a variable using -v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "netcdf pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012 {\n",
      "  dimensions:\n",
      "    bnds = 2 ;\n",
      "    time = UNLIMITED ; // (1140 currently)\n",
      "\n",
      "  variables:\n",
      "    double time(time) ;\n",
      "      time:bounds = \"time_bnds\" ;\n",
      "      time:units = \"days since 0001-01-01\" ;\n",
      "      time:calendar = \"proleptic_gregorian\" ;\n",
      "      time:axis = \"T\" ;\n",
      "      time:long_name = \"time\" ;\n",
      "      time:standard_name = \"time\" ;\n",
      "\n",
      "    double time_bnds(time,bnds) ;\n",
      "\n",
      "  data:\n",
      "    time = 732326.5, 732356, 732385.5, 732416, 732446.5, 732477, 732507.5, 73253\n",
      "8.5, 732569, 732599.5, 732630, 732660.5, 732691.5, 732721, 732750.5, 732781, 732\n",
      "811.5, 732842, 732872.5, 732903.5, 732934, 732964.5, 732995, 733025.5, 733056.5,\n",
      " 733086.5, 733116.5, 733147, 733177.5, 733208, 733238.5, 733269.5, 733300, 73333\n",
      "0.5, 733361, 733391.5, 733422.5, 733452, 733481.5, 733512, 733542.5, 733573, 733\n",
      "603.5, 733634.5, 733665, 733695.5, 733726, 733756.5, 733787.5, 733817, 733846.5,\n",
      "\u001b[Km--More--\u001b[m"
     ]
    }
   ],
   "source": [
    "!ncks -v time /g/data/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/rcp85/mon/ocean/Omon/r1i1p1/latest/pr/pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012.nc | more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ncview\n",
    "\n",
    "**Basic Usage:**\n",
    "\n",
    "From the command line:\n",
    "\n",
    "\n",
    "```\n",
    "$ ncview <file_path>\n",
    "\n",
    "```\n",
    "\n",
    "Example:\n",
    "\n",
    "```\n",
    "$ ncview /g/data/rr3/publications/CMIP5/output1/CSIRO-BOM/ACCESS1-0/rcp85/mon/ocean/Omon/r1i1p1/latest/pr/pr_Omon_ACCESS1-0_rcp85_r1i1p1_200601-210012.nc`\n",
    "\n",
    "```\n",
    "![](images/ncview1_cmip5.png)\n",
    "\n",
    "This will display the main `ncview` window with all the available viewing options. \n",
    "\n",
    "Select a variable from the `pr` box. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A new window will display a plot of the selected variable. \n",
    "\n",
    "If the data includes a `time` dimension, use the single forward/backward arrows from the animation panel to step through time. (The double arrows will produce an animation and the 'delay' option can be used to slow down the speed at which it cycles through time).\n",
    "\n",
    "![](images/ncview2_cmip5.png)\n",
    "\n",
    "\n",
    "A second panel can be used to modify a range of plot settings. \n",
    "\n",
    "For example, you can also make the image bigger by selecting `MX1/2/3` option. See the yellow arrow in the screenshot below.\n",
    "\n",
    "![](images/ncview3_cmip5.png)\n",
    "\n",
    "The `3gauss` option can be selected to change the colour map: \n",
    "\n",
    "![](images/ncview4_cmip5.png)\n",
    "\n",
    "`  `\n",
    "\n",
    "The table below provides a quick overview of some of the useful viewing options available:\n",
    "\n",
    "| Option        | Usage         | \n",
    "| :-------------: |-------------|\n",
    "| 3gauss | Cycle through colour maps |\n",
    "| Inv P | Invert the plot | \n",
    "| Inv C | Invert the colour scale | \n",
    "| Mag X1 | Zoom in/out (right/left click) | \n",
    "| Axes | Modify axes | "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Additional command line options:**\n",
    "\n",
    "```\n",
    "useage:\n",
    "ncview [options] datafiles\n",
    "\n",
    "Options\n",
    "\t-minmax: selects how rapidly minimum and maximum\n",
    "\t\tvalues in the data files will be determined;\n",
    "\t\tby scanning every third time entry (\"-minmax fast\"),\n",
    "\t\tevery fifth time entry (\"-minmax med\"), every tenth\n",
    "\t\t(\"-minmax slow\"), or all entries (\"-minmax all\").\n",
    "\t-frames: Dump out PNG images (to make a movie, for instance)\n",
    "\t-nc: \tSpecify number of colors to use.\n",
    "\t-no1d: \tDo NOT allow 1-D variables to be displayed.\n",
    "\t-repl: \tSet default blowup type to replicate rathern than bilinear.\n",
    "\t-calendar: Specify time calendar to use, overriding value in file. Known: noleap standard gregorian 365_day 360_day.\n",
    "\t-private: Use a private colormap.\n",
    "\t-debug: Print lots of debugging info.\n",
    "\t-beep: \tRing the bell when the movie restarts at frame zero.\n",
    "\t-extra: Put some extra information on the display window.\n",
    "\t-mtitle: My title to use on the display window.\n",
    "\t-noautoflip: Do not automatically flip image, even\n",
    "\t\tif dimensions indicate that it would make sense.\n",
    "\t-w: \tprint the lack-of-warranty blurb.\n",
    "\t-small: Keep popup window as small as possible by default.\n",
    "\t-shrink_mode: Shrink image assuming integer classes, so most common\n",
    "\t\tvalue in sub-block returned instead of arithmetic mean.\n",
    "\t-listsel_max NN: max number of vars allowed before switching to menu selection\n",
    "\t-no_color_ndims: do NOT color the var selection buttons by their dimensionality\n",
    "\t-no_auto_overlay: do NOT automatically put on continental overlays\n",
    "\t-autoscale: scale color map of EACH frame to range of that frame. Note: MUCH SLOWER!\n",
    "\t-missvalrgb RRR GGG BBB: specifies 3 integers (range: 0 to 255) to use for missing value color\n",
    "\t-maxsize: specifies max size of window before scrollbars are added. Either a single\n",
    "              integer between 30 and 100 giving percentage, or two integers separated by a\n",
    "              comma giving width and height. Ex: -maxsize 75  or -maxsize 800,600\n",
    "\t-c: \tprint the copying policy.\n",
    "datafiles:\n",
    "\tYou can have up to 32 of these.  They must all be in\n",
    "\tthe same general format, or have different variables in\n",
    "\tthem.  Ncview tries its best under such circumstances.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other operators include:\n",
    "- __ncap2__: netCDF Arithmetic Processor\n",
    "- __ncatted__: netCDF Attribute Editor\n",
    "- __ncbo__: netCDF Binary Operator\n",
    "- __nces__: netCDF Ensemble Statistics\n",
    "- __ncflint__: netCDF File Interpolator\n",
    "- __ncra__: netCDF Record Averager\n",
    "- __ncwaa__: netCDF Weighted Averager\n",
    "\n",
    "Try these operators out on your netCDF files! For help on a particular operator, type <span style=\"color:red\"> man operator </span> (e.g. man ncbo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "In this example, we show some basic NCO commands for data access and manipulation on CMIP data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference\n",
    "\n",
    "http://nco.sourceforge.net/"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}