From 1889d0ff9b4601153bd073fad65c714ae51b43f3 Mon Sep 17 00:00:00 2001 From: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Date: Thu, 16 Dec 2021 14:12:31 -0500 Subject: [PATCH 1/6] Sync main branch with develop branch (#302) * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting Co-authored-by: John Krasting * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files Co-authored-by: tsjackson-noaa Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano Co-authored-by: John Krasting Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: lettie_roach Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com> --- .github/workflows/mdtf_tests.yml | 6 +++- .../mixed_layer_depth/mixed_layer_depth.py | 7 ++-- .../doc/ocn_surf_flux_diag.rst | 32 +++++++++++-------- .../ocn_surf_flux_diag/ocn_surf_flux_diag.py | 14 +++++--- src/conda/_env_synthetic_data.yml | 9 +++--- src/conda/env_python3_base.yml | 6 ++-- tests/github_actions_test_macos_set3.jsonc | 4 ++- tests/github_actions_test_ubuntu_set3.jsonc | 6 ++-- 8 files changed, 55 insertions(+), 29 deletions(-) diff --git a/.github/workflows/mdtf_tests.yml b/.github/workflows/mdtf_tests.yml index 6c88a2282..2acf9f55a 100644 --- a/.github/workflows/mdtf_tests.yml +++ b/.github/workflows/mdtf_tests.yml @@ -91,7 +91,7 @@ jobs: run: | cd ../ conda activate _MDTF_synthetic_data - pip install mdtf-test-data==1.0.4.post2 + pip install mdtf-test-data mkdir mdtf_test_data ; cd mdtf_test_data # generate the data and run unit tests mdtf_synthetic.py -c GFDL --startyear 1 --nyears 10 --unittest @@ -175,9 +175,13 @@ jobs: # jobs, but we don't want to max out resources that are shared by the NOAA-GFDL repos. curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --disable-epsv --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/temp_extremes_distshape_obs_data.tar --output temp_extremes_distshape_obs_data.tar curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --disable-epsv --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/tropical_pacific_sea_level_obs_data.tar.gz --output tropical_pacific_sea_level_obs_data.tar.gz + curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --disable-epsv --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/mixed_layer_depth_obs_data.tar --output mixed_layer_depth_obs_data.tar + curl --verbose --ipv4 --connect-timeout 8 --max-time 1200 --retry 128 --ftp-ssl --disable-epsv --ftp-pasv -u "anonymous:anonymous" ftp://ftp.gfdl.noaa.gov/perm/oar.gfdl.mdtf/ocn_surf_flux_diag_obs_data.tar --output ocn_surf_flux_diag_obs_data.tar echo "Untarring set 3 test files" tar -xvf temp_extremes_distshape_obs_data.tar tar -zxvf tropical_pacific_sea_level_obs_data.tar.gz + tar -xvf mixed_layer_depth_obs_data.tar + tar -xvf ocn_surf_flux_diag_obs_data.tar # clean up tarballs rm -f *.tar rm -f *.tar.gz diff --git a/diagnostics/mixed_layer_depth/mixed_layer_depth.py b/diagnostics/mixed_layer_depth/mixed_layer_depth.py index 9ddf41583..82c933b4a 100644 --- a/diagnostics/mixed_layer_depth/mixed_layer_depth.py +++ b/diagnostics/mixed_layer_depth/mixed_layer_depth.py @@ -42,7 +42,7 @@ # and Mixed Layer Depth in MDTF Special Issue # -# from __future__ import print_function + import os # undo these for the framework version @@ -63,6 +63,7 @@ import cartopy.feature as cfeature import gsw import time +import datetime months = [ "Jan", @@ -89,8 +90,10 @@ def readinocndata(file, varname="so", firstyr="2010", lastyr="2014"): ds = xr.open_dataset(file) + date_begin = datetime.date(int(firstyr), 1, 1).isoformat() + date_end = datetime.date(int(lastyr), 12, 31).isoformat() ds = ds.sel( - time=slice(firstyr + "-01-01", lastyr + "-12-31") + time=slice(date_begin, date_end) ) # limit to yrs of interest, maybe model dep print( "Limit domain to Arctic to match obs" diff --git a/diagnostics/ocn_surf_flux_diag/doc/ocn_surf_flux_diag.rst b/diagnostics/ocn_surf_flux_diag/doc/ocn_surf_flux_diag.rst index 4998709be..0f4d2561f 100644 --- a/diagnostics/ocn_surf_flux_diag/doc/ocn_surf_flux_diag.rst +++ b/diagnostics/ocn_surf_flux_diag/doc/ocn_surf_flux_diag.rst @@ -1,7 +1,7 @@ Ocean Surface Latent Heat Flux Diagnostic Documentation ================================ -Last update: 10/15/2021 +Last update: 12/10/2021 Tropical intra-seasonal (20-100 day) convection regulates weather patterns globally through extratropical teleconnections. Surface latent heat fluxes @@ -13,20 +13,20 @@ fluxes contribute about 8% of the intra-seasonal precipitation anomaly over the Indian and western tropical Pacific Oceans [Dellaripa and Maloney, 2015, Bui et al., 2020]. -we use in-situ data from TAO/TRITON/RAMA to create a location-based latent +For this diagnostic, we use in-situ data from TAO/TRITON/RAMA to create a location-based latent heat flux matrix determined by specific humidity deficiency at the surface layer (dq) and surface wind speed (sfcWind). By comparing the matrix between observation -and models/reanalysis, we reveal where model/reanalysis latent heat flux biases are -largest in dq-sfcWind space. The latent heat flux biases shown in the matrix +and models/reanalysis, the diagnostic can help revealing where model/reanalysis latent heat flux biases are +in the dq-sfcWind space. The latent heat flux biases shown in the diagnostic demonstrate dependence on both sfcWind and dq. An offline latent heat bias correction -can be performed on model simulations based on the bias latent heat fluxes matrix +can be performed on simulations based on the bias latent heat fluxes matrix as a function of dq and sfcWind. Version & Contact info ---------------------- -- Version/revision information: version 1 (10/15/2021) +- Version/revision information: version 2 (12/10/2021) - PI (Charlotte A. DeMott, Colorado State University, charlotte.demott@colostate.edu) - Developer/point of contact (Chia-Wei Hsu, Colorado State University, Chia-Wei.Hsu@colostate.edu) @@ -76,9 +76,7 @@ Required programming language and libraries ------------------------------------------- The programming language is python version 3 or up. The third-party libraries -include "matplotlib", "xarray", "metpy","numpy","scipy". The conda environment -need to be set to _MDTF_ocn_surf_flux_diag. - +include "matplotlib", "xarray", "metpy","numpy","scipy". Required model output variables ------------------------------- @@ -127,12 +125,20 @@ References .. _ref-Hsu: -1. C.-W. Hsu et al. (2020): Ocean Surface Flux Algorithm Effects on Tropical - Indo-Pacific Intraseasonal Precipitation. *GRL*, under review. - +1. C.-W. Hsu et al. (2021): Ocean Surface Flux Algorithm Effects on Tropical +Indo-Pacific Intraseasonal Precipitation. *GRL*, under review. More about this diagnostic -------------------------- -Surface latent heat flux from ocean to the atmosphere is one of the important processes that provides water vapor and energy to the daily tropical rainfall. A visually intuitive latent heat flux diagnostic is proposed to better understand the model shortfall on its latent heat flux representation. This diagnostic allows a simple assessment of model latent heat flux biases arising either from biases in water vapor or surface wind speed as well as other empirical coefficients in the model. Sample POD result shows that, compared to ''observed'' fluxes also estimated from water vapor and surface wind speed measured at tropical moorings, tropical latent heat fluxes in the NCAR CEMS2 models are significantly overestimated when extreme water vapor or surface wind speed happens. +Surface latent heat flux from ocean to the atmosphere is one of the important +processes that provides water vapor and energy to the daily tropical rainfall. +A visually intuitive latent heat flux diagnostic is proposed to better understand +the model shortfall on its latent heat flux representation. This diagnostic allows +a simple assessment of model latent heat flux biases arising either from biases in +water vapor or surface wind speed as well as other empirical coefficients in the model. +Sample POD result shows that, compared to ''observed'' fluxes also estimated from +water vapor and surface wind speed measured at tropical moorings, tropical latent heat +fluxes in the NCAR CEMS2 models are significantly overestimated when extreme water vapor +or surface wind speed happens. \ No newline at end of file diff --git a/diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.py b/diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.py index 8d28c6c97..c46f7347d 100644 --- a/diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.py +++ b/diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.py @@ -180,7 +180,11 @@ levels=ratiolevel, colors='w',linewidths=6) ) -ax1.clabel(cs, ratiolevel, inline=True, fmt='%0.2f', fontsize=10) + +# verify that plotted data falls within defined ratiolevel range before adding labels +# (useful when working with synthetic data) +if ds_stn_bin['hfls_count']/ds_stn_bin['hfls_count'].sum().any() in ratiolevel: + ax1.clabel(cs, ratiolevel, inline=True, fmt='%0.2f', fontsize=10) # ========== hlfs model =========== ax2 = fig.add_axes([1,0,1,0.8]) @@ -208,8 +212,11 @@ levels=ratiolevel, colors='w',linewidths=6) ) -ax2.clabel(cs, ratiolevel, inline=True, fmt='%0.2f', fontsize=10) +# verify that plotted data falls within defined ratiolevel range before adding labels +# (useful when working with synthetic data) +if ds_model_bin['hfls_count']/ds_model_bin['hfls_count'].sum().any() in ratiolevel: + ax2.clabel(cs, ratiolevel, inline=True, fmt='%0.2f', fontsize=10) ax1.set_yticks(np.arange(bin2_range[0],bin2_range[1]+1,2)) @@ -307,10 +314,9 @@ ax2.set_ylabel('',size=15) ax2.set_xlabel('10m wind speed ($m/s$)',size=15) - # output figures fig.savefig( - os.getenv("WK_DIR") + "/model/PS/LHFLXdiag_modelbias_example.eps", + os.path.join(os.getenv("WK_DIR"), "model/LHFLXdiag_modelbias_example.png"), facecolor="w", edgecolor="w", orientation="portrait", diff --git a/src/conda/_env_synthetic_data.yml b/src/conda/_env_synthetic_data.yml index 8c1070810..fdd7b8ee5 100644 --- a/src/conda/_env_synthetic_data.yml +++ b/src/conda/_env_synthetic_data.yml @@ -6,17 +6,18 @@ channels: dependencies: - python=3.9 - numpy=1.20.2 -- netCDF4=1.5.4 +- netCDF4=1.5.8 - cftime=1.3.0 - xarray>=0.17.0 - setuptools >= 49.1 -- esmpy=8.1.0 -- xesmf=0.5.3 +- esmf=8.2.0 +- esmpy=8.2.0 +- xesmf=0.6.2 - python-dateutil >= 2.8.0 - pandas < 1.3 - pytz=2020.4 - pytest >= 6.2.4 -- pip=21.1.1 +- pip=21.3.1 - pytest>=6.2.4 - pip : - https://files.pythonhosted.org/packages/6d/e7/2b1c334e313b689eb067f34edc82d083e39c01979fb3b00003c9e3cddf57/envyaml-1.8.210417-py2.py3-none-any.whl diff --git a/src/conda/env_python3_base.yml b/src/conda/env_python3_base.yml index 933769091..f6e618ee9 100644 --- a/src/conda/env_python3_base.yml +++ b/src/conda/env_python3_base.yml @@ -10,7 +10,7 @@ dependencies: - python=3.7 - numpy=1.19 - scipy=1.5.2 -- netCDF4=1.5.4 +- netCDF4=1.5.8 - cftime=1.2 - xarray=0.16 - matplotlib=3.3 @@ -20,6 +20,8 @@ dependencies: - dask=2021.03.0 - numba=0.53.1 - scikit-learn=0.24.2 -- xesmf=0.3.0 +- xesmf=0.6.2 +- esmf=8.2.0 +- esmpy=8.2.0 - gsw=3.3 - metpy=1.1 diff --git a/tests/github_actions_test_macos_set3.jsonc b/tests/github_actions_test_macos_set3.jsonc index 482ee547a..3f4d1e4ed 100644 --- a/tests/github_actions_test_macos_set3.jsonc +++ b/tests/github_actions_test_macos_set3.jsonc @@ -12,7 +12,9 @@ "LASTYR" : 10, "pod_list": [ "temp_extremes_distshape", - "tropical_pacific_sea_level" + "tropical_pacific_sea_level", + "ocn_surf_flux_diag", + "mixed_layer_depth" ] } ], diff --git a/tests/github_actions_test_ubuntu_set3.jsonc b/tests/github_actions_test_ubuntu_set3.jsonc index a953b5ef6..20e3a9b16 100644 --- a/tests/github_actions_test_ubuntu_set3.jsonc +++ b/tests/github_actions_test_ubuntu_set3.jsonc @@ -10,8 +10,10 @@ "FIRSTYR" : 1, "LASTYR" : 10, "pod_list": [ - "temp_extremes_distshape", - "tropical_pacific_sea_level" + "temp_extremes_distshape", + "tropical_pacific_sea_level", + "ocn_surf_flux_diag", + "mixed_layer_depth" ] } ], From fcaeb7ad3073afbb66441278164019ab164f0adf Mon Sep 17 00:00:00 2001 From: Aparna Radhakrishnan Date: Fri, 14 Jan 2022 10:11:55 -0500 Subject: [PATCH 2/6] Update CODEOWNERS (#308) Adding myself to codeowners file to be able to conduct code reviews going forward. --- CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CODEOWNERS b/CODEOWNERS index 9ed46dca8..d4f159285 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -1 +1 @@ -* @jkrasting @wrongkindofdoctor +* @jkrasting @wrongkindofdoctor @aradhakrishnanGFDL From 2ccde98c82a82b8a52d63bc431313fe6725d8b3f Mon Sep 17 00:00:00 2001 From: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Date: Wed, 19 Jan 2022 15:18:14 -0500 Subject: [PATCH 3/6] Merge develop branch into main branch (#310) * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting Co-authored-by: John Krasting * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files Co-authored-by: tsjackson-noaa Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano Co-authored-by: John Krasting Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: lettie_roach Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com> From cb00fe0b92d5c00dbf4acbaebbb529f180291240 Mon Sep 17 00:00:00 2001 From: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Date: Tue, 1 Mar 2022 11:00:59 -0500 Subject: [PATCH 4/6] Fix Explicit_file cli definition --- doc/sphinx/ref_data_sources.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/sphinx/ref_data_sources.rst b/doc/sphinx/ref_data_sources.rst index bcd9f6272..5be465503 100644 --- a/doc/sphinx/ref_data_sources.rst +++ b/doc/sphinx/ref_data_sources.rst @@ -47,7 +47,7 @@ When using this data source, ``-c``/``--convention`` should be set to the conven Explicit file data source +++++++++++++++++++++++++ -Selected via ``--data-manager="ExplicitFile"``. +Selected via ``--data-manager="Explicit_file"``. This data source lets the user explicitly assign model data files to each variable requested by a POD using standard shell glob syntax, without needing to move or symlink them to a directory hierarchy (as is needed for, e.g., the :ref:`ref-data-source-localfile`). Files must be on a locally mounted filesystem, and satisfy the requirements in :doc:`ref_data` (with the exception of metadata). From 23a2dcb244cf3d17a2c50b82a7bb468e3d0361e3 Mon Sep 17 00:00:00 2001 From: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Date: Thu, 3 Mar 2022 11:15:02 -0500 Subject: [PATCH 5/6] Develop (#318) * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting Co-authored-by: John Krasting * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition Co-authored-by: tsjackson-noaa Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano Co-authored-by: John Krasting Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: lettie_roach Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com> From bfd285fee67140a738a46f13e642ee48793edc31 Mon Sep 17 00:00:00 2001 From: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Date: Thu, 3 Mar 2022 11:39:54 -0500 Subject: [PATCH 6/6] Merge develop branch into main branch (#319) * add CM4 diag table to repo (#269) * add CM4 diag table to repo * swap diag table docx with trimmed-down text file * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Indicate which settings are required, in website and CLI help * Move --large-file flag higher in list of options, so that data_manager options can come last * Clarify behavior of --site and plug-in settings * Don't do full startup when only printing CLI help message * Apply changes to CLI to NOAA_GFDL site * Correct GFDL default paths to values in default_gfdl.jsonc * Do more validation on input paths * Do more input path validation in GFDL-specific code * Validate CASE_ROOT_DIR before other checks * Only validate non-empty CASE_ROOT_DIR Fixes broken CI. * Remove Tom from codeowners file * Merge of Tropical Sea Level POD (#271) * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * incorporate changes from John * using CESM2 testing * used the xr_ufunc with dask disabled load the dataset before function call * multiple updates 1. allow xaxis and yaxis name option in pod_env_var 2. allow selecting different obs start year and end year in pod_env_var 3. fix the trend plot in ylim and xlim setting * add new pod_env_var for obs year and axis name * fixed the dimension not matching in model cal * function calculate wind stress curl for obs * function calculate cell area in obs * process predefined obs mean trend and season sig * linear regression for linear trend calculation * add the predefine flag for obs to speed up cal * add the predefined obs option in the main.py * pylint and black format corrected * add transpose in functions to make sure dim order * address some LGTM error * create pod * main script update output dir still need to change to env var * altrimetry observational data preprocessing code process the daily data from CMEMS to monthly data * calculate the wind stress curl for model and obs * unfinished OMIP model variable is not in the example model * small function to show used memory * calculate the gridded area for observational data * function designed for xr.Dataset only used the da_linregress func in the main script * create naming convension for OMIP CESM2 * setting of the config file * update to relative path to with respect to $CODE_ROOT * update the file name to MDTF $CASENAME format * include the documentation of the tool * change variable name to match fieldlist * update the jsonc files still cannot run * remove unused function and repeating constant * remove redundent import and multiple defined var * clear unused function * change comment * correct start year and end year to string * change the OUTPUT_DIR setting * correct the comma at the end of the list * fixed the output dir blank * syntex error corrected * correct the jsonc format * add print out * correct the environment variable * removed commented part that is not used * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * Update diagnostics/tropical_pacific_sea_level/doc/tropical_pacific_sea_level.rst Co-authored-by: John Krasting * updated to f-string format for readability * include kwarg for earth radius for flexibility * update the html format for this particular diag * corrcet error and add references * setting file works in version 2 (bk) * add new format for version 3 setting file * unfinished changes in the main script * add observational data access info * use the version2 format for now * add the observational data info in the docstring * save the new version3 format * update preprocessing code remove the open_mfdataset function since it require dask to be installed * update from main branch * testing tropical sea level diag * update to the latest format * update to the latest format and some fix suggested by Tom * remove the previous version files * Updates to AVISO processing script - Implement faster os.scantree() for finding files - Cleaned up xarray Dataset vs. DataArray objects - Replaced for-loop with list comprehension - Save output in 32-bit floats to save data volume - Carry source variable attributes to output NetCDF file * Fixed tropical sea level jsonc syntax errors - removed trailing comma - removed "dimensions_ordered": true, line * Updates default_tests.jsonc - Renamed "CESM2" --> "CESM" - Removed hard-coded anaconda paths * Updates to sea level POD settings file - Simplified settings for debugging - Renamed nlat/nlon to lat/lon - Commented out areacello for testing * Added sea level fields to NCAR and CMIP field lists - areacello, tauuo, tauvo, and zos * Seperate testing jsonc file for NCAR Synthetic - Making life easier for testing * Fix dimension elimination for 2D lat/lon coords * Make translated var axes check more permissive - cf-xarray mod needed to understand curvilinear coordinates; until that fix is in place, this check will fail for grids that have 2-dimensional coordinates - changed fatal exception to a warning - wrapped rest of the coordinate metadata checks inside an if-block * Updated tropical sea level pod settings file * Updated NCAR field list for sea level POD - now includes tauuo and tauvo * Updated year range for default case list * Updated to tropical sea level diagnostic script - Mostly edits to opening datasets * Bypass Dask parallelization - Not working; commented out for now. * Fixed indexing for tropical sea level curl calc - indicies appear to be missing for central differences * Modified model read statements for tropical sl POD - More native Xarray logic * Patches for obs. data time issues - impacts tropical sea level pod - addresses mismatch between time axis and data length - hard-coded date ranges for now; needs to be generalized * Specify dim names before regional averaging - impacts tropical sea level pod - needs to be generalized for all multidimensional coordinate grids * Revert "Make translated var axes check more permissive" This reverts commit d68769c6429edff559c821ab3bca8bb2ed3c3c8c. * fix the missing value casued by model lon range the model lon range for the synthetic is -270~90 the original code did not convert the long range to 0-360 which causes the model to pick region that does not exist in the synthetic data * Remove unused imports from createCMIP6CV.py * Delete default_tests_ncar_synthetic.jsonc * remove user modifications from default_tests.jsonc * remove duplicate variable entries from fieldlist_CMIP.jsonc * remove changes to xr_parser.py Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add surface variable support (#266) * added modifiers.jsonc with entry for atmospheric height added modifiers att to core.VariableTranslator __init__ method added modifier att to data_model.DMDependentVariable class and the comments of classes that inherit it corrected modifier to modifiers * removed long_name att from modifiers.jsonc * added check for modfiers to DMDependentVariable __post_init method * change second temp_d index to entry.modifiers, remove invalid comment * changed argument axes_set to modfiers in translate, from_CF, and from_CF_name * added modifiers to NoTranslateFieldlist TranslateVarlistEntry, and changed axes_set to modifiers in from_CF, and from_CF_name * changed modifiers to modifier * change remaining axes_set arguments to modifier in core.py * changed modifier.jsonc to modifiers.jsonc in core.py * revert change in _ndim_to_axes_set name * added develop branch exit call changes to core.py * attempt to fix .strip .lower spec * change self.modifiers to self.modifier * add modifier att to TREFHT, tas, and t_ref in conventin jsonc files * fixed CMIP fieldlist * added modifier attribute to temperature variable in example settings.jsonc file * add core import and redefine variabletranslator call indata_model.py * remove space in description * added modifiers documentation to ref_settings.rst * add preliminary version of test_variable_translator_bad_modifier * move the fieldlist convention reads to its own method, and replace MDTFFramework.configure call to variable translator with VariableTranslator.read_convention * fix the variableTranslator and read_conventions calls * rework tests that call VariableTranslator to use read_conventions after initialization * added check that modifier is an empty string to from_CF routine removed extraneous parentheses from routines in core.py * refined modifier check to differentitate between 3d and 4d variable entries. Still needs work. * added variable dimension size and argument num_dims to from_CF * Changed default modifier arguments back to None in core.py routines changed from_CF to check if modifier is false before checking lut entries since empty strings and None types will evaluate to False * fixed changes in from_CF that got overwritten by develop branch sync * added atmos_height modifier attribute to tas in temp_extremes_distshape settings.jsonc file removed modifier assignment to frozenset in from_CF because it is a single string rather than a list * missed adding the file with the frozenset change in prior commit message * fix typo in data_model.py DMDependentVariable error message fix spacing for inline comments * fix issues with modifier test in test_core.py add a check for a correct modifier entry to modifier test * revert spacing change to data_manager.py * revert changes to default_tests.jsonc again * Update src/core.py Co-authored-by: John Krasting Co-authored-by: John Krasting * add 3D and 4D wind_speed fields to CMIP fieldlist (#285) This requires PR #266 to be merged first. * Add 3D specific humidity to CMIP fieldlist (#286) * Fix FRE integration and improve documentation (#245) * Update FRE wrapper (mtdf_gfdl.csh) Update the FRE wrapper script to use current flags. Do not load a python env module; instead run the same commands as the interactive wrapper script to invoke the site-installed conda env before running the package. Disable functionality to make a copy of output for hosting on an internal website, since updates aren't being made to Dora. Retain code for when this functionality is re-enabled. * More debugging statements in FRE wrapper * Pass through args in wrapper; set --convention default to GFDL * Add more detail on wrapper script to GFDL docs * Convention defaults for GFDL_PP, LocalFile data sources Set the default --convention assumed for SampleLocalFileDataSource to "CMIP", to be consistent with what's written in the documentation. Set the default --convention assumed for the GFDL_PP data source to "GFDL", since this is the most common use case (but not the only one). Mention this in the docs. Take logic to set this default value out of the mdtf_gfdl.csh wrapper script. * Better regex in FRE wrapper * Restore model component-specific options Allow --component and --chunk_freq flags to be passed to the GFDL_PP data source to restrict the data query to files with those attributes. Add functionality to the FRE wrapper script to handle the real use cases. Pass --multi_component to the wrapper to invoke "frepp mode" (multiple runs of the package on same data, as it's being output from postprocessing); pass --component_only to restrict the model component used. Update and expand the docs for these settings to explain all this. * Refine model component-specific logic in FRE flags Remove the --multi_component/--any_component flag from the mdtf_gfdl.csh wrapper, and replace it with --run_once. This is because the sensible default behavior for the wrapper hould be the opposite of what was previously implemented: the default scenario for when the package is called from FRE is the incremental/online processing use case ("scenario 3B" in the docs). Non-default use cases are when the package is invoked only once, after all needed data is postprocessed (scenario 3A), or when the user wants to manually restrict the operation to the given model components. These are invokes via the --run_once or --component_only flags in the wrapper script. * Update all references to GFDL site installation Remove all paths and references to the old GFDL site installation in oar.gfdl.mdteam and replace with their counterparts in the oar.gfdl.mdtf role account. * Fix .rst formatting bug in docs * More .rst tweaks * Tweaks to default data source convention specification Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change 3d wind_speed to sfcWind in CMIP fieldlist (#288) * change 3 wind_speed to sfcWind in CMIP fieldlist * change 3D hus to huss in CMIP fieldlist hus is for 4D specific humidity, while huss is for 3D (near-surface) specific humidity. * Document internal APIs (#274) * Initial commit of custom CSS for autodoc (only) Attempt to tweak spacing of elements in autodoc pages. Styling of rest of documentation site is unchanged. * Fix docstring 'Returns:' formatting * Docstring formatting fixes * Add manually written doc section to framework TOC * Tweak autodoc settings to remove unneeded output * Add more space, grouping between class docstrings * Improve mamba usage in CI - install mamba (https://github.com/mamba-org/mamba) first, and use it to speed up installation of all other conda envs, including the one for synthetic data. Note that currently only the dependency solve is accelerated (https://github.com/mamba-org/mamba/issues/633), but this is sufficient for a large improvement over conda. - Remove the --mamba flag from the conda_env_setup.sh script's arguments, as this was taken out in NOAA-GFDL/MDTF-diagnostics#166. * Add temp extremes distshape to CI (#273) * Initial commit of code from Arielle Catalano * Correct extension for settings.jsonc * Add freq=day to varlist for backwards compatibility * Commit conda env for temp_extremes_distshape * Commit conda env for temp_extremes_distshape * Run temp_extremes_distshape in python_base instead of custom env * adding new POD, documentation, and obs subset code * Fix case of CF variable names * Set Matplotlib backend in settings.jsonc * Propagate gs command-line flags to GFDL branch * Temporary debugging statements * Update temp_extremes_distshape settings.jsonc * Update format of settings.jsonc * Add missing 'log' argument to GFDL data source attribute classes * log InitVar must be 1st arg to __post_init__ due to inheritance * Restore missing class attributes to GFDL CMIP6 DataSources * Convert .pdf documentation to .rst * Fix obs filenames and templating in html * Write temp files to WK_DIR, not POD_HOME temp_extremes_distshapes code was writing temporary parameter .json files to $POD_HOME, which should be read-only. Change root directory for writing all these files from $POD_HOME to $WK_DIR. * Correct variable names * Correct method for colorbar tick labels * Handle case where no contours are labeled * Correct cartopy longitude set_extents * Correct Moments_plot colorbar positioning * Add documentation for set_extent() fix * add test jsonc files for temp_extremes_distshape CMIP * added test yaml file just for temp_extremes_distshape * add call create the inputdata directory before untarring the obs_data file to the test yaml * add checks to test yaml * Specify full paths in ubuntu set3 jsonc * remove comments from test jsonc files change CASENAME to match the format output by updated synthetic data generator in the set3 test jsonc files * update mdtf_tests.yml to include this branch for testing before PR is submitted * remove this branch from mdtf_tests.yml * Switch paths back to relative locations in ubuntu set 3 jsonc * change model name to synthetic in github actions set 3 jsonc files * changed ls to just show temp_extremes_distshape obs_data directory * fixed path in ls * added print statements for path checks * add pring statement to group_relative_links * removed print statements from output_manager and verify_links * changed test yml to experimental status and added ls for wkdir after POD runs * fix typo * remove some test lines in test_cmip.yml * change exit status for missing files to 0 for debugging in verify_links.py * remove debugging in verify_links comment out pod.deactivate call in verify_links for debugging * removed PS from the output figure path in TempExtDistShape_CircComps_usp.py bc file is png not postscript * changed ls to wildcard in test_cmip.yml * dump output log to terminal * define output figure path as separate variable with os.path.join in TempExtDistShape_ShiftRatio_util.py * revert changes to path name * change set3 env to python3 * change cartopy to version 0.19 in env_python3_base.yml * revert debugging mods to output_manager.py * remove extraneous checks from mdtf_tests.yml test yaml with MDTF_base environment for set 3 tests * replace _MDTF_python3_base with _MDTF_base in mdtf_tests.yml set 3 tests Co-authored-by: Thomas Jackson Co-authored-by: tsjackson-noaa Co-authored-by: Arielle Catalano * Abbreviate logger in function/method signatures * Set more prominent CSS highlight color * Expand docstrings for src/util * Commit manual doc for util subpackage * Stop sphinx-apidoc from importing unit tests * Fix mocking of external imports when docs are built * Sphinx CSS tweaks Increase item spacing, fix shading, add border for code literal blockquotes * Improve coverage of class attributes Set inherited-members to True to include inherited dataclass fields and simplify navigating class heirarchy. Define skip_members_handler() to remove docstrings of methods we don't want to include (eg from stdlib). * Update 'supporting modules' docstrings * Partial commit of updated docstrings for main modules * Fix documentation .rst syntax errors * Initial commit of remaining manually-generated internal API docs * Split up data source docs across multiple pages Include sphinx links to relevant docs in module summary docstrings. * More crossreferences in docs * Fix module docstrings Sphinx autosummary only includes first sentence of docstring, not first paragraph. Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano * update env_base.yml (#289) Incorporate common packages required to run mld (and other) PODs * Revert "update env_base.yml (#289)" (#290) This reverts commit 60632d3a8828a395ab9b5fb365b63b6604adcc11. * Change cf_time calendar logic (#291) * Change cf_time calendar logic * remove check for cftime calendar in tcoord.values The cftime calendar attribute is most likely present in other t_coord and ds features that are checked, so remove unnecessary assumption of calendar att in t_coord.values[0] * Add metpy to python3_base yaml (#294) The metpy package will support incoming the ocn_flux_matrix POD and others that require it * add test yaml for trop pac sl pod (#281) * add test yaml for trop pac sl pod * updated tpsl test yaml * Fix branch name * Change mdtf_test_data version to 1.0.4 * change mdtf-test-data version to 1.0.4.post1 in test yaml * add tropical pacific sea level POD to set3 test yaml * fix branch name in test yaml * fix directory location in test data untar stage * change tauuo and tauvo standard names in settings file to match names in CMIP fieldlist * fix tauuo and tauvo standard_names in tpsl settings.jsonc * update mdtf_test_data version in mdtf_tests.yml add temp_extremes_distshape PODs back to set3 test jsonc files add trop_pac_sl obs data tarball fetch to mdtf_tests.yml * remove test_tpsl.yml from repository * Add base yaml pkgs (#297) * add NCO to env_NCL_base.yml * add subprocess to env_base.yml * fix subprocess module name * Mld debug (#283) * Add CC changes Edit paths temporarily Tidy up Adding mld calculation Working figures Tweaks Separate MLD branch works Add documentation and tidy up html Edit contact Add info to header Fix issues flagged by LGTM Remove testing configuration file and unused pod env vars Edited settings.jsonc, broke something This works, but thetao not added to settings Problem with settings.jsonc file for POD mixed_layer_depth Remove comments from settings.jsonc add depth coordinate to fieldlist_CMIP.jsonc remove comment lines from mld settings file change variable coordinate names to i and j to match model data adjust default_tests.jsonc for testing add lev entry for ocean depth to fieldlist_CMIP.jsonc add check for axis att when defining ds_axes in reconcile_scalar_coords environment variables and path definitions reorganized and cleaned up some routines and calls added logic to rename lat and lon to latitude and longitude if present in model data ds changed so units to match definitions in CMIP fieldlist and ensure functionality with synthetic data general cleanup revert default_tests.jsonc changes remove commented out defintion of ds_axes from xr_parser.py Update src/xr_parser.py Co-authored-by: John Krasting Add density calculation description Update diagnostics/mixed_layer_depth/mixed_layer_depth.py Co-authored-by: John Krasting removed mld environment yaml * add gsw and xesmf to env_python3_base.yml * fixed comments for computemean in mixed_layer_depth.py * remove unused sigma2 computation from computemld; sub pressure.lev to for sigma2.lev in mld Co-authored-by: lettie_roach * Enso mse (#292) * Create xx * Delete xx * Create xx * Create xx * Create xx * Create xx * Delete xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Add files via upload * Delete xx * Create xx * Create xx * Add files via upload * Delete xx * Add files via upload * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Delete xx * Create xx * Add files via upload * Create xx * Add files via upload * Delete xx * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete index.html * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Delete default_tests.jsonc.bak1 * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Remove unused imports from composite.py * Remove unused time import from enso_mse.py * remove unused numpy improt from mse_var_obs.py * remove unused numpy import from mse_var.py * Delete mdtf * Delete env_ENSO_RWS.yml * Delete xr_parser.py * Delete verify_links.py * Delete date_label.py * Revert changes to default_tests.jsonc * Delete get_dimensions.py * Delete get_lon_lat_plevels_in.py * Delete get_season.py * Delete plevs.txt * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Delete xx * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README_LEVEL_01.pdf * Delete README_LEVEL_02.pdf * Delete README_LEVEL_03.pdf * Delete README_LEVEL_04.pdf * Delete ENSO_MSE.pdf * Add files via upload * remove whitespace from default_tests.jsonc Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Feature/ocean flux matrix (New POD) (#293) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Enso rws (#223) * reinitialized files * new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.html new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE.py new file: diagnostics/ENSO_MSE/COMPOSITE/COMPOSITE_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_correlation_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL/plot_regression_all_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_radiation_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/data_routine_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/NCL_CONVERT/write_24month_netcdf_OBS.ncl new file: diagnostics/ENSO_MSE/COMPOSITE/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files.py new file: diagnostics/ENSO_MSE/COMPOSITE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_clima_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_correlation.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_data_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_clima.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_flux_in_24.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_nino_index.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_parameters_in.py new file: diagnostics/ENSO_MSE/COMPOSITE/get_regression.py new file: diagnostics/ENSO_MSE/COMPOSITE/preprocess.py new file: diagnostics/ENSO_MSE/COMPOSITE/write_out.py new file: diagnostics/ENSO_MSE/ENSO_MSE.html new file: diagnostics/ENSO_MSE/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/ENSO_MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE.html new file: diagnostics/ENSO_MSE/MSE/MSE.py new file: diagnostics/ENSO_MSE/MSE/MSE_OBS.py new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all.ncl new file: diagnostics/ENSO_MSE/MSE/NCL/plot_composite_all_OBS.ncl new file: diagnostics/ENSO_MSE/MSE/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/MSE/check_input_files.py new file: diagnostics/ENSO_MSE/MSE/check_input_files_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE/get_data_in.py new file: diagnostics/ENSO_MSE/MSE/get_directories.py new file: diagnostics/ENSO_MSE/MSE/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hadv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_hdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_madv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mdiv.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_mse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_omse.py new file: diagnostics/ENSO_MSE/MSE/moist_routine_tadv.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse.py new file: diagnostics/ENSO_MSE/MSE/write_out_mse_clima.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.html new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR.py new file: diagnostics/ENSO_MSE/MSE_VAR/MSE_VAR_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/NCL_general/plot_bars_composite_OBS.ncl new file: diagnostics/ENSO_MSE/MSE_VAR/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/MSE_VAR/get_anomaly.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_clima_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_data_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_directories_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_flux_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_lonlat_in_OBS.py new file: diagnostics/ENSO_MSE/MSE_VAR/get_parameters_in.py new file: diagnostics/ENSO_MSE/MSE_VAR/moist_routine_variance.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out.py new file: diagnostics/ENSO_MSE/MSE_VAR/write_out_general.py new file: diagnostics/ENSO_MSE/README_general.pdf new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_01.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_02.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_03.ncl new file: diagnostics/ENSO_MSE/SCATTER/NCL/scatter_04.ncl new file: diagnostics/ENSO_MSE/SCATTER/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.html new file: diagnostics/ENSO_MSE/SCATTER/SCATTER.py new file: diagnostics/ENSO_MSE/SCATTER/check_input_files.py new file: diagnostics/ENSO_MSE/SCATTER/get_data_in.py new file: diagnostics/ENSO_MSE/SCATTER/get_scatter_data.py new file: diagnostics/ENSO_MSE/SCATTER/list-models-historical-obs new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.pdf new file: diagnostics/ENSO_MSE/doc/ENSO_MSE.rst new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig1.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig2.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig3.png new file: diagnostics/ENSO_MSE/doc/ENSO_MSE_fig4.png new file: diagnostics/ENSO_MSE/doc/README_LEVEL_01.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_02.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_03.pdf new file: diagnostics/ENSO_MSE/doc/README_LEVEL_04.pdf new file: diagnostics/ENSO_MSE/doc/README_general.pdf new file: diagnostics/ENSO_MSE/html/index.html new file: diagnostics/ENSO_MSE/html/index_mdtf_03.html new file: diagnostics/ENSO_MSE/html/mdtf_composite.html new file: diagnostics/ENSO_MSE/html/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/input_data/obs_data/ENSO_MSE new file: diagnostics/ENSO_MSE/mdtf_diag_banner.png new file: diagnostics/ENSO_MSE/settings.jsonc new file: diagnostics/ENSO_MSE/shared/generate_ncl_call.py new file: diagnostics/ENSO_MSE/shared/get_dimensions.py new file: diagnostics/ENSO_MSE/shared/get_lon_lat_plevels_in.py new file: diagnostics/ENSO_MSE/shared/get_season.py new file: diagnostics/ENSO_MSE/shared/gsnColorRange.ncl new file: diagnostics/ENSO_MSE/shared/parameters.txt new file: diagnostics/ENSO_MSE/shared/plevs.txt new file: diagnostics/ENSO_MSE/shared/read_netcdf_2D.py new file: diagnostics/ENSO_MSE/shared/read_netcdf_3D.py new file: diagnostics/ENSO_MSE/shared/rgb/amwg.old.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg.png new file: diagnostics/ENSO_MSE/shared/rgb/amwg.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/amwg_reverse.rgb new file: diagnostics/ENSO_MSE/shared/rgb/bluered.rgb new file: diagnostics/ENSO_MSE/shared/rgb/blueyellowred.rgb new file: diagnostics/ENSO_MSE/shared/rgb/cloudsim.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr.rgb new file: diagnostics/ENSO_MSE/shared/rgb/corr2.rgb new file: diagnostics/ENSO_MSE/shared/rgb/diff.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rainbow21.rgb new file: diagnostics/ENSO_MSE/shared/rgb/redyellowblue.rgb new file: diagnostics/ENSO_MSE/shared/rgb/rgb.txt new file: diagnostics/ENSO_MSE/shared/rgb/show_colors.ncl new file: diagnostics/ENSO_MSE/shared/rgb/stress.rgb new file: diagnostics/ENSO_MSE/shared/set_variables_AM4.py new file: diagnostics/ENSO_MSE/shared/set_variables_CESM.py new file: diagnostics/ENSO_MSE/shared/set_variables_CMIP.py new file: diagnostics/ENSO_MSE/shared/util.py * modified: diagnostics/ENSO_MSE/settings.jsonc * Delete diagnostics/ENSO_MSE directory * Create ENSO_RWS.pdf * Create ENSO_RWS.pdf * Create util.py * Create LEVEL_01.py * Create LEVEL_02.py * Create LEVEL_03.py * Create LEVEL_04.py * Create data_routine.ncl * Create plot_betastar_clima.ncl * Create plot_RWS_composite.ncl * Create scatter_plot_01.ncl * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create obs_data * Delete obs_data * Create xx * Create xx * Delete xx * Delete xx * Create xx * Delete xx * Create xx * Delete xx * Create enso_mse * Create enso_rws * Delete enso_mse * Delete enso_rws * Create enso_rws * Delete enso_rws * Delete util.py * Create util.py * Add files via upload * Delete ENSO_RWS.pdf * Delete README_LEVEL_01_ENSO_RWS.pdf * Delete README_LEVEL_02_ENSO_RWS.pdf * Delete README_LEVEL_03_ENSO_RWS.pdf * Delete README_LEVEL_04_ENSO_RWS.pdf * Delete README_general_ENSO_RWS.pdf * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete ENSO_RWS.pdf * Add files via upload * Create xx * Delete xx * Create xx * Add files via upload * Add files via upload * Create x * Delete x * Create dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Create dummy * Create dummy * Add files via upload * Delete dummy * Add files via upload * Delete dummy * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete LEVEL_01.html * Delete LEVEL_01.py * Delete check_input_files.py * Delete check_input_files_OBS.py * Delete get_clima_in.py * Delete get_data_in.py * Delete get_dims.py * Delete get_directories.py * Delete get_directories_OBS.py * Delete get_flux_clima.py * Delete get_flux_in.py * Delete get_nino_index.py * Delete process_data.py * Delete write_out.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Update mdtf_framework.py * Update cli.py * Update cli.py * Update cmip6.py * Update env_python3_base.yml * Update core.py * Update data_manager.py * Update data_model.py * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_tests.jsonc.bak1 * Delete default_tests.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete default_test.jsonc * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete README * Delete xx * commit * Delete mdtf * Delete env_ENSO_MSE.yml * Revert changes to default_tests.jsonc * Delete verify_links.py * Delete read_netcdf_2D.py * Delete read_netcdf_3D.py * Add files via upload * Add files via upload * Delete env_ENSO_RWS.yml * Add files via upload * Add files via upload * Add files via upload * Add files via upload * Delete inputdata/obs_data directory * remove extra space from default_tests.jsonc * Delete diagnostics/ENSO_MSE directory Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * change developer cheatsheet link * Feature/ocean flux matrix (merged POD update on rst) (#300) * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add the necessary function and main script have not integrate to the framework yet * POD name change and default test json * add metpy package * testing ocn_surf_flux_diag POD with framework * surface variables add modifier * test synthetic data * update the output format in obs_data * fix the numpy error due to framework inputs * include function docstring * correct the 0.98 factor for salinity effect * add script description and produce netCDF output * finish the rst description for the POD * update the html for POD specific changes * remove comments and correct the "more" part * remove the unused function * remove unused import * remove unused import variable * remove testing files * remove unused print * remove unused plotting command * testing file * update description * remove obs plot * change output plot name * remove testing jsonc * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Update diagnostics/ocn_surf_flux_diag/ocn_surf_flux_diag.html Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * update outdated into in rst Co-authored-by: Jess <20195932+wrongkindofdoctor@users.noreply.github.com> * Add pods to ci (#301) * added checks that plotted values fall within contour levels to prevent synthetic data nans from causing failure and fixed the figure output path definition in ocn_surf_flux_diag.py * added mixed layer depth and ocn surf flux matrix pods to test set 3 yamls * removed the mdtf_test_data version spec from mdtf_tests.yml * added temporary test workflow file for new pods * fix name of ocn_surf_flux_diag tarball in test yaml * convert start and end dates to iso format in mixed_layer_depth.py so that pod works with full range of year integer values * added test lines to dump detailed output for mld pod to screen for debugging * turned on experimental setting * replace cat with printf * add updated test files * add continue on error to step, only build required environments * move log dump to separate step * fix test workflow file * add esmpy to python3_base.yml and update xesmf version * specify pip to install xesmf * change package order * update netcdf4, esmpy versions in python3_env yaml add esmpy 8.2 to python3_env yaml to resolve package dependency issues * update env_synthetic_data yaml package versions to match python3_base add ocn_surf_flux_diag test back to test_tpsl to double check env changes comment out mld log dump step * add ocn_surf_flux_diag and mld PODs to mdtf_tests.yml remove test_tpsl from remote repo uncomment other set3 PODs in test jsonc files * Fix Explicit_file CLI definition * Update the CM4 diag table Add atmos variables to the diag table Note that this currently generates variables required for the MJO_suite POD using GFDL standard output more variables may be required for the other PODs. It is also preferable to output in CMIP6 format Co-authored-by: tsjackson-noaa Co-authored-by: Thomas Jackson <52828051+tsjackson-noaa@users.noreply.github.com> Co-authored-by: Thomas Jackson Co-authored-by: Arielle Catalano Co-authored-by: John Krasting Co-authored-by: chiaweih@climate Co-authored-by: Chia-Wei Hsu Co-authored-by: lettie_roach Co-authored-by: jhafner2 <71046600+jhafner2@users.noreply.github.com> --- sites/NOAA_GFDL/CM4_proto_diag_table.txt | 115 ++++++++++++++--------- 1 file changed, 73 insertions(+), 42 deletions(-) diff --git a/sites/NOAA_GFDL/CM4_proto_diag_table.txt b/sites/NOAA_GFDL/CM4_proto_diag_table.txt index a3e223d8b..fbdfb123c 100644 --- a/sites/NOAA_GFDL/CM4_proto_diag_table.txt +++ b/sites/NOAA_GFDL/CM4_proto_diag_table.txt @@ -1,6 +1,6 @@ -# REQUIRED FIELDS TO RUN MDTF DIAGNOSTICS - -#-------------------------------------------------------------------------------------------------- +# REQUIRED FIELDS TO RUN MDTF DIAGNOSTICS on supported atmosphere and land PODs + +#-------------------------------------------------------------------------------------------------- #-- grid_spec #-------------------------------------------------------------------------------------------------- @@ -19,22 +19,41 @@ "dynamics", "zsurf", "zsurf", "grid_spec", "all", .false., "none", 2 - - #-------------------------------------------------------------------------------------------------- #-- Monthly Data #-------------------------------------------------------------------------------------------------- - -"atmos_month_cmip", 1, "months", 1, "days", "time", - - -"flux", "evspsbl", "evspsbl", "atmos_month_cmip", "all", .true., "none", 2 -"moist", "pr", "pr", "atmos_month_cmip", "all", .true., "none", 2 -"flux", "psl", "psl", "atmos_month_cmip", "all", .true., "none", 2 -"atmos", "ps", "ps", "atmos_month_cmip", "all", .true., "none", 2 -"atmos_plev19", "zg", "zg_unmsk", "atmos_month_cmip", "all", .true., "none", 2 - +"atmos_month", 1, "months", 1, "days", "time", + +"flux", "land_mask", "land_mask", "atmos_month", "all", .false., "none", 2 +"dynamics", "zsurf", "zsurf", "atmos_month", "all", .false., "none", 2 +"dynamics", "bk", "bk", "atmos_month", "all", .false., "none", 2 +"dynamics", "pk", "pk", "atmos_month", "all", .false., "none", 2 +"dynamics", "temp", "temp", "atmos_month", "all", .true., "none", 2 +"dynamics", "sphum", "sphum", "atmos_month", "all", .true., "none", 2 +"moist", "pr", "pr", "atmos_month", "all", .true., "none", 2 +"moist", "prw", "pw", "atmos_month", "all", .true., "none", 2 +"flux", "evspsbl", "evspsbl", "atmos_month", "all", .true., "none", 2 +"flux", "psl", "psl", "atmos_month", "all", .true., "none", 2 +"flux", "t_surf", "t_surf", "atmos_month", "all", .true., "none", 2 +"flux", "hfss", "hfss", "atmos_month", "all", .true., "none", 2 +"flux", "lwflx", "lwflx", "atmos_month", "all", .true., "none", 2 +"flux", "shflx", "shflx", "atmos_month", "all", .true., "none", 2 +"radiation", "swdn_toa", "swdn_toa", "atmos_month", "all", .true., "none", 2 +"radiation", "swup_toa", "swup_toa", "atmos_month", "all", .true., "none", 2 +"radiation", "lwtoa_ad", "lwtoa_ad", "atmos_month", "all", .true., "none", 2 +"radiation", "lwdn_sfc", "lwdn_sfc", "atmos_month", "all", .true., "none", 2 +"radiation", "lwup_sfc", "lwup_sfc", "atmos_month", "all", .true., "none", 2 +"radiation", "swdn_sfc", "swdn_sfc", "atmos_month", "all", .true., "none", 2 +"radiation", "swup_sfc", "swup_sfc", "atmos_month", "all", .true., "none", 2 +"atmos", "ps", "ps", "atmos_month", "all", .true., "none", 2 +"atmos", "zg500", "zg500", "atmos_month", "all", .true., "none", 2 +"atmos_plev19", "zg", "zg_unmsk", "atmos_month", "all", .true., "none", 2 +"atmos_plev19", "ua", "ua_unmsk", "atmos_month", "all", .true., "none", 2 +"atmos_plev19", "va", "va_unmsk", "atmos_month", "all", .true., "none", 2 +"atmos_plev19", "ta", "ta_unmsk", "atmos_month", "all", .true., "none", 2 +"atmos_plev19", "wap", "wap_unmsk", "atmos_month", "all", .true., "none", 2 +"atmos_plev19", "hus", "hus_unmsk", "atmos_month", "all", .true., "none", 2 "land_month_cmip", 1, "months", 1, "days", "time", @@ -43,38 +62,50 @@ "land", "geolat_t", "geolat_t", "land_month_cmip", "all", .false., "none", 1 "cmor_land", "mrsos", "mrsos", "land_month_cmip", "all", .true., "none", 2 - - - -#-------------------------------------------------------------------------------------------------- +-------------------------------------------------------------------------------------------------- #-- Daily Data #-------------------------------------------------------------------------------------------------- - -"atmos_daily_cmip", 24, "hours", 1, "days", "time", - - -"atmos_plev19", "hus", "hus_unmsk", "atmos_daily_cmip", "all", .true., "none", 2 -"radiation", "rlut", "rlut", "atmos_daily_cmip", "all", .true., "none", 2 -"atmos_plev19", "ua", "ua_unmsk", "atmos_daily_cmip", "all", .true., "none", 2 -"atmos_plev19", "va", "va_unmsk", "atmos_daily_cmip", "all", .true., "none", 2 -"atmos_plev19", "wap", "wap_unmsk", "atmos_daily_cmip", "all", .true. , "none", 2 -"atmos_plev19", "zg", "zg_unmsk", "atmos_daily_cmip", "all", .true., "none", 2 -"moist", "pr", "pr", "atmos_daily_cmip", "all", .true., "none", 2 -"atmos", "ps", "ps", "atmos_daily_cmip", "all", .true., "none", 2 - - +"atmos_daily", 24, "hours", 1, "days", "time", + +"flux", "land_mask", "land_mask", "atmos_daily", "all", .false., "none", 2 +"dynamics", "zsurf", "zsurf", "atmos_daily", "all", .false., "none", 2 +"dynamics", "bk", "bk", "atmos_daily", "all", .false., "none", 2 +"dynamics", "pk", "pk", "atmos_daily", "all", .false., "none", 2 +"atmos_plev19", "hus", "hus_unmsk", "atmos_daily", "all", .true., "none", 2 +"atmos_plev19", "omega", "omega_unmsk", "atmos_daily", "all", .true., "none", 2 +"atmos_plev19", "ua", "ua_unmsk", "atmos_daily", "all", .true., "none", 2 +"atmos_plev19", "va", "va_unmsk", "atmos_daily", "all", .true., "none", 2 +"atmos_plev19", "wap", "wap_unmsk", "atmos_daily", "all", .true., "none", 2 +"atmos_plev19", "zg", "zg_unmsk", "atmos_daily", "all", .true., "none", 2 +"moist", "pr", "pr", "atmos_daily", "all", .true., "none", 2 +"moist", "prw", "prw", "atmos_daily", "all", .true., "none", 2 +"atmos", "zg500", "zg500", "atmos_daily", "all", .true., "none", 2 +"atmos", "ps", "ps", "atmos_daily", "all", .true., "none", 2 +"dynamics", "u200", "u200", "atmos_daily", "all", .true., "none", 2 +"dynamics", "v200", "v200", "atmos_daily", "all", .true., "none", 2 +"dynamics", "u850", "u850", "atmos_daily", "all", .true., "none", 2 +"dynamics", "v850", "v850", "atmos_daily", "all", .true., "none", 2 +"flux", "tas", "tas", "atmos_daily", "all", .true., "none", 2 +"flux", "psl", "psl", "atmos_daily", "all", .true., "none", 2 +"dynamics", "omega500", "omega500", "atmos_daily", "all", .true., "none", 2 +"radiation", "rlut", "rlut", "atmos_daily", "all", .true., "none", 2 +"dynamics", "temp", "temp", "atmos_daily", "all", .true., "none", 2 +"dynamics", "sphum", "sphum", "atmos_daily", "all", .true., "none", 2 #-------------------------------------------------------------------------------------------------- #-- 6 hourly data #-------------------------------------------------------------------------------------------------- - - -"atmos_4xdaily_cmip", 6, "hours", 1, "days", "time", - - -"moist", "pr", "pr", "atmos_4xdaily_cmip", "all", .true., "none", 2 -"moist", "prw", "prw", "atmos_4xdaily_cmip", "all", .true., "none", 2 -"atmos", "ps", "ps", "atmos_4xdaily_cmip", "all", .true., "none", 2 -"atmos_plev19", "ta", "ta_unmsk", "atmos_4xdaily_cmip", "all", .true., "none", 2 +"atmos_4xdaily", 6, "hours", 1, "days", "time", + +"flux", "land_mask", "land_mask", "atmos_4xdaily", "all", .false., "none", 2 +"dynamics", "zsurf", "zsurf", "atmos_4xdaily", "all", .false., "none", 2 +"dynamics", "bk", "bk", "atmos_4xdaily", "all", .false., "none", 2 +"dynamics", "pk", "pk", "atmos_4xdaily", "all", .false., "none", 2 +"dynamics", "temp", "temp", "atmos_4xdaily", "all", .true., "none", 2 +"dynamics", "sphum", "sphum", "atmos_4xdaily", "all", .true., "none", 2 +"moist", "pr", "pr", "atmos_4xdaily", "all", .true., "none", 2 +"moist", "prw", "prw", "atmos_4xdaily", "all", .true., "none", 2 +"atmos", "ps", "ps", "atmos_4xdaily", "all", .true., "none", 2 +"atmos_plev19", "ta", "ta_unmsk", "atmos_4xdaily", "all", .true., "none", 2