Corrected halo size in density derivative calculations in smoothed_dRdT_dRdS
This fixes an i-parallelization problem that was recently introduced (as a part
of MOM6 PR#1089) when VERTEX_SHEAR is True, and closes MOM6 issue #1146.  All
answers in the existing MOM6-examples test suite are bitwise identical, but this
does change (correct) answers when VERTEX_SHEAR is true.

- FMS2020 has newer cpu affinity work. These are mostly to fix the
  issues with thread placing and  hyperthreadng under slurm on gaea.
  But it also works on Orion.
- The new affinity module simplifies the thread-placing calls in the
  component models.
- The name of some functions has changed, that's the reason for crashes
  like:
      FATAL: input domain does not have an io_domain.
- This update fixes those issues.
- openmp runs with 1 and 2 threads gives the same answers as non-openmp
- NOTE: I don't rememer why we put the thread placing calls in MOM_domains.F90
        They look as unnecessary and the whole #ifndef NOT_SET_AFFINITY block
        can probably be removed. ocean_nthreads is either set in coupler
or solo_driver.

(*)Corrected halo size in EOS call if VERTEX_SHEAR=T

FMS affinity operations, often used in OpenMP directives, require
explicit CPU affinities such that the number of available CPUs matches
the number of requested PEs.

To accommodate this, we explicit configure OpenMP to use cpu=0 for our
ARM tests.

  Removed old debugging code in blocks of code surrounded by #ifdef statements
and removed unnecessary #ifdef around other blocks of debugging code.  MOM6
standards discourage the use of CPP macros except for a limited set of uses
related to memory where this is unavoidable, so this commit is bringing MOM6
closer to its stated standards.  All answers and output are identical.

  Modified doc_module so that new lines are added only when modules are
documented, and are added in all parameter_doc files in which modules are
documented.  All answers and output are identical, but there are white space
changes in MOM_parameter_doc and SIS_parameter_doc files.

  Added the new optional argument like_default to the log_param and doc_param
routines to help control where the documentation appears.  This new argument is
used for logging EPBL_USTAR_MIN, the diagnosed output value of MAXIMUM_DEPTH
when the input value is negative, and the diagnosed number of columns where
sponges occur with MOM_ALE_sponge.  An '!' was also added to the logging of
EPBL_USTAR_MIN.  All answers are bitwise identical but there are minor changes
in the contents of some MOM_parameter_doc.short files.

…nto add_dens_deriv_diag

Added diagnostics for partial derivatives of density

Removed #ifdef debugging blocks

  Added code to determine whether all parameters in the MOM_grid, MOM_restart,
MOM_write_cputime and MOM_tracer_registry modules are being used with their
default settings, and added all_default arguments to the log_version calls for
these modules.  All answers and output are identical, but there are white space
changes in MOM_parameter_doc.short and SIS_parameter_doc.short files.

Testing: Explicit OpenMP CPU affinity

- Members of a type cannot be individually labelled as shared/private
- One variable was converted to shared since it was defiend in a non-OMP
  section and then labelled as private which meant it was uninitialized.

…A/MOM6 into Hallberg-NOAA-module_doc_newlines

…into dev/gfdl

…hub.com/nikizadehgfdl/MOM6 into nikizadehgfdl-fix_openmp_affinity_issues_with_FMS2020

…into dev/gfdl

- Stanley et al., 2020, adds the Brankart modification to volume mean
  density via linear corrections involving SGS sample variances and
  covariances of T and S. This commit adds the new interfaces that
  allow a call to calculate_density to include the variances and
  covariance as arguments. This correction sits above the particular
  EOS and thus can use any EOS so long as second derivatives are
  available within the EOS module.

- The routines pressure_gradient_plm() and pressure_gradient_ppm()
  were poorly named and had comments referring to the pressure gradient
  calculation even though the only calculate edge values in the vertical
  for T/S using ALE functions. The routines are actually more general
  and can be used outside of the PGF. The comments have been shortened
  and no longer refer to the PGF.

- The integrals of density routines used (mostly) by the PGF calculation
  were part of MOM_EOS. Originally, when writing the analytic FV PGF paper,
  this was the right place to put the integrals. The additional variants
  using the ALE reconstruction functions mean that it is cleaner to have
  these routines sit in a layer above EOS and ALE.

- ppoly_E meant something to someone a while ago but we felt it
  would be better to clean up the ALE APIs. This is a pre-cursor
  to switching to a more precise description of reconstructions.

- Undid nesting of if / select statemnent
- Changed FATAL messages

- Function had been moved to density_integrals already but not
  deleted
- Also ordered public statements in MOM_EOS to help find things.

- Order was frustratingly illogical

- When converting from the two hor_index types to one (HII and HIO
  became HI) I retained the HI%isd:... declaration statements. We
  normally use `SZI_(G)` or `SZI_(HI)` so this switches to that style.

- This adds elemental functions for cellwise operations within the PLM
  construction procedure which allows the operations to be accessed
  from outside of the ALE functions on different array shapes but recover
  bitwise identical results.
- The older subroutines now use the functions and some optimizations were
  obtained in the process.

- Out of curiosity I want to be sure these weren't implicated in the long
  initialization cost.

- Soft index convention was not properly implemented in original PLM
  density integrals. Fixed to avoid confusion.

- Probably makes no difference but we re-computed weights repeatedly
  even though we'd pre-computed them (and used only once).
  - Reduces number of local variables.

- Removes (now) unused functions for quadrature and polynomial evaluation
- Has been tested by setting the logical use_PPM=.false.
  - reproduces PLM mode bitwise

- Use of new PLM functions led to out of date directives

- Now passing in 3D structures and tv so that we can access other fields
  in tv%.
- Corrected calculation of curvatures s6, t6 for interpolated interior
  line integrals.
- Tested with use_PPM=.false. and PLM edge values.
  - Using PLM edge values with use_PPM=.true. does give different answers
    because t6,s6 are non-zero albeit tiny. This is due to FP truncation
    errors.

- Since the majority of code is not about the analytic integration
  I thought it time to rename this module

- Continuation of rename of module

- This routine was suffering from if's inside loops. I've removed this
  one since it was just obfuscation, pretending to care about performance
  and actually reducing it.

- Makes use of the Stanley equation of state to include effects of
  SGS temperature variance, salinity variance and T-S covariance.

- This now calculates the gradient of SGS temperature variance
  using the same discretization as used for the gradient of
  density along coordinate surfaces.
- Added run-time coefficient for Stanley parameterization
- Fixed openmp directives.
- Alters halo over which vert_fill_TS() is called.
- Add Stanley parameter to tc2 to test new code.

- Renamed STANLEY_DET_COEFF to STANLEY_PRM_DET_COEFF to indicate
  this is for the Stanley parameterization as opposed to a related
  approach by Stanley for implementing the Brankart PGF correction.

- PGF_STANLEY_T2_DET_COEFF>=0. now adds the temperature variance
  contribution to the Brankart correction using the Stanley linearization
  of the EOS and parameterization of SGS variance.

- After starting with allocatables and switching to pointers
  I'd forgotten to nullify them.

- Passing the scale= to calculate_density_second_derivs() was the
  double scaling the density contribution from SGS variance in the
  Stanley parameterization.

- select case statements were indented as if once inside a loop.

- The pressure scaling was wrong when trying to call
  calculate_density_second_derivs_array() from within
  calculate_stanley_density_array() because the latter should not do
  any scaling but the former always did. I had to call the lower level
  functions provided by WRIGHT, TEOS10, etc to avoid get the scaling
  tests to pass.

- In preparing to add Brankartc terms to PLM form of presure gradient
  I'm adapting the routine to look like the PPM routine which takes 3D
  arguments but only works on layer "k".

- The Brankart PGF terms are now implemented in the PLM recontruction
  routines, just as they were for the PPM form.

- Added documentation lines for PGF_STANLEY_T2_DET_COEFF

- Per feedback to #1156

- Per feedback on #1156, corrected scaling for calculate_stanley_density_scalar()
  and calculate_stanley_density_1d().

- Per feedback on #1156.

- Per feedback on #1156

  Set a hard-coded maximum value of CS%a_u and CS%a_v of 1e37 m s-1 so that
these can be represented in diagnostics that are written with 32-bit floating
point numbers.  These values are so large that all answers are bitwise identical
in the MOM6-examples test cases, but it is possible that answers could change.

- Per feedback on #1156.

  Eliminated the unused variables CS%uhbt_IC and CS%vhbt_IC from the barotropic
control structure and from the MOM6 restart files.  Also placed a logical test
for CS%Gradual_BT_ICs around all references to CS%ubt_IC and CS%vbt_IC and
eliminated these variables from the restart files when they would not be used.
Additionally some unused arguments were removed from internal subroutines and
some spelling or index case errors were corrected.  All answers are bitwise
identical, but the MOM6 restart files have fewer variables and some unused
entries in the MOM_parameter_doc files have changed.

  Add the new runtime parameter INTEGRAL_BT_CONTINUITY which enables the use of
the time-integrated barotropic velocity to determine the cumulative transport
since the start of the barotropic stepping.  This new option works in all of the
MOM6-examples test cases with a SPLIT=True and USE_BT_CONT_TYPE=True.  By
default all answers are bitwise identical, but there are changes to the entries
in the MOM_parameter_doc files.

  Added the new subroutines find_duhbt_dubt_int and find_dvhbt_dvbt_int, and use
the time-integrated forms to set the inverse face area, the transport correction
for consistency between the transport from the initial barotropic velocity and
the summed layer transports, and the maximum corrective mass source when
INTEGRAL_BT_CONT=True.  Also only log BT_CONT_CORR_BOUNDS when BOUND_BT_CORR is
set to true.  By default, all answers are bitwise identical, but there are minor
changes to the entries in the MOM_parameter_doc files.

  Added code to implement new integral_BT_cont options within the barotropic
open boundary condition code.  A number of new arguments were added to
apply_velocity_OBCS.  In addition, the handling of updates to ubt_sum, uhbt_sum
and ubt_wtd with open boundary conditions were simplified.  There are minor
answer changes if INTEGRAL_BT_CONTINUITY=True, but all answers in the existing
MOM6-examples tess cases are bitwise identical.

  Revised the dimensions of the entries in the local_BT_Cont types when
INTEGRAL_BT_CONT=True for efficiency and to enable find_uhbt and other routines
to be used regardless of the value of INTEGRAL_BT_CONT.  Some arguments that are
no longer needed have been removed from some subroutines. All answers are
bitwise identical in the MOM6-examples test suite, but changes to the order of
arithmetic leads to changes from the previous version when
INTEGRAL_BT_CONT=True, and the rescaling is controlled via new optional
arguments to internal routines.

  Revised ice_shelf_driver.F90 so that it compiles successfully and is similar
to MOM_driver.F90 where possible.  In addition, deleted the files
MOM_surface_forcing.F90 and user_surface_forcing.F90 in the ice_solo_driver
directory, as these are unused and out-of-date versions of the equivalent files
in the solo_driver directory.  Some of the changes are not as streamlines as
they could be if the solo_ice_shelf code used its own version of diag_mediator,
or if the interface to diag_mediator were revised so that some of its
ocean-specific arguments are optional.  There is no test-case exercising this
code, but at least it now compiles, whereas it did not compile before.

- FMS tag 2019.01.03 fixes problems for EMC allowing the newer FV3 to
  work with MOM6 which is not yet compatible with FMS2020.

Move FMS tag to 2019.01.03

  Modified recent additions to avoid using uninitialized arrays.  This was not
detected in tests of the previous PR because the arrays that were created from
uninitialized arrays were themselves never used, so there are no answer changes
but there were errors with intolerant compiler settings.  Also corrected the
dOxygen syntax in a comment and corrected some openMP directives that had been
triggering warnings or errors.  All answers are bitwise identical.

+Adding a halo update for tracer reservoirs.

  Added some missing openMP directives that had been triggering warnings or
errors, and could have created problems with more extensive testing.  Also set
some unset array bounds in the ice_shelf_dynamics code.  All answers are bitwise
identical in all working test cases.

MOM6: +(*)Add option to use time-integrated barotropic continuity

MOM6: (*?)Revised ice_shelf_driver.F90 so it compiles

The "for" loop at L510 should be to nz instead of nz+1.  Due to array allocation sizes, this loop was extending beyond the shope of the argument arrays.

Squash merge of following commits:

* Testing: tc4 installs python-netCDF4 via venv

Currently, users are expected to have numpy and netcdf4 python modules
in order to generate the necessary netCDF input files.  This fails in
environments where these modules are unavailable.

This patch now installs the modules into a virtual environment which are
accessible when generating the tc4 inputs.

This solution is local to tc4 but could be extended to other tests as
needed.

* Testing: remove Python numpy and netCDF4 modules

The numpy and netCDF4 packages are no longer needed since tc4 now
installs these locally.

* Test: Install virtualenv for Python2

tc4 was using the venv module which appears to be python3-specific, and
Travis Ubuntu defaults to python2.  Also virtualenv was not installed in
either case.

This patch adds python-virtualenv to the install packages and uses the
virtualenv module.

* Test: Add python-dev for ARM64 numpy

Travis ARM nodes need to build numpy natively when installed by pip, and
thus require Python headers.  These are provided by the python-dev
package.

* Test: Revert to Py3 for tc4 generation scripts

Reverting the python 2 support (default for Travis) to use Python 3
syntax.

The main reason is that Python 3 includes venv (equivalent to
virtualenv) as its standard library, and is therefore guaranteed to
exist if Python 3 exists.

Python 3's virtualenv must be independently installed, which cannot be
confirmed.

This will cause problems for people without Python 3, but this is
probably the best solution, or at least the starting point for a more
general solution.

* Test: Adding python3-venv to Travis

Ubuntu apparently requires an explicit install of python3-venv despite
it being part of the standard library.  Go figure...

* Test: Arm64 tc4 Configuration support

Arm64 Ubuntu environments require explicit installations which are
otherwise provided on x86 Ubuntu:

* Python 3 Pip must be installed (python3-pip)
* Wheel installation must be explicitly installed
* Cython is required for numpy
* Numpy must be explicitly built before installing python-netCDF4

* Test: tc4 common Python local-env

Build times for setting up the virtual environments can be very
expensive on the Arm64 Ubuntu nodes, so we now create a shared directory
for launching the environments.

* Test: Use '.' in place of 'source' for Make

* Test: Test for required Python modules in tc4

We have reworked the Makefile to conditionally test for required Python
modules in tc4.  If unavailable, we install these in a virtual
environment.

This does not address many scenarios, such as if Python 3 is missing,
venv is missing (as in Ubuntu), or handle the situation if they do not
exist.  It assumes that either the modules exist, or that they can be
installed by venv.

This should be seen as an iterative step to get things working on Travis
x86 and Arm64, as well as GFDL's Gaea and most user Linux platforms.

* Test: Explicit python execs for tc4 input build

This resolves some issues with python2/3 resolution and limited support
of various platforms for module support.

Specifically, older platform with basic Python 3 support may not also
have numpy support.  In this case, we can defer back to Python 2 (or
whatever the system Python may be).

* Test: Setup Python venv at build time

This patch moves the Python virtual environment configuration to the
main Makefile, which is setup at build time, rather than in the model
configuration Makefile, which will typically not have internet access if
run on a compute node.

As before, the venv will only be setup when the numpy and netCDF4
modules are unavailable.  A minor bug in the logic of the check has
also been fixed.

Fixing loop bound error in MOM_PressureForce_Montgomery.F90

Squash merge:

* New diagnostics for barotropic momentum budget calculations are created. In these, different budget terms multiplied by fractional layer thicknesses are saved. Thus, these terms can be added over the whole to obtain the barotropic momentum budget. 'btstep' subroutine in MOM_barotropic module is modified to return fractional thicknesses. Pressure force acceleration multiplied by fractional thickness as diagnostics are added in this commit.

* More thickness weighted diagnostics

* Implemented Andrew Shao's suggestions

* More fractional thickness multiplied diagnostics

* Some barotropic diagnostics obtained from fractional thickness multiplied momentum budget terms

* Removed trailing spaces

* All fractional-thickness multiplied diagnostics implemented

* define diagnostic variables as pointers

* Shorter initialization of 2D arrays

* Modifications in vertical friction diagnostics

* Diagnostics initialization as pointers to save memory allocation

* Vertical friction module

* Removed commented lines and trailing spaces

* Diagnostic description change

* Fractional thickness-weighted diagnostics for acceleration due to relative vorticity and gradient of kinetic energy

* Modifications in vertical friction diagnostics (fractional thickness-weighted)

* Modifications in diag_hfrac_u and diag_hfrac_v calls

* Made allocation of hfrac at u/v points optional. These arrays are only allocated if any of the relevant diagnostics are called.

* Fractional-thickness weighted 3D diagnostics are now commented out as we do not the proper grid remapping option.

* Initialization of 2D diagnostic arrays changed from pointer type to allocatable array style.

* Trailing spaces removed and line lengths reduced to be under 120 characters

* Comment modified

Reviewed by Robert Hallberg <[email protected]>

This patch fixes three dimensionality errors in the OBC segments.

- We add a missing GV%m_to_H conversion for time-dependent eta segments.

- The `adjustSegmentEtaToFitBathymetry` function depends on the
  `segment%Htot` field when computing the dz_src cell spacings.

  While the calculation primarily assumes that all quantities scale as
  Z, the segment%Htot field scales as H, which was causing
  dimensionality errors.

  We resolve this by converting Htot from Z to H whenever it is used in
  the calculation.

- Segment barotropic velocitys based on transports were limited to a
  hard-coded thickness of 1e-12, which was not scaled.  We have added
  H-dimensional scaling to these constants.

The current OBC segment list includes a "segment zero" which corresponds
to when the segment maps (segnum_u, segnum_v) point to OBC_NONE, which
is set to zero.

The purpose of this "segment zero" seems to be for avoiding invalid
references, so that OBC%segment(OBC%segnum_[uv](i,j)) always returns a
valid result, included when set to OBC_NONE.  This results in reading
and setting values which are unused or unneeded.

This patch replaces these redundant accesses to segment zero with
various conditional blocks for avoiding these accesses.

OBC: H-dimensionality fixes