Skip to content

Commit

Permalink
Clean-up temporary rundirs - take 2. (#2753)
Browse files Browse the repository at this point in the history
This PR:
- is a follow-up to a previous PR that aggressively pruned run
directories.
- removes run directories for the current cycle in the clean-up if the
cycle is successful. If the cycle is not successful, cleanup is not
called and all run directories for the cycle are safe from being purged.
- also updates the PR template to list/query for any updates to
submodules.

---------

Co-authored-by: David Huber <[email protected]>
Co-authored-by: David Huber <[email protected]>
Co-authored-by: Walter Kolczynski - NOAA <[email protected]>
  • Loading branch information
4 people authored Aug 6, 2024
1 parent d599fff commit 8fee36f
Show file tree
Hide file tree
Showing 12 changed files with 49 additions and 36 deletions.
11 changes: 11 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,17 @@
# Change characteristics
- Is this a breaking change (a change in existing functionality)? YES/NO
- Does this change require a documentation update? YES/NO
- Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
- [ ] EMC verif-global
- [ ] GDAS
- [ ] GFS-utils
- [ ] GSI
- [ ] GSI-monitor
- [ ] GSI-utils
- [ ] UFS-utils
- [ ] UFS-weather-model
- [ ] wxflow


# How has this been tested?
<!-- Please list any test you conducted, including the machine.
Expand Down
2 changes: 1 addition & 1 deletion jobs/JGFS_ATMOS_FBWIND
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# GFS FBWIND PRODUCT GENERATION
############################################
source "${HOMEgfs}/ush/preamble.sh"
source "${HOMEgfs}/ush/jjob_header.sh" -e "fbwind" -c "base"
source "${HOMEgfs}/ush/jjob_header.sh" -e "fbwind" -c "base fbwind"

###################################
# Specify NET and RUN Name and model
Expand Down
2 changes: 1 addition & 1 deletion jobs/JGFS_ATMOS_GEMPAK_META
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# GFS GEMPAK META PRODUCT GENERATION
############################################
source "${HOMEgfs}/ush/preamble.sh"
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak_meta" -c "base"
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak_meta" -c "base gempak"


###############################################
Expand Down
2 changes: 1 addition & 1 deletion jobs/JGFS_ATMOS_GEMPAK_PGRB2_SPEC
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#! /usr/bin/env bash

source "${HOMEgfs}/ush/preamble.sh"
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak_spec" -c "base"
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak_spec" -c "base gempak"

############################################
# Set up model and cycle specific variables
Expand Down
6 changes: 3 additions & 3 deletions jobs/JGLOBAL_CLEANUP
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "cleanup" -c "base cleanup"

"${SCRgfs}/exglobal_cleanup.sh"
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"
(( status != 0 )) && exit "${status}"

##########################################
# Remove the Temporary working directory
##########################################
cd "${DATAROOT}" || (echo "${DATAROOT} does not exist. ABORT!"; exit 1)
[[ ${KEEPDATA} = "NO" ]] && rm -rf "${DATA}"
# DATAROOT="${STMP}/RUNDIRS/${PSLOT}/${RUN}.${PDY}${cyc}"
# is removed in exglobal_cleanup.sh, nothing to do here.

exit 0

1 change: 0 additions & 1 deletion parm/config/gefs/config.base
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@ export PSLOT="@PSLOT@"
export EXPDIR="@EXPDIR@/${PSLOT}"
export ROTDIR="@COMROOT@/${PSLOT}"

export DATAROOT="${STMP}/RUNDIRS/${PSLOT}" # TODO: set via prod_envir in Ops
export ARCDIR="${NOSCRUB}/archive/${PSLOT}"
export ATARDIR="@ATARDIR@"

Expand Down
1 change: 0 additions & 1 deletion parm/config/gfs/config.base
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,6 @@ export DUMP_SUFFIX=""
if [[ "${PDY}${cyc}" -ge "2019092100" && "${PDY}${cyc}" -le "2019110700" ]]; then
export DUMP_SUFFIX="p" # Use dumps from NCO GFS v15.3 parallel
fi
export DATAROOT="${STMP}/RUNDIRS/${PSLOT}" # TODO: set via prod_envir in Ops
export ARCDIR="${NOSCRUB}/archive/${PSLOT}"
export ATARDIR="@ATARDIR@"

Expand Down
11 changes: 11 additions & 0 deletions parm/config/gfs/config.fbwind
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#! /usr/bin/env bash

########## config.gempak ##########
# GFS fbwind step specific

echo "BEGIN: config.fbwind"

# Get task specific resources
source "${EXPDIR}/config.resources" fbwind

echo "END: config.fbwind"
7 changes: 7 additions & 0 deletions parm/config/gfs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -1124,6 +1124,13 @@ case ${step} in
memory_gfs="2GB"
;;

"fbwind")
walltime="00:05:00"
ntasks=1
threads_per_task=1
memory="4GB"
;;

"mos_stn_prep")
walltime="00:10:00"
ntasks=3
Expand Down
33 changes: 9 additions & 24 deletions scripts/exglobal_cleanup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,41 +12,26 @@ if [[ -d "${DATAfcst}" ]]; then rm -rf "${DATAfcst}"; fi
#DATAefcs="${DATAROOT}/${RUN}efcs???${PDY:-}${cyc}"
rm -rf "${DATAROOT}/${RUN}efcs"*"${PDY:-}${cyc}"

# Search and delete files/directories from DATAROOT/ older than ${purge_every_days} days
# purge_every_days should be a positive integer
#purge_every_days=3

# Find and delete files older than ${purge_every_days} days
#find "${DATAROOT}/"* -type f -mtime "+${purge_every_days}" -exec rm -f {} \;

# Find and delete directories older than ${purge_every_days} days
#find "${DATAROOT}/"* -type d -mtime "+${purge_every_days}" -exec rm -rf {} \;
# In XML, DATAROOT is defined as:
#DATAROOT="${STMP}/RUNDIRS/${PSLOT}/${RUN}.${PDY}${cyc}"
# cleanup is only executed after the entire cycle is successfully completed.
# removing DATAROOT should be possible if that is the case.
rm -rf "${DATAROOT}"

echo "Cleanup ${DATAROOT} completed!"
###############################################################

###############################################################
# Clean up previous cycles; various depths
# PRIOR CYCLE: Leave the prior cycle alone
# shellcheck disable=SC2153
GDATE=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} -${assim_freq} hours")
# PREVIOUS to the PRIOR CYCLE
GDATE=$(date --utc +%Y%m%d%H -d "${GDATE:0:8} ${GDATE:8:2} -${assim_freq} hours")

# Remove the TMPDIR directory
# TODO Only prepbufr is currently using this directory, and all jobs should be
# cleaning up after themselves anyway
COMIN="${DATAROOT}/${GDATE}"
[[ -d ${COMIN} ]] && rm -rf "${COMIN}"

if [[ "${CLEANUP_COM:-YES}" == NO ]] ; then
exit 0
fi

###############################################################
# Clean up previous cycles; various depths

# Step back every assim_freq hours and remove old rotating directories
# for successful cycles (defaults from 24h to 120h).
# Retain files needed by Fit2Obs
last_date=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} -${RMOLDEND:-24} hours" )
last_date=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} -${RMOLDEND:-24} hours")
first_date=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} -${RMOLDSTD:-120} hours")
last_rtofs=$(date --utc +%Y%m%d%H -d "${PDY} ${cyc} -${RMOLDRTOFS:-48} hours")
function remove_files() {
Expand Down
5 changes: 2 additions & 3 deletions ush/jjob_header.sh
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@
# [default: "YES"]
# - $pid : Override the default process id
# [default: $$]
#

OPTIND=1
while getopts "c:e:" option; do
Expand Down Expand Up @@ -99,7 +98,7 @@ for config in "${configs[@]:-''}"; do
status=$?
if (( status != 0 )); then
echo "FATAL [${BASH_SOURCE[0]}]: Unable to load config config.${config}"
exit "${status}"
exit "${status}"
fi
done

Expand All @@ -111,5 +110,5 @@ source "${HOMEgfs}/env/${machine}.env" "${env_job}"
status=$?
if (( status != 0 )); then
echo "FATAL [${BASH_SOURCE[0]}]: Error while sourcing machine environment ${machine}.env for job ${env_job}"
exit "${status}"
exit "${status}"
fi
4 changes: 3 additions & 1 deletion workflow/rocoto/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ def __init__(self, app_config: AppConfig, run: str) -> None:

self.n_tiles = 6 # TODO - this needs to be elsewhere

# DATAROOT is set by prod_envir in ops. Here, we use `STMP` to construct DATAROOT
dataroot_str = f"{self._base.get('STMP')}/RUNDIRS/{self._base.get('PSLOT')}/{self.run}.<cyclestr>@Y@m@d@H</cyclestr>"
envar_dict = {'RUN_ENVIR': self._base.get('RUN_ENVIR', 'emc'),
'HOMEgfs': self.HOMEgfs,
'EXPDIR': self._base.get('EXPDIR'),
Expand All @@ -70,7 +72,7 @@ def __init__(self, app_config: AppConfig, run: str) -> None:
'PDY': '<cyclestr>@Y@m@d</cyclestr>',
'cyc': '<cyclestr>@H</cyclestr>',
'COMROOT': self._base.get('COMROOT'),
'DATAROOT': self._base.get('DATAROOT')}
'DATAROOT': dataroot_str}

self.envars = self._set_envars(envar_dict)

Expand Down

0 comments on commit 8fee36f

Please sign in to comment.