Skip to content

Commit

Permalink
Add timeout to test jobs to prevent hanging docker containers (#15078)
Browse files Browse the repository at this point in the history
Some of the test jobs are hanging - either becasue of some
weird race conditions in docker or because the test hangs (happens
for quarantined tests). This change add maximum timeout we let
the test suite execute to 25 minutes.
  • Loading branch information
potiuk authored Mar 30, 2021
1 parent 741a545 commit a4aee3f
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion scripts/ci/testing/ci_run_airflow_testing.sh
Original file line number Diff line number Diff line change
Expand Up @@ -160,9 +160,10 @@ function run_test_types_in_parallel() {
mkdir -p "${PARALLEL_MONITORED_DIR}/${SEMAPHORE_NAME}/${TEST_TYPE}"
export JOB_LOG="${PARALLEL_MONITORED_DIR}/${SEMAPHORE_NAME}/${TEST_TYPE}/stdout"
export PARALLEL_JOB_STATUS="${PARALLEL_MONITORED_DIR}/${SEMAPHORE_NAME}/${TEST_TYPE}/status"
# Each test job will get SIGTERM followed by SIGTERM 200ms later and SIGKILL 200ms later after 25 mins
# shellcheck disable=SC2086
parallel --ungroup --bg --semaphore --semaphorename "${SEMAPHORE_NAME}" \
--jobs "${MAX_PARALLEL_TEST_JOBS}" \
--jobs "${MAX_PARALLEL_TEST_JOBS}" --timeout 1500 \
"$( dirname "${BASH_SOURCE[0]}" )/ci_run_single_airflow_test_in_docker.sh" "${@}" >${JOB_LOG} 2>&1
done
parallel --semaphore --semaphorename "${SEMAPHORE_NAME}" --wait
Expand Down

0 comments on commit a4aee3f

Please sign in to comment.