From b4cf765ffff98693790de80a5530fcdb42d70e4b Mon Sep 17 00:00:00 2001 From: Omodara Date: Thu, 22 Aug 2024 08:32:56 -0400 Subject: [PATCH 1/3] GREEN-52125: Update webhook retry policy --- source/includes/webhooks/_introduction.md | 24 +++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/source/includes/webhooks/_introduction.md b/source/includes/webhooks/_introduction.md index 4eaa71915ff..7df070c33c4 100644 --- a/source/includes/webhooks/_introduction.md +++ b/source/includes/webhooks/_introduction.md @@ -188,14 +188,14 @@ If a webhook is disabled, it will not trigger when the event occurs. Webhooks be ## Retry policy -In the event of a failed webhook request (due to timeout, a non HTTP 200 response, or network issues), Greenhouse will attempt a maximum of 6 retries according to the formula on the right: +In the event of a failed webhook request (due to timeout, a non HTTP 200 response, or network issues), Greenhouse will make up to 6 attempts according to the formula on the right. This formula increases the amount of time between each retry, while assigning a random number of seconds to avoid consistent failures from overload or contention. -Greenhouse will attempt 6 retries over the course of 15 hours. +Greenhouse will make up to 6 attempts over the course of 7 hours. ```ruby -RETRY_DELAY_MINUTES = [1, 15, 60, 120, 240, 480] +RETRY_DELAY_MINUTES = [0, 1, 15, 60, 120, 240] sidekiq_retry_in do |index, _exception| seconds_delay = (RETRY_DELAY_MINUTES[index] || RETRY_DELAY_MINUTES.last) * 60 @@ -205,13 +205,13 @@ sidekiq_retry_in do |index, _exception| end ``` -The table below outlines the estimated wait time for each retry request, assuming that rand(30) always returns 0. +The table below outlines the estimated wait time for each attempt, assuming that rand(30) always returns 0. -| Retry number | Next Retry in | Total waiting time | -| ------------ | ------------- | ------------------ | -| 1 | 1m | 0h 1m | -| 2 | 15m | 0h 16m | -| 3 | 60m | 1h 16m | -| 4 | 120m | 3h 16m | -| 5 | 240m | 7h 16m | -| 6 | 480m | 15h 16m | +| Attempt number | Next Attempt in | Total waiting time | +| -------------- | --------------- | ------------------ | +| 1 | 1m | 0m | +| 2 | 15m | 0h 1m | +| 3 | 60m | 0h 16m | +| 4 | 120m | 1h 16m | +| 5 | 240m | 3h 16m | +| 6 | -- | 7h 16m | From e61000c476d9670d1dea4c3038994626bf8a1b53 Mon Sep 17 00:00:00 2001 From: Omodara Date: Thu, 22 Aug 2024 16:54:07 -0400 Subject: [PATCH 2/3] GREEN-52125: Update retry policy and associated table. Remove retry formula. --- source/includes/webhooks/_introduction.md | 34 +++++++---------------- 1 file changed, 10 insertions(+), 24 deletions(-) diff --git a/source/includes/webhooks/_introduction.md b/source/includes/webhooks/_introduction.md index 7df070c33c4..feaa1f7ca9e 100644 --- a/source/includes/webhooks/_introduction.md +++ b/source/includes/webhooks/_introduction.md @@ -188,30 +188,16 @@ If a webhook is disabled, it will not trigger when the event occurs. Webhooks be ## Retry policy -In the event of a failed webhook request (due to timeout, a non HTTP 200 response, or network issues), Greenhouse will make up to 6 attempts according to the formula on the right. +In the event of a failed webhook request (due to timeout, a non HTTP 200 response, or network issues), Greenhouse will make up to 6 attempts over the course of 7 hours. -This formula increases the amount of time between each retry, while assigning a random number of seconds to avoid consistent failures from overload or contention. -Greenhouse will make up to 6 attempts over the course of 7 hours. +The table below outlines the estimated wait time for each attempt. -```ruby -RETRY_DELAY_MINUTES = [0, 1, 15, 60, 120, 240] - -sidekiq_retry_in do |index, _exception| - seconds_delay = (RETRY_DELAY_MINUTES[index] || RETRY_DELAY_MINUTES.last) * 60 - offset = rand(30) * (index + 1) - - seconds_delay + offset -end -``` - -The table below outlines the estimated wait time for each attempt, assuming that rand(30) always returns 0. - -| Attempt number | Next Attempt in | Total waiting time | -| -------------- | --------------- | ------------------ | -| 1 | 1m | 0m | -| 2 | 15m | 0h 1m | -| 3 | 60m | 0h 16m | -| 4 | 120m | 1h 16m | -| 5 | 240m | 3h 16m | -| 6 | -- | 7h 16m | +| Approx. Total waiting time | Attempt number | Next Attempt in | +| :--------------------------: | :--------------: | :---------------: | +| 0h 0m | 1 | 1m | +| 0h 1m | 2 | 15m | +| 0h 16m | 3 | 60m | +| 1h 16m | 4 | 120m | +| 3h 16m | 5 | 240m | +| 7h 16m | 6 | -- | From 9d8f628ede88f0c3b085e987be72f9ed89fbc4af Mon Sep 17 00:00:00 2001 From: Omodara Date: Wed, 28 Aug 2024 18:50:53 -0400 Subject: [PATCH 3/3] GREEN-52125: Update retry policy to 7 attempts over 15 hours --- source/includes/webhooks/_introduction.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/source/includes/webhooks/_introduction.md b/source/includes/webhooks/_introduction.md index feaa1f7ca9e..11662ef0300 100644 --- a/source/includes/webhooks/_introduction.md +++ b/source/includes/webhooks/_introduction.md @@ -188,16 +188,17 @@ If a webhook is disabled, it will not trigger when the event occurs. Webhooks be ## Retry policy -In the event of a failed webhook request (due to timeout, a non HTTP 200 response, or network issues), Greenhouse will make up to 6 attempts over the course of 7 hours. +In the event of a failed webhook request (due to timeout, a non HTTP 200 response, or network issues), Greenhouse will make up to 7 attempts over the course of 15 hours. The table below outlines the estimated wait time for each attempt. | Approx. Total waiting time | Attempt number | Next Attempt in | -| :--------------------------: | :--------------: | :---------------: | -| 0h 0m | 1 | 1m | -| 0h 1m | 2 | 15m | -| 0h 16m | 3 | 60m | -| 1h 16m | 4 | 120m | -| 3h 16m | 5 | 240m | -| 7h 16m | 6 | -- | +| :-------------------------- | :--------------: | :--------------- | +| 0h 0m | 1 | 1m | +| 0h 1m | 2 | 15m | +| 0h 16m | 3 | 60m | +| 1h 16m | 4 | 120m | +| 3h 16m | 5 | 240m | +| 7h 16m | 6 | 480m | +| 15h 16m | 7 | -- |