Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

weird error at PR initialization #1453

Closed
dimisjim opened this issue Mar 12, 2021 · 7 comments
Closed

weird error at PR initialization #1453

dimisjim opened this issue Mar 12, 2021 · 7 comments
Labels
waiting-on-response Waiting for a response from the user

Comments

@dimisjim
Copy link
Contributor

Hey there,

I just switched to the latest build (8ca6e164749a4fa6509202124c775769fa17c103)and getting this error when:

GET https://api.github.com/repos/myorg/myrepo/pulls/1313/files?per_page=300: 404 Not Found []
@nishkrishnan
Copy link
Contributor

Need more information here to debug as the latest build works for me.

@nishkrishnan nishkrishnan added the waiting-on-response Waiting for a response from the user label Mar 17, 2021
@dimisjim
Copy link
Contributor Author

dimisjim commented Mar 24, 2021

This appeared only once. It has not happened again ever since. I'll reopen the issue if it reappears with some proper debug info.

@Ninir
Copy link
Contributor

Ninir commented Apr 6, 2021

Hi folks, getting this issue on several PRs (intermittently) using the latest master (ca976d8). We have dozens of PRs a day, and it's not happening on all PRs.
Which kind of debug information do you need?

@nishkrishnan
Copy link
Contributor

Is this consistently happening on only one repo? Or a variety of repos

When this happens, is there a Github incident going on? You can check that online

What is the typical number of files in the PRs that these happen on?

Is there any chance there are network blips occurring with your setup?

Debug logs for the whole command would be useful here.

@dkuebric
Copy link

dkuebric commented Apr 7, 2021

We are experiencing this periodically as well. For a separate issue, I've noticed that GitHub APIs seem to be eventually consistent, and this impacts services that consume webhooks and follow up with API calls. To be more specific, the other use-case I've seen is a review-event webhook that triggers a ListReviews call; the call response is missing the exact review triggering the webhook event. Retrying just a bit later returns the expected result. I wouldn't be surprised if something similar is happening here.

Based on that thinking, it seems like adding retries with backoff on 404 would be a good solution. Does that sound reasonable? My colleague @nrmitchi is thinking of taking a pass at this on our local fork.

@dimisjim
Copy link
Contributor Author

dimisjim commented Apr 7, 2021

I have experienced this again one more time, so based on other accounts here, I am reopening the issue.

@dimisjim dimisjim reopened this Apr 7, 2021
@nrmitchi
Copy link

nrmitchi commented Apr 7, 2021

For some more context, this appears to be a duplicate of #1019, which was addressed in #1131

Either there is a different underlying cause this time, or github just got a little bit more eventually-consistent

@github-actions github-actions bot added the Stale label Jul 27, 2021
@github-actions github-actions bot closed this as completed Aug 1, 2021
jamengual pushed a commit that referenced this issue Oct 4, 2021
* Improve github pull request call retries

Retry with fixed 1 second backoff up to 3 retries was added by #1131 to
address #1019, but the issue continued to show up (#1453).

Increase max attempts to 5 and use exponential backoff for a maximum
total retry time of (2^n - n - 1) seconds, which is roughly 30 seconds
for current max attempts n = 5.

Also move the sleep to the top of the loop so that we never sleep
without sending the request again on the last iteration.

* Fix style with gofmt -s
krrrr38 pushed a commit to krrrr38/atlantis that referenced this issue Dec 16, 2022
* Improve github pull request call retries

Retry with fixed 1 second backoff up to 3 retries was added by runatlantis#1131 to
address runatlantis#1019, but the issue continued to show up (runatlantis#1453).

Increase max attempts to 5 and use exponential backoff for a maximum
total retry time of (2^n - n - 1) seconds, which is roughly 30 seconds
for current max attempts n = 5.

Also move the sleep to the top of the loop so that we never sleep
without sending the request again on the last iteration.

* Fix style with gofmt -s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
waiting-on-response Waiting for a response from the user
Projects
None yet
Development

No branches or pull requests

5 participants