Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: instances of all internet-facing tests failing #54279

Open
dmitshur opened this issue Aug 5, 2022 · 2 comments
Open

x/build: instances of all internet-facing tests failing #54279

dmitshur opened this issue Aug 5, 2022 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Aug 5, 2022 •
edited
Loading 

#!watchflakes
post <- repo == "build" && `https:\/\/maintner\.golang\.org\/.+: 502 Bad Gateway`

Issue created to collect instances where internet-facing tests fail with errors:

[...]
2022/07/28 00:33:59 syncing segment 136 did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs/136: 502 Bad Gateway
2022/07/28 00:34:08 Downloading 15228181 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:17 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs/136: 502 Bad Gateway"}
2022/07/28 00:34:17 syncing segment 136 did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs/136: 502 Bad Gateway
2022/07/28 00:34:33 Downloading 15228181 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:34 Unknown error type *errors.errorString: &errors.errorString{s:"corrupt download"}
2022/07/28 00:34:34 Corpus GetMutations: corrupt download
--- FAIL: TestCorpusCheck (112.99s)
    godata_test.go:52: getting corpus: corrupt download
2022/07/28 00:34:34 Loading data from log *maintner.netMutSource ...
2022/07/28 00:34:34 Downloading 15186069 bytes of https://maintner.golang.org/logs/136 ...
2022/07/28 00:34:35 wrote /root/.cache/golang-maintner/0136.growing.mutlog
2022/07/28 00:35:16 Reloaded data from log *maintner.netMutSource.
FAIL
FAIL	golang.org/x/build/maintner/godata	156.319s
[...]
--- FAIL: TestFindWork (9.05s)
    coordinator_test.go:309: bad http response: 500 Internal Server Error
2022/07/28 00:33:04 gRPCServer.ClearResults: bad status from dashboard: 400 ("400 Bad Request")
2022/07/28 00:33:04 gRPCServer.ClearResults: bad status from dashboard: 308 ("308 Permanent Redirect")
FAIL
FAIL	golang.org/x/build/cmd/coordinator	9.164s
[...]
@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 5, 2022
@dmitshur dmitshur added this to the Unreleased milestone Aug 5, 2022
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Aug 5, 2022
@dmitshur dmitshur mentioned this issue Aug 5, 2022
Closed
@bcmills bcmills mentioned this issue Apr 18, 2023
Closed
This was referenced Jun 18, 2024
Closed
Closed
@dmitshur
Copy link
Contributor Author

The likely explanation here is that the k8s cluster was undergoing an upgrade, resulting in brief windows of service unavailability.

Let's reuse this issue to catch future instances, so that watchflakes doesn't keep opening new issues per package/test.

@dmitshur dmitshur moved this to Active in Test Flakes Jun 18, 2024
This was referenced Jun 18, 2024
Closed
Closed
@gopherbot
Copy link
Contributor

Found new dashboard test flakes for:

#!watchflakes
post <- repo == "build" && `https:\/\/maintner\.golang\.org\/.+: 502 Bad Gateway`
2024-06-17 17:07 x_build-go1.21-windows-amd64-longtest build@def93586 release-branch.go1.21@74ac37e9 x/build/maintner/godata.TestCorpusCheck (log) 
=== RUN   TestCorpusCheck
2024/06/17 10:16:19 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:28 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:28 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:38 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:38 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:51 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:51 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:09 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:09 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:34 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:34 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestCorpusCheck (75.14s)
2024-06-17 17:07 x_build-go1.21-windows-amd64-longtest build@def93586 release-branch.go1.21@74ac37e9 x/build/maintner/maintnerd/maintapi.TestGetRef (log) 
=== RUN   TestGetRef
2024/06/17 10:16:20 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:29 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:29 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:39 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:39 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:52 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:10 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:10 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:35 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:35 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    api_test.go:475: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestGetRef (75.12s)
2024-06-17 17:07 x_build-go1.22-darwin-amd64-longtest build@def93586 release-branch.go1.22@fe9b3c33 x/build/maintner/godata.TestCorpusCheck (log) 
=== RUN   TestCorpusCheck
2024/06/17 10:16:26 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:35 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:35 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:45 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:45 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:16:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:58 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:16 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:16 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:41 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:41 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestCorpusCheck (75.37s)
2024-06-17 17:07 x_build-gotip-windows-amd64-longtest build@def93586 go@8f5c6904 x/build/maintner/godata.TestGerritForeachNonChangeRef (log) 
=== RUN   TestGerritForeachNonChangeRef
2024/06/17 10:16:43 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:52 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:02 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:02 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:15 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:15 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:33 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:33 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:58 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    godata_test.go:52: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestGerritForeachNonChangeRef (75.02s)
2024-06-17 17:07 x_build-gotip-windows-amd64-longtest build@def93586 go@8f5c6904 x/build/maintner/maintnerd/maintapi.TestTryBotStatus (log) 
=== RUN   TestTryBotStatus
2024/06/17 10:16:43 Loading data from log *maintner.netMutSource ...
2024/06/17 10:16:52 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:16:52 fetching server segments did not succeed on attempt 1, will try again in 1s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:02 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:02 fetching server segments did not succeed on attempt 2, will try again in 4s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:15 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:15 fetching server segments did not succeed on attempt 3, will try again in 9s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:33 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:33 fetching server segments did not succeed on attempt 4, will try again in 16s: https://maintner.golang.org/logs: 502 Bad Gateway
2024/06/17 10:17:58 Unknown error type *errors.errorString: &errors.errorString{s:"https://maintner.golang.org/logs: 502 Bad Gateway"}
2024/06/17 10:17:58 Corpus GetMutations: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
    api_test.go:475: getting corpus: after 5 attempts, fetching server segments still failed: https://maintner.golang.org/logs: 502 Bad Gateway
--- FAIL: TestTryBotStatus (75.04s)

— watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Test Flakes
Status: Active
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@dmitshur @gopherbot