Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: flaky network or bad 'git' version on linux-386-longtest builder #49953

Closed
bcmills opened this issue Dec 3, 2021 · 4 comments
Closed

x/build: flaky network or bad 'git' version on linux-386-longtest builder #49953

bcmills opened this issue Dec 3, 2021 · 4 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@bcmills
Copy link
Contributor

bcmills commented Dec 3, 2021 •
edited
Loading

The cmd/go tests involving git requests to external services (mainly golang.org, github.com, gopkg.in) are expected to have a slightly nonzero failure rate due to external service outages. However, the failure rate on the linux-386-longtest builder in particular seems to have increased significantly starting in September, without a corresponding increase in the failure rate on the linux-amd64-longtest builder, and many of the failures are to the Go project's own Git server (go.googlesource.com).

It isn't clear to me whether these failures are due to network drops, or perhaps bugs in the builder's git implementation causing the upstream git server to drop the connection.

@golang/release, were there any builder image or configuration changes around September that might explain the increase in failure rate? Could we talk with the go.googlesource.com admins about whether they have also seen a higher error rate on the server side in the same timeframe?

greplogs --dashboard -md -l -e 'fatal: The remote end hung up unexpectedly' --since=2021-01-01

2021-12-02T22:05:56-e533b57/linux-386-longtest
2021-12-02T22:04:50-bcb98f1/linux-386-longtest
2021-12-02T22:03:54-25f06cb/linux-386-longtest
2021-12-02T07:04:05-d34051b/linux-386-longtest
2021-12-01T17:01:41-8ebb8c9/linux-386-longtest
2021-11-30T15:28:46-682435d/linux-386-longtest
2021-11-29T16:08:23-a59ab29/linux-amd64-longtest
2021-11-24T21:11:45-b77f5f9/linux-amd64-longtest
2021-11-24T21:11:34-8cdfe40/linux-386-longtest
2021-11-24T20:59:14-67dd9ee/linux-386-longtest
2021-11-24T20:57:46-c25bf0d/linux-386-longtest
2021-11-24T20:54:12-696515e/linux-386-longtest
2021-11-24T20:54:07-6ea17aa/linux-386-longtest
2021-11-16T19:41:37-f659183/linux-386-longtest
2021-11-16T15:38:59-7f4a946/linux-386-longtest
2021-11-15T18:02:28-1dc9af5/linux-386-longtest
2021-11-15T17:10:25-f986191/linux-amd64-longtest
2021-11-11T15:34:02-8c73f80/linux-386-longtest
2021-11-11T11:47:08-e9ef931/linux-386-longtest
2021-11-05T21:27:19-58ec925/linux-386-longtest
2021-11-05T16:51:14-c58417b/linux-386-longtest
2021-11-04T23:56:29-0e5f287/linux-386-longtest
2021-11-03T18:37:22-cfd016d/linux-386-longtest
2021-11-02T15:51:21-c3cb1ec/linux-386-longtest
2021-11-01T22:55:50-02e5913/linux-amd64-longtest
2021-10-29T19:30:03-4592933/linux-amd64-longtest
2021-10-28T16:08:36-6f0185b/linux-386-longtest
2021-10-26T21:17:38-091db63/linux-amd64-longtest
2021-10-21T18:19:51-6508fda/linux-386-longtest
2021-09-16T23:56:28-265b59a/linux-386-longtest
2021-09-15T16:32:27-72bb818/linux-386-longtest
2021-09-14T23:03:28-3a72175/linux-386-longtest
2021-09-14T23:03:28-3a72175/linux-amd64-longtest
2021-09-14T00:29:16-71adc65/linux-386-longtest
2021-09-13T22:57:34-9a58aa2/linux-386-longtest
2021-09-02T17:30:32-7609b50/linux-386-longtest
2021-08-12T17:54:49-5805efc/linux-amd64-longtest
2021-06-28T23:31:05-1519271/linux-amd64-longtest
2021-05-26T22:43:54-1d5298d/linux-amd64-longtest
2021-04-13T21:13:11-e69f022/linux-386-longtest
2021-03-30T03:31:44-bb2fc21/linux-386-longtest
2021-03-15T21:49:19-de2b27d/linux-amd64-longtest
2021-03-08T18:56:25-b478720/linux-386-longtest
2021-02-09T05:28:50-e0ac989/linux-amd64-longtest
2021-01-27T21:11:22-00f2ff5/linux-386-longtest
2021-01-27T21:11:22-00f2ff5/linux-amd64-longtest
2021-01-27T21:04:09-35334ca/linux-386-longtest
2021-01-27T21:04:09-35334ca/linux-amd64-longtest
2021-01-27T20:44:24-a5a5e2c/linux-386-longtest
2021-01-27T20:44:24-a5a5e2c/linux-amd64-longtest
@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 3, 2021
@gopherbot gopherbot added this to the Unreleased milestone Dec 3, 2021
@bcmills

This comment has been minimized.

Sign in to view
@bcmills

This comment has been minimized.

Sign in to view
@bcmills
Copy link
Contributor Author

bcmills commented May 6, 2022

greplogs -l -e 'fatal: The remote end hung up unexpectedly' --since=2022-01-01
2022-02-07T16:28:43-faa0409/linux-386-longtest
2022-02-03T20:56:27-1b867ce/linux-386-longtest
2022-01-21T16:59:19-9eba5ff/linux-amd64-longtest
2022-01-19T16:23:21-bec2cc3/linux-386-longtest
2022-01-19T14:29:17-2a061fd/linux-386-longtest
2022-01-14T18:36:49-4f0c32d/linux-amd64-longtest
2022-01-11T21:56:11-13c912d/linux-386-longtest
2022-01-10T22:48:40-6019a52/linux-amd64-longtest

@bcmills
Copy link
Contributor Author

bcmills commented May 6, 2022

It looks like this failure mode disappeared right about the same time that #52490 started. 🤔

Closing on the theory that this bug may have been replaced with a similar-but-different one.

@bcmills bcmills closed this as completed May 6, 2022
@golang golang locked and limited conversation to collaborators May 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
2 participants
@bcmills @gopherbot