x/tools/cmd/godoc: improve diagnostics for server hangs in TestWeb/GOPATH #50014

bcmills · 2021-12-07T03:31:28Z

--- FAIL: TestWeb (37.91s)
    --- FAIL: TestWeb/GOPATH (15.11s)
        godoc_test.go:86: server failed to respond in 15s
FAIL
FAIL	golang.org/x/tools/cmd/godoc	68.450s

greplogs --dashboard -md -l -e 'FAIL: TestWeb .*\n\s+.*\n\s+.*server failed to respond'

The text was updated successfully, but these errors were encountered:

bcmills · 2021-12-07T03:38:37Z

At this failure rate, this looks to me like a release-blocker via #11811. The test should either be fixed or skipped on the affected builders.

Looking at the test, I see at least three problems with the timeout strategy:

If the server actually wedges, its output logs (and, critically, its goroutine dump) is not logged anywhere. (The process is killed with cmd.Process.Kill instead of a signal that would give it the opportunity to dump goroutines.)
The deadlines are hard-coded, inconsistently, without comments explaining why those deadlines should be fundamental. (One server gets only 15s, while the others get two minutes each.)

Probably for this issue we can just fix (2), although it would be nice to address (1) as well.

bcmills · 2021-12-07T03:41:11Z

Looking at this further, the failures are always in TestWeb/GOPATH. Perhaps we should just skip that subtest in -short mode, since at this point most users are in module mode anyway.

bcmills · 2021-12-10T16:11:27Z

TestWeb/GOPATH takes less than a second on my workstation. I think this is uncovering a real bug in either godoc or the test itself.`

cherrymui · 2021-12-16T00:21:42Z

cc @dmitshur

gopherbot · 2021-12-17T05:55:42Z

Change https://golang.org/cl/373005 mentions this issue: internal/moreexec: add a utility package for manipulating os/exec.Cmd

gopherbot · 2022-01-12T03:46:06Z

Change https://golang.org/cl/377835 mentions this issue: internal/testenv: add a Command function that replaces exec.Command

gopherbot · 2022-01-12T03:48:59Z

Change https://golang.org/cl/377836 mentions this issue: cmd/godoc: streamline subprocesses

bcmills · 2022-01-12T03:53:34Z

The test needs better diagnostics. Proposal #50436 is at the bottom of this yak stack, but in the meantime we can polyfill the needed API into an internal package.

bcmills · 2022-01-21T21:05:23Z

Intriguingly, the dragonfly-amd64 failures are no longer occurring after the builder was upgraded for #50538 (thanks, @tuxillo!).

We should still add better failure diagnostics to this test, but since it is no longer failing on the builders it won't mask other regressions, and doesn't need to be a release-blocker.

bcmills · 2022-02-24T20:11:26Z

Now seen on windows-arm64-11 too, so this appears to be completely platform-independent.

greplogs --dashboard -md -l -e 'FAIL: TestWeb .*\n\s+.*\n\s+.*server failed to respond' --since=2022-01-21

2022-02-17T17:38:22-411d040-eaf0405/windows-arm64-11

bcmills · 2022-03-02T19:45:02Z

greplogs --dashboard -md -l -e 'FAIL: TestWeb .*\n\s+.*\n\s+.*server failed to respond' --since=2022-02-24

2022-03-02T11:33:18-ffa170d-fd2e1e7/dragonfly-amd64
2022-03-01T16:32:20-fb3622a-2199841/darwin-amd64-11_0

bcmills · 2022-03-02T19:46:10Z

Marking as release-blocker for Go 1.19 due to the recent failure on the darwin-amd64-11_0 builder. (darwin/amd64 is a first class port.)

bcmills · 2022-04-25T15:27:19Z

Lots of these on dragonfly-amd64 and windows-arm64-11, and a few on other ports too.

greplogs --dashboard -md -l -e 'FAIL: TestWeb .*\n\s+.*\n\s+.*server failed to respond' --since=2022-03-03

gopherbot · 2022-05-02T18:59:57Z

Change https://go.dev/cl/403534 mentions this issue: cmd/godoc: skip TestWeb if waitForServerReady fails

This test fails frequently, with a failure mode that is difficult to diagnose. (golang/go#50436 may help with that eventually.) For now, skip the test to reduce noise on the build dashboard. For golang/go#50014. Change-Id: I182be5c705846631c983bd5b6c51ab90b71a216a Reviewed-on: https://go-review.googlesource.com/c/tools/+/403534 Reviewed-by: Dmitri Shuralyov <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Auto-Submit: Bryan Mills <[email protected]> gopls-CI: kokoro <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

gopherbot · 2022-05-04T14:14:48Z

Change https://go.dev/cl/403849 mentions this issue: cmd/godoc: expand skips in TestWeb

After CL 403534, another TestWeb failure was observed in https://build.golang.org/log/24cca44e99d848317a6bb73b6d15528b500e5643. This change expands the skips to try to cover more of the cases where cmd/godoc might hang. For golang/go#50014. Change-Id: If3f08da69a68dc00c399b89ffb2c4f049114aeb6 Reviewed-on: https://go-review.googlesource.com/c/tools/+/403849 Auto-Submit: Bryan Mills <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> gopls-CI: kokoro <[email protected]> TryBot-Result: Gopher Robot <[email protected]>

bcmills · 2022-05-13T14:03:15Z

This test now has skips for these failure modes, so it is no longer a release-blocker.

abhishekdighePersIbm1 · 2022-08-11T08:25:15Z

Hello Team,

I am facing issue mentioned below

--- FAIL: TestWeb (7.99s)
--- FAIL: TestWeb/GOPATH (1.44s)
godoc_test.go:423: GET http://127.0.0.1:46541/src/fmt/: wanted substring "scan_test.go" in body
godoc_test.go:446: GET http://127.0.0.1:46541/src/fmt/: got:
Let me know if there is any progress/solution?

The function is derived from the testenv.Command function in the main repo as of CL 446875, with a couple of modifications to allow it to build (with more limited functionality) with Go versions as old as 1.16 (currently needed in order to test gopls with such versions). testenv.Command sets up an exec.Cmd with more useful termination behavior in the context of a test: namely, it is terminated with SIGQUIT (to get a goroutine dump from the subprocess) shortly before the test would otherwise time out. Assuming that the test logs the output from the command appropriately, this should make deadlocks and unexpectedly slow operations easier to diagnose in the builders. For golang/go#50014. Updates golang/go#50436. Change-Id: I872d4b24e63951bf9b7811189e672973d366fb78 Reviewed-on: https://go-review.googlesource.com/c/tools/+/377835 Reviewed-by: Dmitri Shuralyov <[email protected]> Reviewed-by: Joedian Reid <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Auto-Submit: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Robert Findley <[email protected]> gopls-CI: kokoro <[email protected]>

- Use testenv.Command to obtain Cmd instances that terminate (with useful goroutine dumps!) before the test's timeout, and remove arbitrary hard-coded timeouts. - Execute the test binary itself as cmd/godoc instead of invoking (and cleaning up after) 'go build'. - Use context cancellation to reduce the number of ad-hoc goroutines and channels needed by the tests and to provide stronger invariants on process cleanup. For golang/go#50014 Change-Id: I19ae4d10da691db233c79734799ae074ffdf6a03 Reviewed-on: https://go-review.googlesource.com/c/tools/+/377836 Run-TryBot: Bryan Mills <[email protected]> Reviewed-by: Joedian Reid <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]> TryBot-Result: Gopher Robot <[email protected]> gopls-CI: kokoro <[email protected]> Auto-Submit: Bryan Mills <[email protected]>

bcmills · 2022-11-21T16:59:35Z

Using x/tools/internal/testenv CL 50014 should produce better diagnostics in case of hang. If that exposes any other failure modes we can handle those as separate issues.

bcmills added Testing An issue that has been verified to require only test changes, not just a test failure. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 7, 2021

gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Dec 7, 2021

gopherbot added this to the Unreleased milestone Dec 7, 2021

bcmills modified the milestones: Unreleased, Go1.18 Dec 7, 2021

bcmills added okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker labels Dec 7, 2021

bcmills changed the title ~~x/tools/cmd/godoc: frequent failures in TestWeb on dragonfly-amd64 builder~~ x/tools/cmd/godoc: frequent failures in TestWeb/GOPATH on dragonfly-amd64 builder Dec 7, 2021

cherrymui removed the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Dec 14, 2021

bcmills mentioned this issue Jan 4, 2022

os/exec: add fields for managing termination signals and pipes #50436

Closed

bcmills self-assigned this Jan 12, 2022

dmitshur self-assigned this Jan 12, 2022

bcmills removed the release-blocker label Jan 21, 2022

bcmills modified the milestones: Go1.18, Backlog Jan 21, 2022

bcmills changed the title ~~x/tools/cmd/godoc: frequent failures in TestWeb/GOPATH on dragonfly-amd64 builder~~ x/tools/cmd/godoc: improve diagnostics for server hangs in TestWeb/GOPATH Feb 3, 2022

bcmills added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 3, 2022

gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 3, 2022

bcmills removed this from the Backlog milestone Feb 24, 2022

bcmills added this to the Go1.19 milestone Feb 24, 2022

bcmills added the release-blocker label Mar 2, 2022

bcmills mentioned this issue May 2, 2022

build dashboard triage log #52653

Closed

bcmills removed the release-blocker label May 13, 2022

dmitshur removed their assignment May 17, 2022

bcmills modified the milestones: Go1.19, Backlog Jun 1, 2022

bcmills closed this as completed Nov 21, 2022

golang locked and limited conversation to collaborators Nov 21, 2023

gopherbot added the FrozenDueToAge label Nov 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x/tools/cmd/godoc: improve diagnostics for server hangs in TestWeb/GOPATH #50014

x/tools/cmd/godoc: improve diagnostics for server hangs in TestWeb/GOPATH #50014

bcmills commented Dec 7, 2021

bcmills commented Dec 7, 2021

bcmills commented Dec 7, 2021

bcmills commented Dec 10, 2021

cherrymui commented Dec 16, 2021

gopherbot commented Dec 17, 2021

gopherbot commented Jan 12, 2022

gopherbot commented Jan 12, 2022

bcmills commented Jan 12, 2022

bcmills commented Jan 21, 2022

bcmills commented Feb 24, 2022

bcmills commented Mar 2, 2022

bcmills commented Mar 2, 2022

bcmills commented Apr 25, 2022

gopherbot commented May 2, 2022

gopherbot commented May 4, 2022

bcmills commented May 13, 2022

abhishekdighePersIbm1 commented Aug 11, 2022 •

edited

Loading

bcmills commented Nov 21, 2022

x/tools/cmd/godoc: improve diagnostics for server hangs in TestWeb/GOPATH #50014

x/tools/cmd/godoc: improve diagnostics for server hangs in TestWeb/GOPATH #50014

Comments

bcmills commented Dec 7, 2021

bcmills commented Dec 7, 2021

bcmills commented Dec 7, 2021

bcmills commented Dec 10, 2021

cherrymui commented Dec 16, 2021

gopherbot commented Dec 17, 2021

gopherbot commented Jan 12, 2022

gopherbot commented Jan 12, 2022

bcmills commented Jan 12, 2022

bcmills commented Jan 21, 2022

bcmills commented Feb 24, 2022

bcmills commented Mar 2, 2022

bcmills commented Mar 2, 2022

bcmills commented Apr 25, 2022

gopherbot commented May 2, 2022

gopherbot commented May 4, 2022

bcmills commented May 13, 2022

abhishekdighePersIbm1 commented Aug 11, 2022 • edited Loading

bcmills commented Nov 21, 2022

abhishekdighePersIbm1 commented Aug 11, 2022 •

edited

Loading