go/build: importing with UseAllFiles on Go 1.20 and 1.21 fails for build-constrained files that use Go 1.22 #cgo directives

[corhere]

What version of Go are you using (go version)?

$ go version
go version go1.21.1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/corysnider/Library/Caches/go-build'
GOENV='/Users/corysnider/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Volumes/Workspace/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Volumes/Workspace'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/Cellar/go/1.21.1/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/Cellar/go/1.21.1/libexec/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.21.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/corysnider/tmp/cgovendorbug/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/8p/tzn5bkn967vfgttpr2d2kpjm0000gq/T/go-build1712739847=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

$ go mod init example.com/cgovendorbug
go: creating new go.mod: module example.com/cgovendorbug
$  go get github.com/golang-fips/openssl/v2
go: added github.com/golang-fips/openssl/v2 v2.0.0-rc.3
$ cat > main.go
package main

import _ "github.com/golang-fips/openssl/v2"

func main() {}
^D
$ go mod vendor

What did you expect to see?

No output; go mod vendor succeeds.

What did you see instead?

internal error: failed to find embedded files of github.com/golang-fips/openssl/v2: /Volumes/Workspace/pkg/mod/github.com/golang-fips/openssl/[email protected]/cgo_go122.go: invalid #cgo line: #cgo noescape go_openssl_EVP_PKEY_derive

The cgo_go122.go file has a //go:build go1.22 build constraint so the module is otherwise compatible with Go 1.21; it builds without error. The module just cannot be vendored.

[bcmills]

Huh, yep. This was probably introduced in https://go.dev/cl/283641, and is related to this TODO:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/modcmd/vendor.go;l=304-306;drc=33cdafed5213a17d23baa92568ede4bb16e7af0c

It looks like we're also missing a vendorE check to allow go mod vendor -e to push past the error. 🤦‍♂️

The right solution might be to always ignore errors from ctxt.ImportDir, although that's not entirely obvious to me.

[qmuntal]

Even if this issue is not new, given that it now will occur more due to the addition of the new #cgo directives in go1.22, would it make sense to mark this as release blocker?

[bcmills]

I think we should consider reworking the new #cgo directive so that it doesn't break the parsers in older releases. Marking as release-blocker to consider the options.

[bcmills]

@golang/compiler: what are your opinions on adjusting the directive syntax?

[cherrymui]

@bcmills do you have a suggestion for a different syntax? Is that any #cgo directive that is not previously recognized would error out? That means we cannot use #cgo directive for this?

[bcmills]

Yes, it looks that way. The Go 1.21 parser implementation is here:
https://cs.opensource.google/go/go/+/release-branch.go1.21:src/go/build/build.go;l=1674-1751;drc=d22f287f12f4782082cd93785ad91e78dbe0d4d6

Notably, it has a default: return fmt.Errorf(…) at the end of each line.

To me, that suggests that maybe we could use a //go:-style directive instead?

[mknyszek]

In compiler/runtime triage, we're wondering if it's an option to update older supported releases to accept the new cgo directive?

[bcmills]

That may be an option, yes. It would involve at least backporting portions of https://go.dev/cl/497837 in the go/build and cmd/go/internal/modindex packages.

[cherrymui]

Another option would be have the go command to not error for directives in files with a build tag of newer version of Go. This would also help if in the future we add other cgo directives.

Would syntax like #cgo go1.22 noescape help? I guess the go command is already capable to understand the go:build build tags, so maybe this is not really necessary?

[bcmills]

Another option would be have the go command to not error for directives in files with a build tag of newer version of Go.

Yep, we should probably do that but it's a somewhat more involved change, and of course it only helps once a fix is backported and released and users upgrade to it. 🙃

Would syntax like #cgo go1.22 noescape help?

Unfortunately no: any program that uses go/build to parse files containing #cgo directives will end up rejecting any #cgo directive that doesn't match one of the existing tags.

(Perhaps for the purpose of future-proofing we should make go/build more robust to new #cgo directives, though.)

[bcmills]

@corhere, for the moment I suggest that you file an issue against https://github.com/golang-fips/openssl to roll back the use of these new directives.

[gopherbot]

Change https://go.dev/cl/539235 mentions this issue: cmd/cgo: disable #cgo noescape/nocallback until Go 1.23