cmd/go: AWS CodeCommit repo URL incompatible with vgo major version import paths?

[computermouth]

What version of Go are you using (go version)?

$ vgo version
go version go1.10.3 linux/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

Yes. I was initially reporting the issue on go1.10, but the issue persists on latest.

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/byoung/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/byoung/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build461856878=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Created a private repository on AWS CodeCommit for a library.

$ cat go.mod 
module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2

$ cat main.go 
package return2

func Doit() int {
	return 2
}

Created a program to use it.

$ cat go.mod 
module git-codecommit.us-east-1.amazonaws.com/v1/repos/test

$ cat main.go 
package main

import "fmt"
import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2"

func main(){
	fmt.Println(return2.Doit())

}

Created an ssh redirect in my ~/.gitconfig

[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/"]
        insteadOf = https://git-codecommit.us-east-1.amazonaws.com/v1/repos/

Included this library

What did you expect to see?

Direct clones work just fine. Neither go get nor vgo build are able to import:

$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2
Cloning into 'return2'...
remote: Counting objects: 19, done.
Receiving objects: 100% (19/19), 1.67 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1/1), done.

$ go get git-codecommit.us-east-1.amazonaws.com/v1/repos/return2
package git-codecommit.us-east-1.amazonaws.com/v1/repos/return2: unrecognized import path "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2" (parse https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2?go-get=1: no go-import meta tags ())

$ vgo build
vgo: resolving import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2"
vgo: import "github.com/computermouth/runret" ->
	import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2": unknown module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2: no go-import tags

What did you see instead?

Everything works flawlessly with private repos on Gitlab:

$ git clone https://gitlab.com/computermouth/return2
Cloning into 'return2'...
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 16 (delta 1), reused 0 (delta 0)
Receiving objects: 100% (16/16), done.
Resolving deltas: 100% (1/1), done.

$ go get gitlab.com/computermouth/return2

$ vgo build
vgo: resolving import "gitlab.com/computermouth/return2"
vgo: finding gitlab.com/computermouth/return2 (latest)
vgo: adding gitlab.com/computermouth/return2 v0.0.0-20180607214308-f8814bf22457
vgo: finding gitlab.com/computermouth/return2 v0.0.0-20180607214308-f8814bf22457
vgo: downloading gitlab.com/computermouth/return2 v0.0.0-20180607214308-f8814bf22457

I have a sneaking suspicion that the cause is actually based in the way that AWS handles git authentication.

[url "ssh://[email protected]/"]
        insteadOf = https://gitlab.com/
[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/"]
        insteadOf = https://git-codecommit.us-east-1.amazonaws.com/v1/repos/

Note that the codecommit urls skip the git@. I presume that since I have yet to see any kind of restrictions in regards to repository names, and since my user/org is not accounted for in the repo URL, that the repository is presented to the user after some Amazon authentication middleware.

But I thought I'd post this here anyways.

Love vgo, it's the package management tool Go has always needed, thanks for all your work!

-- Ben

[ziyel]

The https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2?go-get=1 need to return meta tag to let the vgo identify the module.
You can refer to the blog:

Similar to current go get, vgo expects a page with a <meta> tag to help translate from a module name to the tree of files for that module. For example, to look up swtch.com/testmod, the vgo command fetches the usual page:

$ curl -sSL 'https://swtch.com/testmod?go-get=1'
<!DOCTYPE html>
<meta name="go-import" content="swtch.com/testmod mod https://storage.googleapis.com/gomodules/rsc">
Nothing to see here.
$ 

[myitcv]

@computermouth just as @ziyel mentioned, this is all happening pre-git being invoked.

Also take a look at https://golang.org/cmd/go/#hdr-Remote_import_paths.

I'll close this for now because I don't think there's a bug/issue per se, particularly not with vgo (the go-import meta-tag custom import paths is a general feature of the go tool).

[computermouth]

Interesting, I've never seen the vcs type used in the imports before.

$ vgo build
vgo: resolving import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git"
vgo: import "git-codecommit.us-east-1.amazonaws.com/v1/repos/testtest" ->
	import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git": unknown module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git: no go-import tags

$ go get git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git
(succeeds)

Having changed all the path references to use return2.git instead of return2. The go get yields a normal checkout and build, but vgo still reports the same error.

So I can only assume that vgo either has different behaviour that I'm still not understanding, or this is a bug, as this normally works with the go tool.

[myitcv]

@computermouth

I've never seen the vcs type used in the imports before

Just to confirm, what do you mean by "the vcs type"?

Quoting from the import paths link:

The vcs is one of "git", "hg", "svn", etc,

Back to the issue you're having:

Having changed all the path references to use return2.git instead of return2. The go get yields a normal checkout and build, but vgo still reports the same error.

Please can you include here the HTML <meta> tags that are returned?

[computermouth]

By vcs type I meant adding the .git or .svn.

After reading the linked pages, it sounded like adding the .git would be the solution to hitting a provider that doesn't provide the meta tags.

Following the example from @ziyel :

$ curl -sSL 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2?go-get=1'
<NotAuthorizedException>
  <Message>SPNEGO token required</Message>
</NotAuthorizedException>

As I mentioned before, CodeCommit has no organization/user namespacing in the URL. As such, if you were to open an account, and create a git repo called return2 on us-east-1, you would clone from the same URL. All repositories are private, and so AWS serves different content from that URL based on the credentials used when requesting the git contents. So it would make sense that this curl request does not give the desired response.

You mentioned that this happens before any git processing starts, so I was under the impression that adding the .git skips this request for the <meta> tags, and goes directly to the git processing, where git would request the files using my git helper cached credentials.

So it seemed to me that the following was kind of an oxymoron, where the .git suffix should prevent the search for <meta> tags.

unknown module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git: no go-import tags

Reproducible command summary :
(full repo path git-codecommit.us-east-1.amazonaws.com/v1/repos/return2 truncated to return2)

cmd	import	result
go get	return2	fails - no import tags
go get	return2.git	succeeds!
vgo get	return2	fails - no import tags
vgo get	return2.git	fails - no import tags

[myitcv]

After reading the linked pages, it sounded like adding the .git would be the solution to hitting a provider that doesn't provide the meta tags.

@computermouth I have to admit, I'd forgotten that go get does some parsing of the import path to try to determine the VCS to use (hence why it succeeds in your table) which is why the go-import meta tag is not necessary there.

I have no real experience with CodeCommit but I would make the following observations:

You are looking to use the URL git-codecommit.us-east-1.amazonaws.com/v1/repos/return2 as your import path. This URL has a /v1 element in it which I suspect will cause issues with respect to vgo version resolution (even though the /v1 has nothing to do with your module version, and everything to do with the API version of CodeCommit).

It's also unlikely that you'll be able to return custom <meta> tags via CodeCommit.

The better thing to do here therefore would be to instead map a domain you own to the the CodeCommit repo via a custom import.

Let's say you own the domain example.com and you want to map the import path "example/return2" to git-codecommit.us-east-1.amazonaws.com/v1/repos/return2. You would then simply need to ensure that:

curl -sSL 'https://example.com/return2?go-get=1'

returns the <meta> tag:

<meta name="go-import" content="example.com/return2 git https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2">

Your main.go would then be rewritten as:

package main

import "fmt"
import "example.com/return2"

func main(){
	fmt.Println(return2.Doit())
}

[computermouth]

Mmm, noted. I appreciate the suggested workaround.

Given what we know here, should this then be kept open as a bug in vgo? Or should I expect that this is simply the way that the behavior will remain?

If it's the latter, I wonder if this won't cause a more widespread failure when vgo is rolled into the go distribution. Assuming vgo get would replace the go get, wouldn't this cause a regression for folks using go get from CodeCommit?

[myitcv]

@rsc @bcmills - re-opening for a decision from you on this:

• do we want to introduce something similar to the go get import path parsing to determine remote repo type (e.g. .git)?
• if we do, then do we want to support import paths of the form git-codecommit.eu-west-1.amazonaws.com/v1/repos/myitcvtest.git, where the v1 is the AWS CodeCommit API version number?

[oiooj]

Duplicate of #25654

[myitcv]

@oiooj thanks, I hadn't spotted #25654.

I don't think this is quite a duplicate, however. Yes, the .git part is covered by #25654, but the second point in #25792 (comment) is definitely relevant and separate from #25654 if we do implement the import path parsing.

[computermouth]

if we do, then do we want to support import paths of the form git-codecommit.eu-west-1.amazonaws.com/v1/repos/myitcvtest.git, where the v1 is the AWS CodeCommit API version number?

Keep in mind here, that this would mean tracking the list of AWS regions. It looks like there are approximately 20 right now.

Adding the .git functionality seems to be the most reasonable option.

[myitcv]

Adding the .git functionality seems to be the most reasonable option.

That was my first point; if we do want to support the import path parsing, the second point was dealing with the fact the import path has v1 in it, and that v1 has nothing to do with our module version; it's the API version of CodeCommit.

[rsc]

The latest vgo does use the same logic now as "go get", including support for ".git" to mark the end of the git repo portion of an import path. I don't believe the /v1/ in the repo URL matters at all.

@computermouth could you go get -u golang.org/x/vgo and see if the return2.git form starts working for you?

Note that if you have a domain you like, you could serve the Go <meta> tags on that domain and point to the AWS git server, and then you won't be forced to have things like "us-east-1" in your import paths. (curl https://rsc.io/quote for example.)

[rsc]

Going to assume this was fixed by reusing the existing go get resolver.

[computermouth]

Hi @rsc, sorry about the late reply.

It's fixed with one small detail.

Package imports as git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git require that packages go.mod contains module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git.