cmd/go: go get -d -u on AWS code commit repository fails

[antgustech]

What version of Go are you using (go version)?

$ go version 12.5

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"

GOBIN=""

GOCACHE="/home/CORPUSERS/xxx/.cache/go-build"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOOS="linux"

GOPATH="/home/CORPUSERS/xxx/projects/go"

GOPROXY=""

GORACE=""

GOROOT="/usr/local/go"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"

GCCGO="gccgo"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=

What did you do?

Trying to run go get on a repository hosted on code commit:

go get -d -u git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo

What did you expect to see?

The repository cloned and dependencies installed.

What did you see instead?

package git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo: unrecognized import path "git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo" (parse https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo?go-get=1: no go-import meta tags ())

My machine is properly setup with SSH keys so git clone ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo works fine. Also take note that my end goal is to the the build pipeline on AWS Code build run the dockerfile in the end but as of now it gives the same result as running it locally.

[AlexRouSg]

Try go get -d -u git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo.git

See https://golang.org/cmd/go/#hdr-Remote_import_paths

[antgustech]

Try go get -d -u git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo.git

See https://golang.org/cmd/go/#hdr-Remote_import_paths

Adding .git to the end gave me another problem, so at least some progress. :)

package git-codecommit.eu-west-1.amazonaws.com/v1/repos/battery-historian.git: cannot download, git://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo uses insecure protocol

Isn't go get using HTTPS by default? How can that be insercure? If I add either SSH or HTTPS as the protocol, it fails sating the import path is malformed. I've seen solutions on how to specify using SSH for github but not understood how to do that for Code commit. I've tried adding

[url "[email protected]:"]
	insteadOf = https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/

To my .gitconfig file but it did not help.

[thepudds]

Perhaps not directly related, but #25792 is older AWS CodeCommit vgo bug, and includes final comment:

It's fixed with one small detail.

Package imports as git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git require that packages go.mod contains module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git.

@baxtex Probably worth at least skimming that bug.

[bcmills]

@baxtex, the go command probes .git paths using all known protocol schemes for the VCS that meet security requirements (see here).

Can you git clone or git ls-remote the https URL for the path?

[antgustech]

After some more research, I discovered that I probably don't need to do go get/git clone at all. As I'm running this on AWS code build, it already has access to the repository. So I realized I could use the COPY command instead. Here is my updated dockerfile:

#Use latest golang version 
FROM golang:latest 
 
MAINTAINER Me 
 
#Install Java 
RUN apt-get -y update && apt-get install -y openjdk-8-jre-headless 

#Get the battery historian code 
WORKDIR /go/src/codecommit/battery-historian 
COPY . /go/src/codecommit/battery-historian 
 
RUN go get ./... 
#RUN go get -d -u 
RUN go run setup.go 
 
#Open default port 
EXPOSE 8888 
 
#Run the start file for battery historian in a container 
CMD go run cmd/battery-historian/battery-historian.go --port 8888 --name my-battery-historian 

As you can see, I use copy to copy in my own battery historian repository instead of using go get. Do I have to do something more? Because when I study my logs, it looks like it cannot find all my dependencies:

Step 7/9 : RUN go run setup.go 
 ---> Running in f1735a0eb865 
 
Generating JS runfiles... 
Couldn't generate runfile: failed to run command "python /go/src/codecommit/battery-historian/third_party/closure-library/closure/bin/build/depswriter.py --root=/go/src/codecommit/battery-historian/third_party/closure-library/closure/goog --root_with_prefix=js ../../../../js": 
  exit status 2 
  python: can't open file '/go/src/codecommit/battery-historian/third_party/closure-library/closure/bin/build/depswriter.py': [Errno 2] No such file or directory 

For reference, the repo I got on Code commit is a clone of googles battery historian: https://github.com/google/battery-historian

[agnivade]

@baxtex - Have you manually checked that third_party/closure-library/closure/bin/build/depswriter.py exists ?

Could you use rebuild=true and run docker build again ?

And in any case, this does not seem to be a Go issue anymore as you are just copying files.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[GimignanoF]

I don't know if you are still having issues, but i've managed to solve this by running this command
git config --global url."ssh://git@git-codecommit.${AWS_REGION}.amazonaws.com".insteadOf "ssh://git-codecommit.${AWS_REGION}.amazonaws.com"
Wich added those lines in the global .gitconfig
[url "ssh://git@git-codecommit.${AWS_REGION}.amazonaws.com"] insteadOf = ssh://git-codecommit.${AWS_REGION}.amazonaws.com

And so now i'm able to get Go packages from AWS CodeCommit using this command
go get git-codecommit.${AWS_REGION}.amazonaws.com/v1/repos/${REPOSITORY_NAME}.git

The ${AWS_REGION} placeholder has to be replaced with the actual region the repository is, for example in my case is EUWest1, so in the go get command i replaced it with "eu-west-1".
The ${REPOSITORY_NAME} placeholder is, well you guessed it, the name of the repository your GO package is in. For example, if your package is in a repository called "TestPackage" you must replace that placeholder with "TestPackage" as well (i don't know if it works having all lowercase too).

TL;DR

• Run this command
  git config --global url."ssh://git@git-codecommit.${AWS_REGION}.amazonaws.com".insteadOf "ssh://git-codecommit.${AWS_REGION}.amazonaws.com"
  replacing ${AWS_REGION} with the Region the CodeCommit repository is in (for example eu-west-1)
• Get the package via go get using this command
  go get git-codecommit.${AWS_REGION}.amazonaws.com/v1/repos/${REPOSITORY_NAME}.git
  where the ${AWS_REGION} placeholder is the same as before and the ${REPOSITORY_NAME} is the name of the repository (in your case should be myRepo). Don't forget to add the .git at the end, otherwise it won't work