Why is there a difference between sources by go get and dep ensure.

[kyoh86]

What version of dep are you using (dep version)?

$ dep version
dep:
 version     : devel
 build date  :
 git hash    :
 go version  : go1.9
 go compiler : gc
 platform    : darwin/amd64

$ cd $GOPATH/src/github.com/golang/dep && git describe --tags
v0.3.2-83-g13df5561

What dep command did you run?

dep ensure -add gopkg.in/src-d/go-git.v4

What did you expect to see?

$ go get gopkg.in/src-d/go-git.v4
$ cat $GOPATH/src/gopkg.in/src-d/go-git.v4/repository.go | head -n40
package git

import (
	"context"
	"errors"
	"fmt"
	stdioutil "io/ioutil"
	"os"
	"path/filepath"
	"strings"

	"gopkg.in/src-d/go-git.v4/config"
	"gopkg.in/src-d/go-git.v4/internal/revision"
	"gopkg.in/src-d/go-git.v4/plumbing"
	"gopkg.in/src-d/go-git.v4/plumbing/object"
	"gopkg.in/src-d/go-git.v4/plumbing/storer"
	"gopkg.in/src-d/go-git.v4/storage"
	"gopkg.in/src-d/go-git.v4/storage/filesystem"
	"gopkg.in/src-d/go-git.v4/utils/ioutil"

	"gopkg.in/src-d/go-billy.v3"
	"gopkg.in/src-d/go-billy.v3/osfs"
)

var (
	ErrInvalidReference        = errors.New("invalid reference, should be a tag or a branch")
	ErrRepositoryNotExists     = errors.New("repository not exists")
	ErrRepositoryAlreadyExists = errors.New("repository already exists")
	ErrRemoteNotFound          = errors.New("remote not found")
	ErrRemoteExists            = errors.New("remote already exists	")
	ErrWorktreeNotProvided     = errors.New("worktree should be provided")
	ErrIsBareRepository        = errors.New("worktree not available in a bare repository")
	ErrUnableToResolveCommit   = errors.New("unable to resolve commit")
)

// Repository represents a git repository
type Repository struct {
	Storer storage.Storer

	r  map[string]*Remote

What did you see instead?

$ cat vendor/gopkg.in/src-d/go-git.v4/repository.go | head -n40
package git

import (
	"errors"
	"fmt"

	"gopkg.in/src-d/go-git.v4/config"
	"gopkg.in/src-d/go-git.v4/plumbing"
	"gopkg.in/src-d/go-git.v4/plumbing/object"
	"gopkg.in/src-d/go-git.v4/plumbing/protocol/packp/sideband"
	"gopkg.in/src-d/go-git.v4/plumbing/storer"
	"gopkg.in/src-d/go-git.v4/storage/filesystem"
	"gopkg.in/src-d/go-git.v4/storage/memory"
	osfs "gopkg.in/src-d/go-git.v4/utils/fs/os"
)

var (
	ErrObjectNotFound     = errors.New("object not found")
	ErrInvalidReference   = errors.New("invalid reference, should be a tag or a branch")
	ErrRepositoryNonEmpty = errors.New("repository non empty")
	ErrRemoteNotFound     = errors.New("remote not found")
	ErrRemoteExists       = errors.New("remote already exists")
)

// Repository giturl string, auth common.AuthMethod repository struct
type Repository struct {
	r map[string]*Remote
	s Storer

	// Progress is where the human readable information sent by the server is
	// stored, if nil nothing is stored and the capability (if supported)
	// no-progress, is sent to the server to avoid send this information
	Progress sideband.Progress
}

// NewMemoryRepository creates a new repository, backed by a memory.Storage
func NewMemoryRepository() *Repository {
	r, _ := NewRepository(memory.NewStorage())
	return r
}

What's difference???

[kyoh86]

Sorry, it's issue of the go-git.

src-d/go-git#635

[kyoh86]

It's strange after all.

My Gopkg.toml:

[[constraint]]
  name = "gopkg.in/src-d/go-git.v4"
  version = "4.0.0-rc15"

When I run dep ensure -v:

Root project is "github.com/kyoh86/deject"
 8 transitively valid internal packages
 11 external packages imported from 9 projects
(0)   ✓ select (root)

: :

(8)     ? attempt gopkg.in/src-d/go-git.v4 with 1 pkgs; 16 versions to try
(8)         try gopkg.in/src-d/[email protected]
(8)     ✗   unable to deduce repository and source type for "srcd.works/go-billy.v1": unable to read metadata: go-import metadata not found
(8)         try gopkg.in/src-d/[email protected]
(9)     ✗   gopkg.in/src-d/go-git.v4 at v4.0.0-rc8 has problem subpkg(s):
(9)             gopkg.in/src-d/go-git.v4 has err (*pkgtree.NonCanonicalImportRoot); required by (root).
(8)         try gopkg.in/src-d/[email protected]
(8)     ✗   unable to deduce repository and source type for "srcd.works/go-billy.v1": unable to read metadata: go-import metadata not found
(8)         try gopkg.in/src-d/[email protected]
(8)     ✗   unable to deduce repository and source type for "srcd.works/go-billy.v1": unable to read metadata: go-import metadata not found
(8)         try gopkg.in/src-d/[email protected]
(8)     ✓ select gopkg.in/src-d/[email protected] w/29 pkgs

Why is the versions are traced in the sequence of rc9, rc8, rc7...?
I think that it should be traced in the sequence of rc15, rc14, rc13...

[darkowlzz]

@kyoh86 hi, thanks for opening an issue.

I'm trying to reproduce it but it seems to work as expected. Can you verify you have version = "4.0.0-rc15" in your Gopkg.toml file? I have the same and it picks the exact tag.

I can answer your other question about the sequence. Since 4.0.0-rc15 isn't a proper semver but a pre-release tag, it's a plain tag. And dep fails to understand how to sort them. Sorting by characters makes rc9 newer than rc15. So, you see that weird sorting order.

I'm not sure if there's a way to handle this because depending on the projects, their pre-release tag name formats would be different.

[imoverclocked]

When I view http://gopkg.in/src-d/go-git.v4 in a browser I see that the v4 tag points to v4 instead of v4.0.0-rc15 but dep somehow resolves the latter and not the former. Is it possible that the upstream authors of go-get should be doing something slightly different with their versioning?

[imoverclocked]

Also, I ran into this when I did a dep init and then my code started not compiling due to an older commit being chosen by dep for the go-git.v4 dependency.

[sdboyer]

but dep somehow resolves the latter and not the former.

this occurs because of a fundamental and irreconcilable mismatch between how gopkg.in and dep model the problem space: gopkg.in considers either branches or tags as candidates, and mostly sticks to semver. dep considers every possible branch and tag, but visits them in a particular order, and only tags can qualify as semver and therefore come up first; branches, even if named as strictly valid semver, are still just visited alphabetically.

so, concretely: dep chooses to visit the v4.0.0-rc15 tag before the v4 branch simply because, absent any further constraints, dep prefers tags to branches. this is a deliberate choice, in keeping with dep's general philosophy of making choices that prefer stability as a default. unfortunately, it does sometimes put us at odds with things like gopkg.in.

[kyoh86]

@darkowlzz

I tried to reproduce it in https://github.com/kyoh86/dep-tryout, and I can do it.

$ brew install dep
$ dep ensure -v
Root project is "github.com/kyoh86/dep-tryout"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
(0)   ✓ select (root)
(1)     ? attempt gopkg.in/src-d/go-git.v4 with 1 pkgs; 16 versions to try
(1)         try gopkg.in/src-d/[email protected]
(1)     ✗   unable to deduce repository and source type for "srcd.works/go-billy.v1": unable to read metadata: go-import metadata not found
(1)         try gopkg.in/src-d/[email protected]
(2)     ✗   gopkg.in/src-d/go-git.v4 at v4.0.0-rc8 has problem subpkg(s):
(2)             gopkg.in/src-d/go-git.v4 has err (*pkgtree.NonCanonicalImportRoot); required by (root).
(1)         try gopkg.in/src-d/[email protected]
(1)     ✗   unable to deduce repository and source type for "srcd.works/go-billy.v1": unable to read metadata: go-import metadata not found
(1)         try gopkg.in/src-d/[email protected]
(1)     ✗   unable to deduce repository and source type for "srcd.works/go-billy.v1": unable to read metadata: go-import metadata not found
(1)         try gopkg.in/src-d/[email protected]
(1)     ✓ select gopkg.in/src-d/[email protected] w/29 pkgs
(2)     ? attempt github.com/sergi/go-diff with 1 pkgs; 1 versions to try
(2)         try github.com/sergi/go-diff@master
(2)     ✓ select github.com/sergi/go-diff@master w/1 pkgs
(3)     ? attempt github.com/src-d/gcfg with 1 pkgs; 7 versions to try
(3)         try github.com/src-d/[email protected]
(3)     ✓ select github.com/src-d/[email protected] w/4 pkgs
(4)     ? attempt golang.org/x/crypto with 2 pkgs; 1 versions to try
(4)         try golang.org/x/crypto@master
(4)     ✓ select golang.org/x/crypto@master w/5 pkgs
(5)     ? attempt gopkg.in/warnings.v0 with 1 pkgs; 3 versions to try
(5)         try gopkg.in/[email protected]
(5)     ✓ select gopkg.in/[email protected] w/1 pkgs
  ✓ found solution with 40 packages from 5 projects

Solver wall times by segment:
  b-deduce-proj-root: 31.570373934s
         b-list-pkgs:  4.867769584s
     b-list-versions:  4.752623501s
              b-gmal:  2.463922195s
     b-source-exists:  1.226949716s
             satisfy:    5.420894ms
         select-atom:     1.62774ms
            new-atom:     218.887µs
         select-root:      67.468µs
               other:       33.46µs

  TOTAL: 44.889007379s

(1/5) Wrote github.com/src-d/[email protected]
(2/5) Wrote github.com/sergi/go-diff@master
(3/5) Wrote gopkg.in/[email protected]
(4/5) Wrote gopkg.in/src-d/[email protected]
(5/5) Wrote golang.org/x/crypto@master

If you want to do it, please clone the repository, and do dep ensure -v.

@sdboyer

visits them in a particular order

In semver.org, a following rule is described.

A larger set of pre-release fields has a higher precedence than a smaller set, if all of the preceding identifiers are equal. Example: 1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-alpha.beta < 1.0.0-beta < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0.

First, gopkg.in/src-d/go-git.v4 is mistaken a rule of the semver.

wrong: ..., v4.0.0-rc9, v4.0.0-rc10, v4.0.0-rc11, ... (rc and following numbers are not separated)
right: ..., v4.0.0-rc.9, v4.0.0-rc.10, v4.0.0-rc.11, ... (dots are put between rc and numbers)

Did I get that right?

Second, I know that dep cannot prefer rc9 to rc10. But can the dep prefer rc.9 to rc.10...?

dep prefers tags to branches

Branches are not for managing versions. dep SHOULD prefer tags to branches, I think so.

[darkowlzz]

I'm able to reproduce the issue with the provided repo. Idk what I did the last time that I couldn't reproduce it.

Until the pre-release tag format is fixed, you can use version ="=4.0.0-rc15" as a workaround. I've seen src-d guys preparing for a proper v4 release soon.

Second, I know that dep cannot prefer rc9 to rc10. But can the dep prefer rc.9 to rc.10...?

dep prefers rc.10 to rc.9 (rc.9 < rc.10), which is the expected order (newer greater). We have a test for it in version_test.go. To validate this behavior, I added 3 versions in that test (v3.0.1-alpha.1, v3.0.1-alpha.2, v3.0.1-alpha.3), not in correct order. Running SortForUpgrade() returned them in proper expected upgrade order v3.0.1-alpha.3 v3.0.1-alpha.2 v3.0.1-alpha.1.