Glide update does not update packages to the latest version unless git cache is cleared

[a13xb]

Your usage example states:

$ glide up                                # Update to newest versions of the package

However, it doesn't seem to do that at all. If I start with a clean repository with glide.yaml, where none of the packages are constrained, and a matching glide.lock, none of the dependencies are updated, even though upstream has new commits. Repository remains clean.

When (pre-0.12) Glide was still not stripping VCS files, I used to be able to go into venvor tree, checkout a specific version of a single dependency and run glide update, it would update the lock file to pin the dependency at the version I checked out. So I've been left with the impression that all glide update does is transfer the state of your vendor tree into glide.lock, that's it.

Obviously now vendor tree doesn't retain VCS information, I'm not sure what exactly glide update is supposed to do, and how to actually properly update dependencies to the latest version of the upstream.

[mattfarina]

I'm a little confused about something and I think I need some more information to understand what's going on.

The glide.yaml file holds the intent while the glide.lock file holds the specific revisions being pinned to. So, a version for a dependency in the glide.yaml file that's empty, tracks a branch, or has a version range causes Glide to go to the dependency source, pull down any updates, and store them locally. It does that automatically.

There was never the intent for someone to manually checkout a version inside the repo within the vendor/ folder. Glide tells the VCS (such as Git) to go out to the source and fetch updates.

This happens via the github.com/Masterminds/vcs package. For example, with Git it will drop down to git and use commands like git fetch and git pull expecting them to go out.

Is that not happening for you?

[a13xb]

Is that not happening for you?

When I run glide update, yes, it pulls the latest from the upstream VCS into cache, but the versions in the vendor/ workspace are not updated.

Let me clarify again: my glide.yaml is a plain list of dependencies, none of them specify a branch, a tag, a commit, or range of versions, none whatsoever. My expectation was that this will result in glide update always updating the glide.lock to the HEAD of master.

This doesn't seem to be happening. After the update of local cache, my glide.lock does not change, even though there are new commits in some upstream repos.

[INFO]  Replacing existing vendor dependencies
[INFO]  Versions did not change. Skipping glide.lock update.

There was never the intent for someone to manually checkout a version inside the repo within the vendor/ folder. Glide tells the VCS (such as Git) to go out to the source and fetch updates.

Yes, that was my misguided attempt to get a pinned version of a package updated in glide.lock because glide update would not do it otherwise.

[mattfarina]

@a13xb it sounds like there is a problem and one I'm not experiencing.

Can you provide an example project or glide.yaml file? And, can you share your OS/version and Go environment (go env)?

[a13xb]

OSX: 10.11.6
Glide: 0.12.0 (via Homebrew)
Go: 1.7
Go environment:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/alex/workspace/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.7/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.7/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/73/jds74fnd03q8qkfyx_8zx8400000gn/T/go-build423096790=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

Sample project:

glide.yaml

package: github.com/a13xb/glidetest
import:
- package: github.com/gorilla/mux

glide.lock

hash: be6ffa009beb31b542f3a93c368fa8213aaf2fed7fc7e10914021349fe65ce9f
updated: 2016-08-31T23:38:13.074624366+10:00
imports:
- name: github.com/gorilla/mux
  version: 9fa818a44c2bf1396a17f9d5a3c0f6dd39d2ff8e
testImports: []

Commit 9fa818a in the lock file is from June, actual HEAD of master is 0b13a92.

Running glide update:

[INFO]  Downloading dependencies. Please wait...
[INFO]  --> Fetching updates for github.com/gorilla/mux.
[INFO]  Resolving imports
[INFO]  Downloading dependencies. Please wait...
[INFO]  Setting references for remaining imports
[INFO]  Exporting resolved dependencies...
[INFO]  --> Exporting github.com/gorilla/mux
[INFO]  Replacing existing vendor dependencies
[INFO]  Versions did not change. Skipping glide.lock update.
[INFO]  Project relies on 1 dependencies.

[mattfarina]

Quick question, do you have github.com/gorilla/mux imported in your code? Glide code scans and follows imports found in the code to find the packages to use. If you have this in your glide.yaml file but didn't use it in your code it's dependencies won't be followed.

I setup an example with a file:

package main

import (
    "fmt"

    _ "github.com/gorilla/mux"
)

func main() {
    fmt.Println("test")
}

It fetched the dependencies.

Does this fix your problem?

[a13xb]

I've added this sample Go file and it adds the transitive dependency, so it completes the example, but the problem is still there.

I think it has something to do with cache.
How to replicate:

1. Starting with the empty cache, do a glide update, the dependencies are properly updated, git repos in cache end up checked out at master.
2. Discard changes to glide.lock file.
3. Do a glide install, old versions from lock file get installed, cache repos are checked out at the versions in the lock file (detached HEAD).
4. Do a glide update, update won't happen and the cache stays pointing at the commit in the lock file.
5. Remove the cache, glide update will update correctly.

[karamani]

+1
I had the same problem.

[oschwald]

I had a similar issue. Removing the cache fixed it for me.

[vuleetu]

Same here. Seems like Glide does not switch to the latest version before export to vendor.

[KatelynHaworth]

👍 This also seems to affect me.

Go Version: go version go1.6.2 linux/amd64
Glide Version: glide version v0.12.3
Go Environment:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/liam/.golib"
GORACE=""
GOROOT="/usr/lib/go-1.6"
GOTOOLDIR="/usr/lib/go-1.6/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT="1"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

[jmazzitelli]

I confirm this is happening to me too. I was wondering if I was doing something wrong, until I read this issue :) I even removed the entry of my dep from the glide.lock file (my dep is on github), ran update again, saw that the .lock file put the dep back, but vendor/ is still the old version and the entry in the .lock file is still the old hash. I want to pull the lastest from master branch, but it isn't setting my dep to the latest hash - the lock file is showing the git hash from the first time I pulled the dep via glide. Workaround: "rm -rf ~/.glide/cache/src/my-dep-directory" then run glide update.

[adangert]

same problem I was having "rm -rf ~/.glide/cache/" worked for me, glide.lock was not updating to the latest commit hash regardless of doing a glide update

[sdboyer]

i'm pretty sure the answer is "no," but just to be sure: have any of you who are experiencing this bug done any direct manipulation of the ~/.glide/cache dir?

also, if anyone is able to create some exact steps to reproduce, it would be tremendously helpful in solving this problem.

[jmazzitelli]

@sdboyer the answer for me is "no" - in fact, I didn't even know this cache existed let alone where it was until I researched this issue and found someone said "clear you cache" and I had to go hunt for what this cache even was :)

[jmazzitelli]

@sdboyer

if anyone is able to create some exact steps to reproduce,
it would be tremendously helpful in solving this problem.

I just ran through a set of replication steps for you to try.

FYI, my versions are as follows:
$ glide -v ; git version ; go version
glide version v0.12.3
git version 2.5.5
go version go1.7.1 linux/amd64

1: Clone this repo and make it your GOPATH: https://github.com/hawkular/hawkular-openshift-agent

2: Make sure glide is in your path obviously (to do this, I run "make install_glide" from the root GOPATH directory - there is a Makefile in there - and glide goes in my GOPATH/bin that I then run)

3: cd src/github.com/hawkular/hawkular-openshift-agent/ (you will find the glide yaml and lock files here)

4: Run "glide update" - notice this:

--> Fetching github.com/hawkular/hawkular-client-go.

5: See that you have the latest code for that dependency (hawkular-client-go) in vendor directory by grepping for "Username" - this was a new field added in the latest PR merge and is now in the HEAD of master branch:

$ grep Username vendor/github.com/hawkular/hawkular-client-go/metrics/types.go 
    Username    string

6: edit the glide.yaml and add a "ref" so that dependency gets locked on a new hash (this hash is the one before the latest PR merge and did NOT have that "Username string" field) - so add the two lines "vcs" and "ref" below under the hawkular-client-go package dependency:

- package: github.com/hawkular/hawkular-client-go
  vcs: git
  ref: d3a0e3ccb068e354fc61c5811a24414792e6159b

7: Run "glide update" - notice this:

--> Setting version for github.com/hawkular/hawkular-client-go to d3a0e3ccb068e354fc61c5811a24414792e6159b.

8: Run the grep again, and notice the Username is missing (this is correct, it means glide updated the dependency source under vendor properly)

$ grep Username vendor/github.com/hawkular/hawkular-client-go/metrics/types.go 
(...no output here - which is what you want...)

9: Now go back to glide.yaml, remove that "vcs" and "ref" that you added in step 6 thus bringing back your dependency to the master head.

10: Run "glide update" - notice this:

--> Fetching github.com/hawkular/hawkular-client-go.

11: Now, you are back to where we were (or we should be) - the latest code of master HEAD that has the new Username field in the dependency code. But run that grep command and see that it is missing under vendor directory:

$ grep Username vendor/github.com/hawkular/hawkular-client-go/metrics/types.go 
(...no output here - which is NOT what you want... we should be back on master and this Username field should exist)

12: To confirm the cache is now bad, go to it and look at what is checked out of git:

$ cd ~/.glide/cache/src/https-github.com-hawkular-hawkular-client-go
$ git rev-parse HEAD
d3a0e3ccb068e354fc61c5811a24414792e6159b

That is NOT the git hash of the latest master. That's the git hash we put in in step 6. But we want master, which currently is:

$ git rev-parse origin/master
4f5da7e951592445b686f95a0acdb0f860095d70

So when I removed the ref: from the yaml file and I "glide update", it never pulled the HEAD from master - the cache stayed on the old hash that I pulled before and thus my vendor/ never updated.

Hope this helps.

[sdboyer]

@jmazzitelli 👼 👼 ✨ 💖 🎉

i won't have time to actually run these through until later today or tomorrow, but these kinds of reproducibility procedures are like manna from heaven.

[jmazzitelli]

@sdboyer - any word on this? Did my replication procedures yield any clues as to the problem and potential fix?

[sdboyer]

sorry, i've had a ridiculous couple weeks, and just totally forgot about this 😞 i'll look today

[jmazzitelli]

@sdboyer those replicate steps might be slightly different - I just committed a change to that repo that changes the project structure to be more like a "normal" golang project. Just follow the README.adoc to install it (it should now be more as you would expect anyway). Its not really different, I just move the src/ files up to the top directory rather than committing the files under the src/github.com.... directory structure.

[sdboyer]

4: Run "glide update"

This is hitting another, i guess different issue:

[ERROR] Error scanning k8s.io/client-go/1.4/kubernetes: open /Users/sdboyer/.glide/cache/src/https-k8s.io-client-go/1.4/kubernetes: no such file or directory

it went away when I glide install'd, after which i could glide update successfully. I'll have to investigate this later, too...

12: To confirm the cache is now bad, go to it and look at what is checked out of git:

Yep, I figured the issue would be something like this. The git commands glide is issuing to get information from the repo aren't working correctly because they're using porcelain commands that have subtle dependencies on local repo state. Specifically, git pull doesn't do anything unless you have the right local branch checked out, and it's set up to track the right remote branch.

I need to track down where the actual calls are in the code to fix this up, but it shouldn't be too terrible to deal with.

Also, just a note for the future - the cache having a particular version checked out in the working directory isn't necessarily an indication that it's "bad". The technique we use to quickly get files onto disk under vendor/ is not copying what's checked out, but rather doing some under-the-hood git goodness to perform that process quickly. Right now, it's probably an accurate indicator, but in the future the working directory may be something totally different, but you'll still have the right thing in your vendor/.

[jmazzitelli]

Also, just a note for the future - the cache having a particular version checked out in
the working directory isn't necessarily an indication that it's "bad"

This is why I have step 11 in my replication procedure. I actually do look in /vendor to see that the code is "bad" there.

[stevenroose]

I have the same issue. I track a package as a fork I made and when I push updates to the fork, glide update does not make the lock file update to the latest version of the fork. Also worked after cleaning the cache.

[sdboyer]

So, the fix over in #703 should address this. The only note to make is that you have to specify the branch you want to follow in glide.yaml - it won't move properly if you don't. (Yeah, I'm not happy about that either).

I've verified this against @jmazzitelli's replication procedure, but if someone else could try with that PR, that'd be awesome.

[jmazzitelli]

@sdboyer I just tried it using your PR branch and it still failed. But maybe I'm not doing it right. What do you mean "you have to specify the branch you want to follow in glide.yaml". I was using:

vcs: git
ref: d3a0e3ccb068e354fc61c5811a24414792e6159b

So I'm giving a hash reference, not an actual branch name. Is that not correct anymore?

And what about tags? Can you use tag names rather than branch names?

[metral]

Fwiw I worked around this issue by reverting back to v0.11.1 https://github.com/Masterminds/glide/releases/tag/v0.11.1

[sjmc7]

I also see this with glide v0.13; i've been in the habit of manually deleting the cached checkout of a dependency before updating.

[jmcfarlane]

The behavior suggests that glide is using the latest tag if no version is specified in glide.yaml. My observation is that most folks either want a tag or a branch (likely master) specified as the version. If this observation happens to be true, I personally recommend glide use master as the version when it's not specified, versus the latest tag.

But to be clear, I might be completely confused here :)

[XANi]

@jmcfarlane I had to use origin/master to actually force it to update on glide update. Leaving it on master made it update once to latest (when cache was empty) and then never do it again

[G-Harmon]

I think I hit this as well.

glide update; go build ./... ==> build failed because k8s.io/api was not update
rm -rf ~/.glide/cache/; glide update; go build ./... ==> build succeeded

[mattfarina]

@G-Harmon Can you run glide --debug update. I'm curious what happened?

[jaekwon]

If a glide dependency ever gets --force updated, VcsVersion() will silently fail upon Update() and UpdateVersion(ver) which merges anything that already exists for that local branch. Update() definitely shouldn't call git pull, and UpdateVersion() should probably do more than just git checkout. This causes the dependency to get pinned to the wrong version, after an unwanted merge. (assuming that glide.yaml has a dependency set to a branch).

See #961

[G-Harmon]

Hi @mattfarina, are you asking for debug output on the faulty update or the successful one? I assume the faulty one, but I'm not quite sure what it takes to get into that state. I normally only run glide update once in awhile.

[mattfarina]

Faulty one

[wellsjo]

Any updates here? This is still an issue...

[G-Harmon]

Hey @wellsjo, I haven't had a time to repro and provide debug data. Can you?

[wellsjo]

@G-Harmon I'll see if I can find time in the next couple days to reproduce this in a digestible way.

I'm baffled though, this is a glaringly obvious issue that many people are facing, and it breaks one of the most important features of a dependency management system. Many projects' dependencies rely on the master branch...and they don't update. This has been an open issue since August 2016, which makes me inclined to drop glide. It's somewhat reassuring that GCP is using it, but this is clearly buggy/unfinished software.

[JelteF]

If possible I would definitely try switching to dep: https://github.com/golang/dep/ It's the "official" dependency tool and has much less of these bug bugs in my opinion. The glide README also recommends it actually.

…

On Thu, Feb 1, 2018, 03:24 Wells Johnston ***@***.***> wrote: @G-Harmon <https://github.com/g-harmon> I'll see if I can find time in the next couple days to reproduce this in a digestible way. I'm baffled though, this is a glaringly obvious issue that many people are facing (and easily reproduced), and it breaks one of the most important features of a dependency management system. Many projects dependencies rely on the master branch...and they don't update. This has been an open issue since *August 2016*, which makes me inclined to drop glide. It's somewhat reassuring that GCP is using it, but this is clearly buggy/unfinished software. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#592 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG8JkXUD0dXGVT0Y2t06UQb2dvq7DD-ks5tQSBogaJpZM4JxUEK> .

[wellsjo]

@JelteF dep didn't work for me right out of the box because of this. It just hangs without any output on dep init...

[ashubalike]

I was facing the same issue that 'glide up' wasn't updating my vendor folder,and project was referring to an older version of the dependency which was already present. I see from this thread that clearing glide cache also did not help some people, same for me too.
Figured out a workaround till this issue is fixed:

• run 'glide cc' to clear cache
• update the particular dependency in glide.yaml with the exact commit version I wanted(mostly the last commit id)
  • • package : xxx-xxx-xxx(your dependency package)
      version: required_commit_id
• run 'glide up'

This refreshed my vendor folder with required commit or created a dependency if it dint exist.

[ghost]

@ashubalike that worked for me!