Dep init takes forever

[mageddo]

What version of dep are you using (dep version)?

dep:
 version     : devel
 build date  : 
 git hash    : 
 go version  : go1.9
 go compiler : gc
 platform    : linux/amd64

What dep command did you run?

dep -v init

What did you expect to see?

The project being created

What did you see instead?

Getting direct dependencies...
Checked 72 directories for packages.
Found 12 direct dependencies.

Then it never get completed. I've waited for 6 minutes the type CTRL+C then get the follow output

Signal received: waiting for 1 ops to complete...
Root project is "bk-api"
 53 transitively valid internal packages
 43 external packages imported from 12 projects
(0)   ✓ select (root)
(1)	  ← no more versions of github.com/mageddo/bk-api to try; begin backtrack
  ✗ solving failed

Solver wall times by segment:
     b-list-versions: 6m25.565681041s
         select-root:      3.667148ms
            new-atom:        32.766µs
  b-deduce-proj-root:         8.784µs
               other:         6.533µs
     b-source-exists:           666ns

  TOTAL: 6m25.569396938s

this SourceMgr has been released, its methods can no longer be called

[sdboyer]

hi, thanks for the issue! sorry, we know this is not a great first-time experience. unfortunately, dep often has a bunch of up-front work to do cloning repositories for the first time.

when you look in $GOPATH/pkg/dep/sources/, do you at least see some of the dependencies of your project there? if so, just re-running dep init may get you over the hump.

@carolynvs do you think, at least as a stopgap, we might reinstitute the feedback we used to have in dep init where we inform the user that syncs are happening? perhaps this could be ordered separately from the discovery/importer feedback.

[carolynvs]

Yes, I'd like to add more feedback so that we don't sit for a long time without any indication of activity. But I don't recall which previous feedback you are referring to? Do you remember what the message was like at all? Or where we were printing it? I'm happy to dig it up from there.

[mageddo]

Thanks for answer, @sdboyer Yes I can see some dependencies at $GOPATH/pkg/dep/sources/. I think that I found a way of reproduce this problem in a similar way:

$ mkdir -p ~/golang/src/mypjt && \
export GOPATH=~/golang && \
cd $GOPATH/src/mypjt && \
echo 'package main
import (
_ "github.com/mageddo/not-found-project" // this project do not exists
_ "github.com/mageddo/go-logging" // this project exists
)' > app.go

Then run

$ dep init -v
# (no output)

The project will never get completed. When type CTRL+C

^CSignal received: waiting for 1 ops to complete...
Root project is "mypjt"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
(0)   ✓ select (root)
(1)	  ← no more versions of github.com/mageddo/go-logging to try; begin backtrack
  ✗ solving failed

Solver wall times by segment:
     b-list-versions: 28.39937683s
         select-root:      88.99µs
            new-atom:     15.511µs
               other:       3.53µs
  b-deduce-proj-root:      1.685µs
     b-source-exists:        312ns

  TOTAL: 28.399486858s

this SourceMgr has been released, its methods can no longer be called

Dep sources folder

ls $GOPATH/pkg/dep/sources/
https---github.com-mageddo-go--logging

I hope it help you, maybe it can be a bug

[sdboyer]

@mageddo thanks for the additional info. it's surprising that pointing to a nonexistent repository is causing a hang - what it's no doubt doing there is trying to reach out to that URL using a progression of four protocols - https, ssh, git, then http - and getting 404s/the equivalent back from github. This happens pretty quickly on my local - about 1.5s for all four, executed in serial. It seems, though, that this is taking much longer on your system. perhaps this has something to do with your local environment, or your network topology vis-a-vis github?

if you could report the output of the following timeed commands, that might tell us if that is indeed where the slowdowns are:

time git ls-remote https://github.com/mageddo/not-found-project
time git ls-remote ssh://[email protected]/mageddo/not-found-project
time git ls-remote git://github.com/mageddo/not-found-project
time git ls-remote http://github.com/mageddo/not-found-project

@carolynvs it's super old school - #194. my big concern there is that we absolutely want to do that in parallel, but need to keep the output from it cleanly separated from the other output we're generating. actually, the more i think about this, the better an idea it seems, because we can explicitly cap the concurrency to say, 4, and i suspect the heavily parallel network activity is part of what makes this so costly.

[mageddo]

Okay, After read you answer I got the problem, ssh port is blocked in my network

$ time git ls-remote https://github.com/mageddo/not-found-project
remote: Repository not found.
fatal: repository 'https://github.com/mageddo/not-found-project/' not found

real	0m2.023s
user	0m0.000s
sys	0m0.004s


$ time git ls-remote ssh://[email protected]/mageddo/not-found-project
ssh: connect to host github.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

real	4m14.445s
user	0m0.000s
sys	0m0.001s

[sdboyer]

iiiinteresting. also, unfortunately, i'm not sure we can do anything about that 😢 . at most, better feedback.

[mageddo]

Is not a problem for now because I just fixed the not found project URL, DEP uses HTTPS by default then everything works great, thanks for the help. Maybe reduce the connection timeout will increase the use experience?

[sdboyer]

unfortunately we can't really directly control that, as we're invoking a subprocess. but we can terminate the subprocess after some period of time. that's currently causing a lot of headaches, so i'm generally trying to find alternatives to timeouts where possible.

will think on it, though. in any case, closing, as the base problem seems to be resolved.

[carolynvs]

@sdboyer I created #1144 to track adding back the prefetch and logging during init. If I captured the ask incorrectly, feel free to just fix it directly. 😀

[ghost]

I'm having this issue, except I don't see anything in $GOPATH/pkg/dep/sources

[Wulfheart]

Maybe a small progressbar?

[mrdulin]

Too slow, time out finally

☁  golang [master] dep init -v
Getting direct dependencies...
Checked 40 directories for packages.
Found 5 direct dependencies.
Root project is "github.com/mrdulin/golang"
 28 transitively valid internal packages
 6 external packages imported from 5 projects
(0)   ✓ select (root)
(1)       ← no more versions of cloud.google.com/go to try; begin backtrack
  ✗ solving failed

Solver wall times by segment:
     b-list-versions: 1m20.768848296s
     b-source-exists: 1m15.301073786s
         select-root:       629.021µs
               other:        75.587µs
            new-atom:        10.296µs
  b-deduce-proj-root:         9.862µs

  TOTAL: 2m36.070646848s

init failed: unable to solve the dependency graph: Solving failure: 
        (1) failed to list versions for https://code.googlesource.com/gocloud: fatal: unable to access 'https://code.googlesource.com/gocloud/': Failed to connect to code.googlesource.com port 443: Operation timed out
: exit status 128