Slow compile

[Rizhiy]

Describe the issue briefly here.

Environment Versions

1. OS Type: Ubuntu 19.04
2. Python version: Python 3.7.3
3. pip version: pip 19.0.3
4. pip-tools version: pip-compile, version 3.7.0

Steps to replicate

I have a new environment with the following requirements.in:

Cython
torch
torchvision

when I run pip-compile it just hangs.
Here is the output of pip-compile -v:

Using indexes:
  https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  Cython
  torch
  torchvision

Finding the best candidates:
  found candidate cython==0.29.10 (constraint was <any>)
  found candidate torch==1.1.0 (constraint was <any>)
  found candidate torchvision==0.3.0 (constraint was <any>)

Finding secondary dependencies:
  torchvision==0.3.0        requires numpy, pillow>=4.1.1, six, torch>=1.1.0
  torch==1.1.0 not in cache, need to check index

Expected result

Compiles normally

Actual result

Hangs on checking index?

[atugushev]

Hello @Rizhiy,

Looks like torch package has large size. Would be nice to see a progress-bar though.

[Rizhiy]

The thing is, I haven't noticed that problem before, even though I was using torch with pip-compile for more than 6 months.

[Rizhiy]

And on my other machine using python3.5 this problem doesn't seem to occur, although it does have faster CPU

[atugushev]

And on my other machine using python3.5 this problem doesn't seem to occur, although it does have faster CPU

Let me guess, was it macosx?

[Rizhiy]

Nope, both Linux

[atugushev]

Nope, both Linux

Strange, because macox wheel weight 88.9 MB, but linux wheel - 676.9 MB.

[Rizhiy]

What does it do when checking the package? Does it download it? Or does it hash the contents?

[atugushev]

What does it do when checking the package? Does it download it? Or does it hash the contents?

It does download it (the same as pip install torch, note the speed of checking index and install should be the same). It will hash if you pass --generate-hashes to the pip-compile.

[Rizhiy]

Normally when I run pip-compile, I have already installed the package with pip. Is it possible to add functionality, such that pip-compile uses pip cached version of the package instead of downloading it again?

[atugushev]

Is it possible to add functionality, such that pip-compile uses pip cached version of the package instead of downloading it again?

Actually it works this way. The pip-compile uses pip API to download package (to check dependencies), so it should use cached wheel for a package if it had been installed before. See:

$ # clear cache
$ rm -fr ~/Library/Caches/pip*

$ # make sure torch is not installed
$ pip uninstall torch
WARNING: Skipping torch as it is not installed.

$ # pip-compile torch
$ echo torch | pip-compile - -qo-

$ # See that pip uses cached torch
$ pip install torch
Collecting torch
  Using cached https://files.pythonhosted.org/packages/c9/e2/40f7f017437139e5dc076ea5ad4207da61250cc078bceef7d6f333c1d05c/torch-1.1.0-cp37-none-macosx_10_7_x86_64.whl

[Rizhiy]

TBH, I don't see why it would take so long if it is already downloaded.

[atugushev]

Could you make sure it has been cached by pip? Try to install torch on a temporary virtualenv, if it's cached pip should install it from cache. For example:

$ virtualenv /tmp/torch
$ . /tmp/torch/bin/activate
$ pip install torch
...
# check the output
...

[Rizhiy]

I don't use virtualenv, I use conda, can that be a problem?
I usually do:

conda create -n name python=3
conda activate name
pip install -r requirements.in
pip-compile

[atugushev]

Should work with conda.

[atugushev]

I'm 99% sure that's the problem when pip-tools downloads large wheels. We need a progress-bar for that process.

[atugushev]

@Rizhiy

Could you try your requirements.in with this version of pip-tools (wip #949)? Instructions:

$ pip install git+https://github.com/atugushev/pip-tools.git@downloading-progress-bar#egg=pip-tools

$ pip-compile -v