Trying to get pip-compile manager working

[MrNaif2018]

Hi! Thank you for your quick implementation once again @rarkins!
I know that pip-compile doesn't yet support custom mapping scheme we've got in our repo, but decided to start setting it up
In our repo we have configure renovate PR pending, so I am changing config and initially testing the detecting of package files. Not all files were detected, so it might be even an issue with existing pip_requirements manager?
PR here: bitcart/bitcart#227

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "local>bitcartcc/renovate-config"
  ],
  "pip-compile": {
    "fileMatch": [
      "^requirements(\/.*)\\.txt$"
    ],
    "ignorePaths": [
      "requirements/deterministic/**"
    ]
  }
}

For now I have configured a fileMatch to fetch all .txt files in requirements folder, excluding actual output files. But instead of 10 files, it has parsed only 3. In the logs:

Matched 10 file(s) for manager pip-compile: requirements/base.txt, requirements/daemons/bch.txt, requirements/daemons/bsty.txt, requirements/daemons/btc.txt, requirements/daemons/gzro.txt, requirements/daemons/ltc.txt, requirements/dev.txt, requirements/production.txt, requirements/test.txt, requirements/web.txt
DEBUG: Found circleci package files
DEBUG: Found pip-compile package files
DEBUG: Found 4 package file(s)
INFO: Dependency extraction complete
{
  "baseBranch": "master",
  "stats": {
    "managers": {
      "circleci": {
        "fileCount": 1,
        "depCount": 3
      },
      "pip-compile": {
        "fileCount": 3,
        "depCount": 3
      }
    },
    "total": {
      "fileCount": 4,
      "depCount": 6
    }
  }
}

The only files detected are requirements/daemons/bch.txt, requirements/daemons/ltc.txt, requirements/test.txt

From investigating what all those files have in common, I see that all them have one dependency pinned with ranges, others unpinned. Is that an issue or something with my config? I have :preserveSemverRanges in my base renovate preset

If possible, we can discuss my attempts at making pip-compile working there, till it gets to a working state (:

[rarkins]

Some Renovate concepts to understand:

• The files you match for a manager are referred to as "package files"
• Other files which are derived from package files are referred to as lock files or more generally "artifacts"
• Renovate typically doesn't update artifact files directly but instead updates them whenever the package files need updating

For pip-compile, we are treating the input files as package files and the output files as artifacts. Hence it only does work when the input file needs updating, and if you had an input file with no constraints then it would never need updating.

Also historically Renovate has ignored dependencies with no constraints, however we could reconsider that for pip-compile.

Renovate also has a concept of "lock file maintenance" which essentially means "deleting the artifact file and regenerating it, even if the package file hasn't changed". This is perhaps what you were expecting with pip-compile. It wasn't initially enabled but I added it just now in #10408 and it will be live in the app sometime later this weekend.

FYI here is the example repo I tested with: https://github.com/renovate-tests/pip-compile-1

[rarkins]

I would prefer that we base our command off the comments like pip-compile --allow-unsafe --generate-hashes --output-file=requirements/deterministic/web.txt requirements/web.txt instead of the env variable CUSTOM_COMPILE_COMMAND so that behavior is more predictable. Would it be a reasonable restrictions to say that if you want to use Renovate with pip-compile then you need to avoid using the env variable, or are there use cases for where that would be an unreasonable requirement?

[craigds]

Yes it's probably reasonable to not support usage of that env var, if it makes the pip-compile args detection simpler.

[terencehonles]

Sorry if this is bumping an old thread or is hijacking it for a different issue. I'm also trying to get the pip-compile manager working. I raised an issue previously with pip-tools (jazzband/pip-tools#1365 (comment)) and replied more recently with its relation to renovate and the issue I was running into. Part of the underlying issue is that pip-compile doesn't have its own resolver and it can run into issues relatively easily.

I have pulled out the config into a test repository and this is the pull request I am having an issue with terencehonles/test-renovate#1

The problem is that renovate is only updating the requirements.in file and not the requirements.txt file. I optimistically tried the range strategy update-lockfile since there was some discussion above by @rarkins, but it looks like that has not been implemented (the logs said it was unknown and replace would be used instead). The reason this runs into issues is because as mentioned in jazzband/pip-tools#1365 (comment) pip-compile will use the requirements.txt to select the package version to use even though it's no longer selected in requirements.in. As mentioned in the comment you can use --upgrade or potentially another option, but I'm not sure if that's appropriate in this context because it may be upgrading too many packages.

[MrNaif2018]

@terencehonles pip-compile support is unfinished and currently isn't very usable. I plan to implement it properly soon, but I suggest to move to #2334.
Here's how I will probably implement it: #2334 (comment)

[terencehonles]

I'm already in that thread too 😅. I'm not sure where this discussion should be happening, but I'm describing a much simpler case than you're describing and I would expect what I have to have at least worked with the initial implementation. I understand at a high level why it does not, but in the short term I think the pip-compile manager could be configured to bump version constraints in both requirements.in and requirements.txt and I think that would allow it to work. I've not tested this, and it's quite possible the command would need to include --upgrade-package as you mention, but I'm not sure the argument's behavior for transitive dependencies.