pip-compile fails AttributeError: 'NoneType' object has no attribute 'specifier'

[fj-dt]

Hello everyone,

My project is depending on a lot of git repositories. These packages are also dependent on other git repositories.
When I generate one requirements.txt then I do pip-sync it works fine. But when I generate one requirements.txt and one requirements-dev.txt, when I do pip-sync requirements.txt requirements-dev.txt then I get this error:

Traceback (most recent call last):
  File "/Users/fj/PycharmProjects/my-project/.venv/bin/pip-sync", line 8, in <module>
    sys.exit(cli())
  File "/Users/fj/PycharmProjects/my-project/.venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/fj/PycharmProjects/my-project/.venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/fj/PycharmProjects/my-project/.venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/fj/PycharmProjects/my-project/.venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/fj/PycharmProjects/my-project/.venv/lib/python3.7/site-packages/piptools/scripts/sync.py", line 123, in cli
    merged_requirements = sync.merge(requirements, ignore_conflicts=force)
  File "/Users/fj/PycharmProjects/my-project/.venv/lib/python3.7/site-packages/piptools/sync.py", line 116, in merge
    if ireq.specifier != existing_ireq.specifier:
  File "/Users/fj/PycharmProjects/my-project/.venv/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 263, in specifier
    return self.req.specifier
AttributeError: 'NoneType' object has no attribute 'specifier'

requirements.txt contains:

git+ssh://[email protected]/private-bucket/[email protected]

and requirements-dev.txt contains the same package which is a sub-dependency of another git repository.

So I tried to debug it, by adding a print into piptools/sync.py line 113:

                    # NOTE: We check equality here since we can assume that the
                    # requirements are all pinned
                    print(ireq, " - ", existing_ireq, " - ", ireq.req, " - ", existing_ireq.req, " - ", key)
                    if ireq.specifier != existing_ireq.specifier:
                        raise IncompatibleRequirements(ireq, existing_ireq)

And this is the output:

ireq: git+ssh://****@bitbucket.org/private-bucket/[email protected] (from -r requirements-dev.txt (line 33))  
existing_ireq: git+ssh://****@bitbucket.org/private-bucket/[email protected] (from -r requirements.txt (line 25))  
ireq.req: None  
existing_ireq.req: None  
key: git+ssh://****@bitbucket.org/private-bucket/[email protected]

ireq.req and existing_ireq.req are None. I think they don't know the package name. Because if I edit the requirements.txt from this:
git+ssh://[email protected]/private-bucket/[email protected]
into that:
enums @ git+ssh://[email protected]/private-bucket/[email protected]

Then the output seems correct:

ireq: enums @ git+ssh://[email protected]/private-bucket/[email protected] from git+ssh://****@bitbucket.org/private-bucket/[email protected] (from -r requirements-dev.txt (line 33))  
existing_ireq: git+ssh://[email protected]/private-bucket/[email protected] from git+ssh://****@bitbucket.org/private-bucket/[email protected] (from -r requirements.txt (line 25))  
ireq.req: enums @ git+ssh://[email protected]/private-bucket/[email protected]  
existing_ireq.req: enums @ git+ssh://[email protected]/private-bucket/[email protected]  
key: enums

And pip-sync works.

Also, if I only update the requirements.txt to PEP 508 format, it works as well.

Thanks for your help, I hope I was clear, please let me know if you have any question, I'll be happy to give you more details!

Environment Versions

1. OS Type: macOS 11.2.3 (20D91)
2. Python version: 3.7.10
3. pip version: 21.0.1
4. pip-tools version: 6.1.0

Steps to replicate

1. have two files: requirements.in and requirements-dev.in
2. declare git dependencies (same git repo in both file)
3. pip-compile requirements.in
4. pip-compile requirements-dev.in
5. pip-sync requirements.txt requirements-dev.txt

Expected result

It should install.

Actual result

It fails.

[fj-dt]

I dug a bit more and I think I understand the issue now.

We have some projects which specify the dependency in the requirements.txt or setup.py like this:

package @ git+ssh://url@version

I think we should specify the dependency in the requirements.txt like this:

git+ssh://url@version#egg=package

And add the egg if it's in the setup.py, like this (since pip-tools seems to drop package @ to be compliant with the previous format):

install_requires=["package @ git+ssh://url@version#egg=package"]

Could you confirm this is correct for both?

Thanks!

EDIT:

If I install the project using pip and I do pip freeze > req.txt then the output is this format:

package @ git+ssh://url@version

Which makes me think pip-tools should handle this format as well when generating the .txt file.