Exclude packages required only by unsafe packages

[jdufresne]

Recent versions of setuptools have dependencies:

https://github.com/pypa/setuptools/blob/6d89f362c9657e12fe2500c061e77747305b76e4/setup.py#L165-L168

If a package requires setuptools, it is commented out in the compiled
requirements.txt. As setuptools isn't included, its dependencies should also
not be included.

Should fix #445

[IvanAnishchuk]

@jdufresne This is an important fix for many people. Could you please fix the travis build?

[jdufresne]

Sure, I can revisit this.

It occurred to me that this needs to happen recursively. For example, packages included by packages included by setuptools should also not be included in the requirements. Until that is fixed, this is not ready for merge. I'll take a look at that problem once resolving the merge conflict and Travis build.

[davidovich]

@jdufresne I am really not convinced that this needs to happen at output time. I think the analysis should take place at resllolve time. What is your take on this?

[IvanAnishchuk]

@davidovich Yeah, that makes sense. Somewhere in _resolve_one_round() we could avoid adding unsafe requirements and their own requirements won't be added either.

[jdufresne]

The updated PR is ready for review. I have:

• Rebased to resolve merge conflicts
• Reworked logic to run at resolve time instead of output time
• As a result, we don't need allow_unsafe at output time which allowed some removal of dead code

Just a thought, is there a good use case for allow_unsafe? I was curious about removing it entirely, but I'm not sure if the backwards compatibility would be a problem for some.

[davidovich]

@jdufresne could you add a test to show that appdirs, packaging, etc is not added to the compiled requirements due to being unsafe?

[jdufresne]

@davidovich I have updated the test fixture to include appdirs and packaging as dependencies of setuptools. In test_resolver.py, these dependencies are not included in the resolved set of the the html5lib test case. I believe this demonstrates what you're asking for.

Let me if you had something else in mind. If so, could you provide a bit of guidance as to what the test might look like? I'd be happy to write additional tests, just want to make sure I'm adding what you're looking for.

[davidovich]

Yes that is what I had in mind.

I will try to merge tomorrow, or next Monday.

[davidovich]

Thanks @jdufresne !