Fix dependency conflict for issue

[NeolithEra]

#70

[CLAassistant]

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[NeolithEra]

@erickpeirson
Could you please help me review this issue?
Thank you very much!

BTW, dependency conflict problems are very common in practice in python ecosystem.
I have encountered the issues several times. What's your oppinion on the challenges of detecting and repairing such issues?
And how to avoid dependency conflict problems among python projects, especially from the ecosystem's perspective?

[erickpeirson]

These changes look good; thanks @NeolithEra for doing this.

Indeed, "dependency hell" is real. A few thoughts:

• As you pointed out in Potential dependency conflicts between arxiv-submission-core and bleach #70, we will want to be as permissive as possible in our upstream libraries, e.g. arxiv-base, arxiv-auth, arxiv-vault, etc. arxiv-submission-core is a bit of a weird one, in that it does provide a library (arxiv.auth package), but I think it's OK to be more specific here.
• We should be more systematic in pinning dependencies to specific versions in downstream applications.
• Pipenv is far from perfect, and sometimes conflicts are a little opaque. The error message for a dependency conflict suggests using pipenv --graph, but I find that this is almost always useless, since the conflict is generally buried beneath a couple of layers of dependencies. Instead, I'd suggest using pipenv lock --verbose, and using the output to retrace how pipenv ended up with the specific conflict. The latter approach has almost always yielded a clear answer.

Hope that answers your question

[NeolithEra]

@erickpeirson Thanks for your answer.

The dependency resolution in Python world is far from being easy.
I think solving dependency dell faces three challenges:

1. The first is that users cannot see the full dependency tree of their own project to decide how they want to solve it.
   To my knowledge, the state-of-the-art tools like pipenv etc. only show which libraries have been installed, rather than all the required dependencies.

2. The second concerns the non-determinism in library version installation.
   For each required library, pip installs its latest version satisfying the constraints specified in dependency configuration script.
   If the updates of libraries in PyPI ecosystem affect their installations in downstream projects, build failures may occur instantly.

3. The third is lack of an overall view of dependency conflicts between upstream and downstream projects in the ecosystem.
   Setting a version range for a library may just remove the conflicts specifically in one project.
   In some situation, this update will introduce more conflicts for other project whose dependency tree looks different.

Maybe a better mechanism of maintaining the rationality of dependency constraints among projects in PypI ecosystem is much-needed!

[erickpeirson]

1. To our knowledge, the state-of-the-art tools like pipenv etc. only show which libraries have been installed, rather than all the required dependencies.

This is where pipenv lock --verbose comes in handy. It generates a lot of output, but it does provide all of the details of package dependencies for the whole graph.

If the updates of libraries in PyPI ecosystem affect their installations in downstream projects, build failures may occur instantly.

Indeed, although using the Pipfile.lock in our Docker images helps guard against this. Pipenv will install from the lock file rather than going out and getting the most recent version.

In some situation, this update will introduce more conflicts for other project whose dependency tree looks different.

Yes. Although I think it's OK in this case, as these changes would make sense for any of the downstream apps.

Maybe a better mechanism of maintaining the rationality of dependency constraints among projects in PypI is much-needed!

I couldn't agree more! :-)

[NeolithEra]

@erickpeirson Yes. Thanks for your sharing your experience.
I saw what the output of pipenv lock --verbose look like in issue pypa/pipenv#2767, just now.
This command can output all the details of dependencies between packages that have already been installed. It is a really good tool.
Maybe it still useless for the conflicting packages that cannot be installed.

Ideally, if we can see the topological structure of a project's full dependency tree before installation or build process, then the dependency conflicts can be diagosed more intuitively (like the dependency tree shown by Maven for Java projects).
Python projects need a better tool like that.

But it is not easy. The full tree should be constructed by mining all the required packages in a topological order according to requirment.txt.

Thanks for discussing this problem with me.

Best,
Neolith

[mhl10]

Closing this as the dependencies in question where updated in a recent PR.