Although not overly complicated from a pure technical point of view, this QEP is an unusual long one. Certain gaps in QGIS' relationship to the Python ecosystem make QGIS fall far behind its true potential. This document proposes to close the gaps. However, figuring out an actually good, sustainable solution - not just any solution - is not trivial. It requires a thorough analysis of (1) the problem, (2) the current codebase, (3) all possible options forward and (4) potential pitfalls. Unfortunately, the topic of Python modules and packaging is also paved with common misconceptions. This QEP has therefore intentionally been developed in great detail in a separate GitHub repository. The following table of contents provides direct links to the individual chapters of the QEP within the separate repository. Once the discussion ends, a final version of this QEP might replace this "TL;DR" section.
The described approach does not introduce any changes to the QGIS "legacy" Python plugin "package" format. It is therefore perfectly feasible to distribute a QGIS Python plugin in multiple formats, e.g. in the legacy format and as a conda package, while maintaining the entire project in a simple directory tree within a git repository or similar.
Because the support for conda and pip should not replace the existing "legacy" QGIS plugin "package" format, all three distribution paths are supposed to coexist. It is proposed to design the new pluginmanager module in a modular way, containing one "backend" each for conda, pip and the "legacy" QGIS Python plugin "package" format. Each backend is on its own responsible for installing & uninstalling QGIS Python plugins as well as handling their dependencies. Dependencies can intentionally not be handled across multiple backends, which would blow up the complexity of the implementation beyond a manageable point. The pluginmanager will only handle direct conflicts between the different backends and prohibit broken installations by exploiting dry-run capabilities of all involved tools. If a plugin is available from multiple backends (and/or in multiple versions), a user will be asked to choose a backend (and a version) for installation.
QGIS already supports package repositories. The concept will simply be extended by allowing conda and pip repositories with configuration options specific to those backends. In the case of conda, different conda repositories within a QGIS configuration could refer to different conda package channels. In the case of pip, different pip repositories could refer to specific package sources.