[Suggestion] Force the import library selection priority as the same order as dll_path

[daviddwlee84]

I created an additional virtual environment with venv and then installed XGBoost.
There is an older version XGBoost installed by Conda in another virtual environments directory.

core.py

Maybe it is better to break the lib_paths for-loop when lib_success is True.
Which will first use the library along with the package instead of other installed XGBoost in the system somewhere else.

    for lib_path in lib_paths:

      ...

      # Add this
      if lib_success:
          break

libpath.py

As the comment in libpath.py, the dll_path last item was designed to be the last option. But currently, it will be selected if available.

    curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
    dll_path = [
        # normal, after installation `lib` is copied into Python package tree.
        os.path.join(curr_path, "lib"),
        # editable installation, no copying is performed.
        os.path.join(curr_path, os.path.pardir, os.path.pardir, "lib"),
        # use libxgboost from a system prefix, if available.  This should be the last
        # option.
        os.path.join(sys.base_prefix, "lib"),
    ]

And might cause error like this which is misleading:

Python 3.8.13 (default, Mar 28 2022, 11:38:47)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.3 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import xgboost as xgb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 1
----> 1 import xgboost as xgb

File /mnt/NAS/sda/ShareFolder/lidawei/ExperimentNotebook/daweilee_research/lib/python3.8/site-packages/xgboost/__init__.py:7
      1 """XGBoost: eXtreme Gradient Boosting library.
      2
      3 Contributors: https://github.com/dmlc/xgboost/blob/master/CONTRIBUTORS.md
      4 """
      6 from . import tracker  # noqa
----> 7 from . import collective, dask, rabit
      8 from .core import (
      9     Booster,
     10     DataIter,
   (...)
     15     build_info,
     16 )
     17 from .tracker import RabitTracker  # noqa

File /mnt/NAS/sda/ShareFolder/lidawei/ExperimentNotebook/daweilee_research/lib/python3.8/site-packages/xgboost/collective.py:12
      9 import numpy as np
     11 from ._typing import _T
---> 12 from .core import _LIB, _check_call, c_str, from_pystr_to_cstr, py_str
     14 LOGGER = logging.getLogger("[xgboost.collective]")
     17 def init(**args: Any) -> None:

File /mnt/NAS/sda/ShareFolder/lidawei/ExperimentNotebook/daweilee_research/lib/python3.8/site-packages/xgboost/core.py:266
    262     return lib
    265 # load the XGBoost library globally
--> 266 _LIB = _load_lib()
    269 def _check_call(ret: int) -> None:
    270     """Check the return value of C API call
    271
    272     This function will raise exception when error occurs.
   (...)
    278         return value from API calls
    279     """

File /mnt/NAS/sda/ShareFolder/lidawei/ExperimentNotebook/daweilee_research/lib/python3.8/site-packages/xgboost/core.py:260, in _load_lib()
    251     libver_str = ".".join((str(v) for v in libver))
    252     msg = (
    253         "Mismatched version between the Python package and the native shared "
    254         f"""object.  Python package version: {pyver_str}. Shared object """
   (...)
    258         "please remove one of the installations."
    259     )
--> 260     raise ValueError(msg)
    262 return lib

ValueError: Mismatched version between the Python package and the native shared object.  Python package version: 2.0.2. Shared object version: 1.5.0. Shared object is loaded from: /mnt/NAS/sda/ShareFolder/anaconda3/envs/research/lib/libxgboost.so.
Likely cause:
  * XGBoost is first installed with anaconda then upgraded with pip. To fix it please remove one of the installations.

After adding if lib_success: break the error is fixed.

[trivialfis]

There is an older version XGBoost installed by Conda in another virtual environments directory.

I'm curious how the two environments got mixed together in the first place.

[daviddwlee84]

XGBoost looks for sys.base_prefix, I think that's why.

ipdb> pprint(dll_path)
['/mnt/NAS/sda/ShareFolder/lidawei/ExperimentNotebook/daweilee_research/lib/python3.8/site-packages/xgboost/lib',
 '/mnt/NAS/sda/ShareFolder/lidawei/ExperimentNotebook/daweilee_research/lib/python3.8/site-packages/xgboost/../../lib',
 '/mnt/NAS/sda/ShareFolder/anaconda3/envs/research/lib']
ipdb> sys.base_prefix
'/mnt/NAS/sda/ShareFolder/anaconda3/envs/research'

sys — System-specific parameters and functions — Python 3.12.0 documentation

sys.base_prefix
Set during Python startup, before site.py is run, to the same value as prefix. If not running in a virtual environment, the values will stay the same; if site.py finds that a virtual environment is in use, the values of prefix and exec_prefix will be changed to point to the virtual environment, whereas base_prefix and base_exec_prefix will remain pointing to the base Python installation (the one which the virtual environment was created from).

[trivialfis]

cc @hcho3 @oliverholworthy

Related #9349 .

[mvsjober]

We have the same issue in an HPC environment where the base system is installed with conda and a user wants to install a newer version with pip install xgboost --user. It still tries to load the libxgboost.so from the old conda environment, even though the pip package has the newer one and it's the first in the list returned by find_lib_path().

Adding the if lib_success: break as suggested should do the right thing (selects the first appropriate choice).

[trivialfis]

I'm not against of the change if it reduces confusion for users. But it's a bit odd to me that one would mix environments and sources of packages, it's an invitation of troubles.

[trivialfis]

I think the comment #9790 (comment) makes sense, I will prepare a PR for that change.

[trivialfis]

Hi, I opened #9860, could you please take a look when you are available?