python module: add option to specify a python environment to install to

The default behavior of installing relative to prefix may be unexpected,
and is definitely wrong in many cases.

Give users control in order to specify that yes, they actually want to
install to a venv.

This is particularly useful for projects that use meson as a build
system for a python module, where *all* files shall be installed into
the python site-packages.

docs/markdown/Builtin-options.md

@@ -271,10 +271,11 @@ name with the module name: `-D<module>.<option>=<value>` (e.g. `-Dpython.platlib
 
 ### Python module
 
-| Option           | Default value | Possible values | Description |
-| ------           | ------------- | --------------- | ----------- |
-| platlibdir       |               | Directory path  | Directory for site-specific, platform-specific files (Since 0.60.0) |
-| purelibdir       |               | Directory path  | Directory for site-specific, non-platform-specific files  (Since 0.60.0) |
+| Option           | Default value | Possible values             | Description |
+| ------           | ------------- | -----------------           | ----------- |
+| install_env      | prefix        | {auto,prefix,system,venv}   | Which python environment to install to (Since 0.62.0) |
+| platlibdir       |               | Directory path              | Directory for site-specific, platform-specific files (Since 0.60.0) |
+| purelibdir       |               | Directory path              | Directory for site-specific, non-platform-specific files  (Since 0.60.0) |
 
 *Since 0.60.0* `python.platlibdir` and `python.purelibdir` options are used by
 python module methods `python.install_sources()` and `python.get_install_dir()`.
@@ -283,3 +284,14 @@ relative to the installation `prefix`, which will often result in installed pyth
 modules to not be found by the interpreter unless `prefix` is `/usr` on Linux,
 or for example `C:\Python39` on Windows. These options can be absolute paths
 outside of `prefix`.
+
+*Since 0.62.0* The `python.install_env` option is used to detect the correct
+installation path. Setting to `system` will avoid making the paths relative to
+`prefix` and instead use the global site-packages of the selected python
+interpreter directly, even if it is a venv. Setting to `venv` will instead use
+the paths for the virtualenv the python found installation comes from (or fail
+if it is not a virtualenv).  Setting to `auto` will check if the found
+installation is a virtualenv, and use `venv` or `system` as appropriate (but
+never `prefix`). This option is mutually exclusive with the `platlibdir`/`purelibdir`.
+
+For backwards compatibility purposes, the default `install_env` is `prefix`.

docs/markdown/snippets/python_module_env.md

@@ -0,0 +1,9 @@
+## New option to choose python installation environment
+
+It is now possible to specify `-Dpython.install_env` and choose how python modules are installed.
+
+- `venv`: assume that a virtualenv is active and install to that
+- `system`: install to the global site-packages of the selected interpreter
+  (the one that the venv module calls --system-site-packages)
+- `prefix`: preserve existing behavior
+- `auto`: autodetect whether to use venv or system

mesonbuild/coredata.py

@@ -1218,6 +1218,8 @@ def add_to_argparse(self, name: str, parser: argparse.ArgumentParser, help_suffi
     (OptionKey('force_fallback_for'), BuiltinOption(UserArrayOption, 'Force fallback for those subprojects', [])),
 
     # Python module
+    (OptionKey('install_env', module='python'),
+     BuiltinOption(UserComboOption, 'Which python environment to install to', 'prefix', choices=['auto', 'prefix', 'system', 'venv'])),
     (OptionKey('platlibdir', module='python'),
      BuiltinOption(UserStringOption, 'Directory for site-specific, platform-specific files.', '')),
     (OptionKey('purelibdir', module='python'),

mesonbuild/modules/python.py

@@ -338,11 +338,13 @@ def links_against_libpython():
 print(json.dumps({
   'variables': variables,
   'paths': paths,
+  'sysconfig_paths': sysconfig.get_paths(),
   'install_paths': install_paths,
   'sys_paths': sys.path,
   'version': sysconfig.get_python_version(),
   'platform': sysconfig.get_platform(),
   'is_pypy': '__pypy__' in sys.builtin_module_names,
+  'is_venv': sys.prefix != variables['base_prefix'],
   'link_libpython': links_against_libpython(),
 }))
 '''
@@ -352,7 +354,9 @@ class PythonIntrospectionDict(TypedDict):
 
         install_paths: T.Dict[str, str]
         is_pypy: bool
+        is_venv: bool
         link_libpython: bool
+        sysconfig_paths: T.Dict[str, str]
         paths: T.Dict[str, str]
         platform: str
         suffix: str
@@ -377,7 +381,9 @@ def __init__(self, name: str, command: T.Optional[T.List[str]] = None,
         self.info: 'PythonIntrospectionDict' = {
             'install_paths': {},
             'is_pypy': False,
+            'is_venv': False,
             'link_libpython': False,
+            'sysconfig_paths': {},
             'paths': {},
             'platform': 'sentinal',
             'variables': {},
@@ -422,7 +428,22 @@ def _get_path(self, state: T.Optional['ModuleState'], key: str) -> None:
             return rel_path
         value = state.get_option(f'{key}dir', module='python')
         if value:
+            if state.is_user_defined_option('install_env', module='python'):
+                raise mesonlib.MesonException(f'python.{key}dir and python.install_env are mutually exclusive')
             return value
+
+        install_env = state.get_option('install_env', module='python')
+        if install_env == 'auto':
+            install_env = 'venv' if self.info['is_venv'] else 'system'
+
+        if install_env == 'system':
+            rel_path = os.path.join(self.info['variables']['prefix'], rel_path)
+        elif install_env == 'venv':
+            if not self.info['is_venv']:
+                raise mesonlib.MesonException('python.install_env cannot be set to "venv" unless you are in a venv!')
+            # inside a venv, deb_system is *never* active hence info['paths'] may be wrong
+            rel_path = self.info['sysconfig_paths'][key]
+
         # Use python's path relative to prefix, and warn if that's not a location
         # python will lookup for modules.
         abs_path = Path(state.get_option('prefix'), rel_path)