installation of EasyBuild fails in 2021.12 pilot after upgrade to more recent setuptools version in compat layer

[boegel]

The installation of the latest version of EasyBuild using eb --install-latest-eb-release fails after the upgrade of setuptools in the compat layer from 59.4.0 to 65.2.0 (see also EESSI/compatibility-layer#155).

The problem seem to boil down to a change in how the installation is being done: rather than a "flat" installation, .egg directories are being installed:

# note: $EESSI_SOFTWARE_SUBDIR_OVERRIDE was set to '`just-a-test`' at the top of EESSI-pilot-install-software.sh
$ ls -l /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/intel/skylake_avx512-test/software/EasyBuild/4.6.1/lib/python3.9/site-packages/
total 4
-rw-rw-r-- 1 vsc40023 vsc40023 117 Oct 17 18:21 easy-install.pth
drwxrwxr-x 2 vsc40023 vsc40023  39 Oct 17 18:20 easybuild_easyblocks-4.6.1-py3.9.egg
drwxrwxr-x 2 vsc40023 vsc40023  39 Oct 17 18:20 easybuild_easyconfigs-4.6.1-py3.9.egg
drwxrwxr-x 2 vsc40023 vsc40023  77 Oct 17 18:20 easybuild_framework-4.6.1-py3.9.egg

Although an accompanying easy-install.pth file is installed that is supposed to ensure that the .egg directories get added to the Python search path, this is not happening, which leads to import easybuild test failing.
The latter is probably the result of a deliberate breaking change in setuptools 49.0.0, see also easybuilders/easybuild-easyblocks#2693 + pypa/setuptools#2166.

What is not clear to me is why the .egg directories are now suddenly being installed, I haven't found out yet what triggers this.
It does seem to be related to recent breaking changes in setuptools though, primarily the switch to using a vendored copy of distuils rather than using the distutils that comes with the Python installation itself; see https://setuptools.pypa.io/en/latest/history.html#v60-0-0, and the deprecation of using setup.py install (cfr. pypa/setuptools#2824, although that's in place since setuptools 58.3.0).

What is becoming clear is that we need to actively move away from using setup.py install when installing EasyBuild with EasyBuild, and favor the use of pip install instead whenever it makes sense...
As a workaround, the following leads to a working installation of the latest EasyBuild release:

eb --install-latest-eb-release --try-amend=use_pip=1

[boegel]

In addition to this, the subsequent installation of EasyBuild v4.5.0 also fails, but for a different reason: EasyBuild v4.5.0 does not include the fix implemented in easybuilders/easybuild-easyconfigs#15206 yet, and because setuptools > 61.0 is used we're hitting the outright confusing installation error that mentions that path.py is not a regular file (it's not, it's a directory, see here).

[boegel]

To fix these issues, we should:

• Update the EB_EasyBuildMeta easyblock so it conditionally enables the use of pip install;
• Make that same easyblock inject "packages = []" into setup.py prior to installing to installing older easybuild-easyconfigs packages which do not have the fix yet that was added in fix installation of easybuild-easyconfigs with setuptools>=61 by explicitly declaring there are no Python packages easybuilders/easybuild-easyconfigs#15206 yet;
• Pull in the updated easyblock using --include-easyblocks-from-pr when installing EasyBuild;

[boegel]

Updated EasyBuildMeta easyblock that fixes these issues: easybuilders/easybuild-easyblocks#2805

[boegel]

Closing this, fixes are included in latest EasyBuild release (v4.6.2)