Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bazel] [rules_foreign_cc] ModuleNotFoundError: No module named 'tools' #1420

Open
allsey87 opened this issue Jul 1, 2024 · 11 comments · May be fixed by #1423
Open

[bazel] [rules_foreign_cc] ModuleNotFoundError: No module named 'tools' #1420

allsey87 opened this issue Jul 1, 2024 · 11 comments · May be fixed by #1423

Comments

@allsey87
Copy link
Contributor

allsey87 commented Jul 1, 2024

When building a simple hello world binary with Meson, I am hitting the following error

mesonbuild.utils.universal.EnvironmentException: Unknown compiler(s): [['/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/external/emsdk/emscripten_toolchain/emcc.sh']]

Digging a bit deeper, it seems emcc.py --version is failing and unable to find the tools module.

The Meson build system
Version: 1.1.1
Source dir: /home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/testing/hellomeson
Build dir: /home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/bazel-out/k8-fastbuild/bin/testing/test_meson.build_tmpdir
Build type: native build
Project name: tutorial
Project version: undefined
running cmd: ['/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/external/emsdk/emscripten_toolchain/emcc.sh', '--version']
out: 
err: Traceback (most recent call last):
  File "/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/9/execroot/_main/external/emscripten_bin_linux/emscripten/emcc.py", line 23, in <module>
    from tools.toolchain_profiler import ToolchainProfiler
ModuleNotFoundError: No module named 'tools'

Perhaps this isn't included in the compiler_files filegroup which is passed to the toolchain?

@allsey87
Copy link
Contributor Author

allsey87 commented Jul 1, 2024

Perhaps this isn't included in the compiler_files filegroup which is passed to the toolchain?

This can't be the case, this module import is the first statement in emcc.py...

I checked the contents of external/emscripten_bin_linux/emscripten/ inside the sandbox and emcc.py + the tools directory is there. I also tried running emcc.py from my workspace and it didn't seem to have a problem with importing the tools module (although it complains that there is no config file or LLVM_ROOT set as expected).

python3 /home/develop/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/sandbox/processwrapper-sandbox/10/execroot/_main/external/emscripten_bin_linux/emscripten/emcc.py --version
emcc: warning: config file not found: /home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/external/emscripten_bin_linux/emscripten/.emscripten.  You can create one by hand or run `emcc --generate-config`
emcc: error: LLVM_ROOT not set in config (/home/developer/.cache/bazel/_bazel_developer/25e07d78077dfe1eca932359d50e41ef/external/emscripten_bin_linux/emscripten/.emscripten), and `clang` not found in PATH

@walkingeyerobot
Copy link
Collaborator

I don't quite understand here. My understanding is that Meson is a build system like bazel. Are you using both at once? Is one invoking the other?

@allsey87
Copy link
Contributor Author

allsey87 commented Jul 1, 2024

Indeed, there is often a need in Bazel to build software that is primarily built using other build systems. In this case, I am looking at building numpy which uses the Meson build system. rules_foreign_cc allows you to build this software in Bazel by integrating support for calling into other build systems such as automake, cmake, meson etc and exporting the build artefacts from them for use in other rules.

@allsey87
Copy link
Contributor Author

allsey87 commented Jul 9, 2024

Finally got to the root cause of why the configure_make rule from rules_foreign_cc works but the meson rule does not.

Since Meson is written in Python, it uses Bazel's py_binary which sets PYTHONSAFEPATH by default. This environment variable is inherited by emcc.sh and emcc.py, the latter relying on having the containing directory of emcc.py added to sys.path. However PYTHONSAFEPATH disables this behaviour and hence the import of the tools module fails.

@allsey87
Copy link
Contributor Author

allsey87 commented Jul 9, 2024

@walkingeyerobot I think the solution to this problem is to unset the PYTHONSAFEPATH environment variable in env.sh and env.bat. I have tested this and it works. See #1423.

@allsey87 allsey87 linked a pull request Jul 9, 2024 that will close this issue
@kleisauke
Copy link
Collaborator

FWIW, I see a similar issue when trying to run Emscripten's test suite with Python 3.12.

$ python -V
Python 3.12.4
$ ./test/runner other.*code_size* other.*metadce* --rebase
Traceback (most recent call last):
  File "/home/kleisauke/emscripten/./test/runner.py", line 38, in <module>
    import jsrun
  File "/home/kleisauke/emscripten/test/jsrun.py", line 6, in <module>
    import common
  File "/home/kleisauke/emscripten/test/common.py", line 34, in <module>
    import clang_native
  File "/home/kleisauke/emscripten/test/clang_native.py", line 10, in <module>
    from tools.shared import PIPE, run_process, CLANG_CC, CLANG_CXX
ModuleNotFoundError: No module named 'tools.shared'

I'm currently working around this by removing the -E here:
https://github.com/emscripten-core/emscripten/blob/060c3fc4f8a8fd1040fc25107d200df38fdb5202/test/runner#L35

And by setting the PYTHONPATH=. env variable.

@sbc100
Copy link
Collaborator

sbc100 commented Jul 12, 2024

@kleisauke, do you think that is somehow related? Is this related to PYTHONSAFEPATH? The runner.py script contains the following lines near the top.. can you figure out why they are not working:

__rootpath__ = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))    
sys.path.append(__rootpath__)    

@sbc100
Copy link
Collaborator

sbc100 commented Jul 12, 2024

3.12.2 seem to work find for me..

$ python3.12 -V
Python 3.12.2
$ python3.12 ./test/runner.py 
Test suites:
[]

@kleisauke
Copy link
Collaborator

@sbc100 Sorry for the noise. I initially thought my issue was similar to this one, but it turns out to be different.

It seems that installing Detectron2 triggers this issue:

$ pip install 'git+https://github.com/facebookresearch/detectron2.git'

After installation, it creates the following files in Python's per user site-packages directory (site-packages/tools/*).

$ ls $(python3 -m site --user-site)/tools
analyze_model.py  benchmark.py  convert-torchvision-to-d2.py  __init__.py  lazyconfig_train_net.py  lightning_train_net.py  plain_train_net.py  __pycache__  train_net.py  visualize_data.py  visualize_json_results.py

Since this path is included in Python's default search path, it conflicts with the tools directory in Emscripten.

Here's a standalone reproducer:

$ git clone https://github.com/emscripten-core/emscripten.git
$ cd emscripten
$ docker run -v $(pwd):/app -w /app -it fedora:40 sh -c 'mkdir -p $(python3 -m site --user-site)/tools && touch $(python3 -m site --user-site)/tools/__init__.py && python3 ./test/runner.py'
Traceback (most recent call last):
  File "/app/./test/runner.py", line 38, in <module>
    import jsrun
  File "/app/test/jsrun.py", line 6, in <module>
    import common
  File "/app/test/common.py", line 34, in <module>
    import clang_native
  File "/app/test/clang_native.py", line 10, in <module>
    from tools.shared import PIPE, run_process, CLANG_CC, CLANG_CXX
ModuleNotFoundError: No module named 'tools.shared'

@sbc100
Copy link
Collaborator

sbc100 commented Jul 12, 2024

I guess that solution to that problem is to put our directory first in sys.path?

sbc100 added a commit to sbc100/emscripten that referenced this issue Jul 12, 2024
This avoids issues where something called e.g. `tools` already
exists in the python path.

See emscripten-core/emsdk#1420 (comment)
@sbc100
Copy link
Collaborator

sbc100 commented Jul 12, 2024

See emscripten-core/emscripten#22230

sbc100 added a commit to emscripten-core/emscripten that referenced this issue Jul 13, 2024
This avoids issues where something called e.g. `tools` already exists in
the python path.

See
emscripten-core/emsdk#1420 (comment)
verhovsky pushed a commit to verhovsky/emscripten that referenced this issue Jul 30, 2024
…e#22230)

This avoids issues where something called e.g. `tools` already exists in
the python path.

See
emscripten-core/emsdk#1420 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants