Using Bazel to build jaxlib on Windows: cannot complete build through proxy server

[dmsol98]

Description of the bug:

System Info

• Windows 10
• Python 3.9.12
• Bazel 5.1.1

I am building JAX from source on Windows but I cannot complete the build due to an SSL certificate issue. My environment variables are set up to work with my proxy, and I've configured various other programs to work with it as well. I need help configuring Bazel to work with my proxy.

When I attempt to build jaxlib with the command python .\build\build.py
I get the following output:

     _   _  __  __
    | | / \ \ \/ /
 _  | |/ _ \ \  /
| |_| / ___ \/  \
 \___/_/   \/_/\_\


Bazel binary path: C:\ProgramData\chocolatey\bin\bazel.EXE
Bazel version: 5.1.1
Python binary path: C:/Users/dmsol/Anaconda3/python.exe
Python version: 3.9
NumPy version: 1.21.5
MKL-DNN enabled: yes
Target CPU: AMD64
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
C:\ProgramData\chocolatey\bin\bazel.EXE run --verbose_failures=true --config=mkl_open_source_only :build_wheel -- --output_path=C:\Users\dmsol\Documents\ROTE\jax\dist --cpu=AMD64
WARNING: Option 'experimental_strict_action_env' is deprecated: Use --incompatible_strict_action_env instead
INFO: Reading 'startup' options from c:\users\dmsol\documents\rote\jax\.bazelrc: --host_jvm_args=-Djavax.net.ssl.trustStore=C:Program Files (x86)Javajre1.8.0_261libsecuritycacerts, --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=80
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Options provided by the client:
  Inherited 'build' options: --python_path=C:/Users/dmsol/Anaconda3/python.exe
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.bazelrc:
  Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.jax_configure.bazelrc:
  Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=C:/Users/dmsol/Anaconda3/python.exe --action_env=PYENV_ROOT --python_path=C:/Users/dmsol/Anaconda3/python.exe --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file c:\users\dmsol\documents\rote\jax\.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:mkl_open_source_only in file c:\users\dmsol\documents\rote\jax\.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:windows in file c:\users\dmsol\documents\rote\jax\.bazelrc: --copt=/D_USE_MATH_DEFINES --host_copt=/D_USE_MATH_DEFINES --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI --copt=/Zc:preprocessor --cxxopt=/std:c++17 --host_cxxopt=/std:c++17 --linkopt=/DEBUG --host_linkopt=/DEBUG --linkopt=/OPT:REF --host_linkopt=/OPT:REF --linkopt=/OPT:ICF --host_linkopt=/OPT:ICF --experimental_strict_action_env=true
INFO: Repository org_tensorflow instantiated at:
  C:/users/dmsol/documents/rote/jax/WORKSPACE:8:13: in <toplevel>
Repository rule http_archive defined at:
  C:/users/dmsol/_bazel_dmsol/hjx2szml/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in <toplevel>
WARNING: Download from https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz failed: class javax.net.ssl.SSLHandshakeException PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
ERROR: An error occurred during the fetch of repository 'org_tensorflow':
   Traceback (most recent call last):
        File "C:/users/dmsol/_bazel_dmsol/hjx2szml/external/bazel_tools/tools/build_defs/repo/http.bzl", line 100, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz] to C:/users/dmsol/_bazel_dmsol/hjx2szml/external/org_tensorflow/temp13347771389098100590/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
ERROR: C:/users/dmsol/documents/rote/jax/WORKSPACE:8:13: fetching http_archive rule //external:org_tensorflow: Traceback (most recent call last):
        File "C:/users/dmsol/_bazel_dmsol/hjx2szml/external/bazel_tools/tools/build_defs/repo/http.bzl", line 100, column 45, in _http_archive_impl
                download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz] to C:/users/dmsol/_bazel_dmsol/hjx2szml/external/org_tensorflow/temp13347771389098100590/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
INFO: Reading rc options for 'run' from c:\users\dmsol\documents\rote\jax\.bazelrc:
  Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false
WARNING: Option 'experimental_strict_action_env' is deprecated: Use --incompatible_strict_action_env instead
ERROR: @org_tensorflow//tensorflow/compiler/xla/python:enable_gpu :: Error loading option @org_tensorflow//tensorflow/compiler/xla/python:enable_gpu: no such package '@org_tensorflow//tensorflow': java.io.IOException: Error downloading [https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz] to C:/users/dmsol/_bazel_dmsol/hjx2szml/external/org_tensorflow/temp13347771389098100590/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
b''
Traceback (most recent call last):
  File "C:\Users\dmsol\Documents\ROTE\jax\build\build.py", line 528, in <module>
    main()
  File "C:\Users\dmsol\Documents\ROTE\jax\build\build.py", line 523, in main
    shell(command)
  File "C:\Users\dmsol\Documents\ROTE\jax\build\build.py", line 53, in shell
    output = subprocess.check_output(cmd)
  File "C:\Users\dmsol\Anaconda3\lib\subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "C:\Users\dmsol\Anaconda3\lib\subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['C:\\ProgramData\\chocolatey\\bin\\bazel.EXE', 'run', '--verbose_failures=true', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=C:\\Users\\dmsol\\Documents\\ROTE\\jax\\dist', '--cpu=AMD64']' returned non-zero exit status 2.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Set up a proxy server / firewall. Follow these instructions to build JAX from source on Windows for CPU only. For Bazel, I followed these instructions to install Bazelisk on my device.

After configuring the proxy for pip, MSYS2, git, and my device (using environment variables), I followed the JAX instructions, opened PowerShell as admin, and ran the command:

python .\build\build.py

Which operating system are you running Bazel on?

Windows 10

What is the output of bazel info release?

Starting local Bazel server and connecting to it... WARNING: Option 'experimental_strict_action_env' is deprecated: Use --incompatible_strict_action_env instead INFO: Reading 'startup' options from c:\users\dmsol\documents\rote\jax.bazelrc: --host_jvm_args=-Djavax.net.ssl.trustStore=C:Program Files (x86)Javajre1.8.0_261libsecuritycacerts, --host_jvm_args=-Djavax.net.ssl.trustStorePassword=changeit INFO: Options provided by the client: Inherited 'common' options: --isatty=1 --terminal_columns=213 INFO: Reading rc options for 'info' from c:\users\dmsol\documents\rote\jax.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Options provided by the client: Inherited 'build' options: --python_path=C:/Users/dmsol/Anaconda3/python.exe INFO: Reading rc options for 'info' from c:\users\dmsol\documents\rote\jax.bazelrc: Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir. --@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false INFO: Reading rc options for 'info' from c:\users\dmsol\documents\rote\jax.jax_configure.bazelrc: Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=C:/Users/dmsol/Anaconda3/python.exe --action_env=PYENV_ROOT --python_path=C:/Users/dmsol/Anaconda3/python.exe --distinct_host_configuration=false INFO: Found applicable config definition build:short_logs in file c:\users\dmsol\documents\rote\jax.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:windows in file c:\users\dmsol\documents\rote\jax.bazelrc: --copt=/D_USE_MATH_DEFINES --host_copt=/D_USE_MATH_DEFINES --copt=-DWIN32_LEAN_AND_MEAN --host_copt=-DWIN32_LEAN_AND_MEAN --copt=-DNOGDI --host_copt=-DNOGDI --copt=/Zc:preprocessor --cxxopt=/std:c++17 --host_cxxopt=/std:c++17 --linkopt=/DEBUG --host_linkopt=/DEBUG --linkopt=/OPT:REF --host_linkopt=/OPT:REF --linkopt=/OPT:ICF --host_linkopt=/OPT:ICF --experimental_strict_action_env=true WARNING: info command does not support starlark options. Ignoring options: [--@org_tensorflow//tensorflow/compiler/xla/python:enable_gpu=false, --@org_tensorflow//tensorflow/compiler/xla/python:enable_tpu=false] release 5.1.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

git remote get-url origin --> https://github.com/google/jax

git rev-parse main --> 1089c792d5e84403ee3eddcfeed15ec33dd28f20

git rev-parse HEAD --> 1089c792d5e84403ee3eddcfeed15ec33dd28f20

Have you found anything relevant by searching the web?

Similar Issues

• Unable to find valid certification path
• Bazel build behind proxy
• Work behind a proxy
• tensorflow building issue on ubuntu 14.04: Accessing remote repository through a proxy

Any other information, logs, or outputs that you want to share?

I have also tried cloning the dependencies to my local device and pointing the repository variables by using:

python build/build.py --bazel_options=--override_repository=org_tensorflow=/path/to/tensorflow

and the like for the remaining dependencies. This gets rid of the SSL error but end of giving a slew of other errors. Besides, I really need this to specifically work through the proxy so that others at my company can replicate the process with relative ease.

[sgowroji]

Dupe of #13617 (comment)

[dmsol98]

That issue looks to be a jaxlib build error. Mine is one with my company's proxy and the dependencies I'm using to build jaxlib (one of which is Bazel). This is the error line that draws my attention:

Error in download_and_extract: java.io.IOException: Error downloading [https://github.com/tensorflow/tensorflow/archive/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz] to C:/users/dmsol/_bazel_dmsol/hjx2szml/external/org_tensorflow/temp13347771389098100590/b6ab86a3f12d1f7dda9c4be801f9920b4ef2ef7a.tar.gz: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Is this not an issue with Bazel? I found similar issues here and here.

[meteorcloudy]

@dmsol98 Is it possible to construct a minimal reproduce case for this issue?

[dmsol98]

@meteorcloudy In the original post, I explained how to reproduce this issue. To summarize the steps, I followed the instructions for building JAX from source (Windows CPU only) posted here and installed Bazelisk by following these instructions.

I've recently realized that this issue has more to do with certificate authentication than working through a proxy.

[Wyverald]

It would be very helpful to have a minimal reproduction case. Although you did provide reproduction steps (and thanks for that!), building the entirety of JAX is rather complicated and it's very hard for us to pinpoint where the problem is.