Need help on installing psycopg2 library

[hariharasudhan-nineleaps]

🐞 bug report

Affected Rule

Need help on installing psycopg2 library

Is this a regression?

n/a

Description

I am new to the bazel build tool, when i am trying to install psycopg2, its failing with error message ld: library not found for -lssl, how can i pass LD_LIBRARY_PATH to rules_python

🔬 Minimal Reproduction

# WORKSPACE
# download rules-python
http_archive(
    name = "rules_python",
    sha256 = "0a8003b044294d7840ac7d9d73eef05d6ceb682d7516781a4ec62eeb34702578",
    strip_prefix = "rules_python-0.24.0",
    url = "https://github.com/bazelbuild/rules_python/releases/download/0.24.0/rules_python-0.24.0.tar.gz",
)

# register python toolchain
load("@rules_python//python:repositories.bzl", "python_register_toolchains")

python_register_toolchains(
    name = "python3_11",
    python_version = "3.11",
)

# install pip packages with toolchain
load("@python3_11//:defs.bzl", "interpreter")
load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "app_deps",
    python_interpreter_target = interpreter,
    requirements_lock = "//:requirements.txt",
)

load("@app_deps//:requirements.bzl", "install_deps")

install_deps()

# BUILD
load("@rules_python//python:defs.bzl", "py_binary")
load("@app_deps//:requirements.bzl", "requirement")

py_binary(
    name = "user-service",
    srcs = ["main.py"],
    main = "main.py",
    deps = [
        requirement("uvicorn"),
        requirement("fastapi"),
        requirement("psycopg2")
    ]
)

# requirements.txt
annotated-types==0.5.0 ; python_version >= "3.10" and python_version < "4.0"
anyio==3.7.1 ; python_version >= "3.10" and python_version < "4.0"
click==8.1.6 ; python_version >= "3.10" and python_version < "4.0"
colorama==0.4.6 ; python_version >= "3.10" and python_version < "4.0" and platform_system == "Windows"
exceptiongroup==1.1.2 ; python_version >= "3.10" and python_version < "3.11"
fastapi==0.100.0 ; python_version >= "3.10" and python_version < "4.0"
h11==0.14.0 ; python_version >= "3.10" and python_version < "4.0"
httptools==0.6.0 ; python_version >= "3.10" and python_version < "4.0"
idna==3.4 ; python_version >= "3.10" and python_version < "4.0"
psycopg2==2.9.6 ; python_version >= "3.10" and python_version < "4.0"
pydantic-core==2.4.0 ; python_version >= "3.10" and python_version < "4.0"
pydantic==2.1.1 ; python_version >= "3.10" and python_version < "4.0"
python-dotenv==1.0.0 ; python_version >= "3.10" and python_version < "4.0"
pyyaml==6.0.1 ; python_version >= "3.10" and python_version < "4.0"
sniffio==1.3.0 ; python_version >= "3.10" and python_version < "4.0"
starlette==0.27.0 ; python_version >= "3.10" and python_version < "4.0"
typing-extensions==4.7.1 ; python_version >= "3.10" and python_version < "4.0"
uvicorn[standard]==0.23.1 ; python_version >= "3.10" and python_version < "4.0"
uvloop==0.17.0 ; (sys_platform != "win32" and sys_platform != "cygwin") and platform_python_implementation != "PyPy" and python_version >= "3.10" and python_version < "4.0"
watchfiles==0.19.0 ; python_version >= "3.10" and python_version < "4.0"
websockets==11.0.3 ; python_version >= "3.10" and python_version < "4.0"

🔥 Exception or Error

ld: warning: directory not found for option '-L/var/folders/c8/z52g4gc5679gygt8z0dx0jqm0000gn/T/tmpe6a8prsz/tools/deps/lib'
      ld: warning: directory not found for option '-L/install/lib'
      ld: library not found for -lssl
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for psycopg2
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "", line 198, in _run_module_as_main
  File "", line 88, in _run_code
  File "/private/var/tmp/_bazel_nineleaps/9af3b1d4a829087f684be9f7091f1e90/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 452, in 
    main()
  File "/private/var/tmp/_bazel_nineleaps/9af3b1d4a829087f684be9f7091f1e90/external/rules_python/python/pip_install/tools/wheel_installer/wheel_installer.py", line 429, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_nineleaps/9af3b1d4a829087f684be9f7091f1e90/external/python3_11_aarch64-apple-darwin/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_nineleaps/9af3b1d4a829087f684be9f7091f1e90/external/python3_11_aarch64-apple-darwin/bin/python3', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/var/folders/3k/h6_h4n8j2rl3xbpm0plbhy080000gp/T/tmpfr9rk5nm']' returned non-zero exit status 1.
) error code: '1'

🌍 Your Environment

Operating System:

  
    ProductName:		macOS
    ProductVersion:		13.4.1
  

Output of bazel version:

  
       6.2.1
  

Rules_python version:

  
      0.24.0
  

Anything else relevant?
When i am trying to install without bazel its installing and working as expected, its happening only inside bazel

[jvolkman]

Have you tried using the psycopg2-binary package instead? It ships pre-built binary wheels.

[hariharasudhan-nineleaps]

Yes it is working, but using psycopg2-binary is not recommended for all right https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary

Also what should i do, in-case if some other library requires lssl which doesn't have binary like this?

[chrislovecnm]

You can use: bazel build //:myproject --action_env=MYENV=myvalue to set the environment variable. Or you can build your own version of lib SSL, or you can include the library in your own source tree and have pip pick it up.

@hariharasudhan-nineleaps take a look at: #48 if you want to host or build your own lib ssl.

[michaeljs1990]

You can use: bazel build //:myproject --action_env=MYENV=myvalue to set the environment variable. Or you can build your own version of lib SSL, or you can include the library in your own source tree and have pip pick it up.

@hariharasudhan-nineleaps take a look at: #48 if you want to host or build your own lib ssl.

Thank you for this! This is what I did to get around this for now but I really want to come back and provide a proper fix for this later. I think some of the work @jvolkman has been doing on shipping cross platform python packages may be part of that but I need to take a deeper look and see if it's feasable to start using in my repo.

for now this fixed it but it will break at some point if brew info openssl path ever changes.

In my .bazelrc file.

build:macos --action_env=LDFLAGS="-I/opt/homebrew/opt/openssl@3/include -L/opt/homebrew/opt/openssl@3/lib"

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"