Issues with torch in x86 Docker

[Pavidiran]

I tried to import torch in the x86 Docker and got the following error:

  Traceback (most recent call last):
  File "<stdin>", line 1, in <module> 
  File "/usr/local/lib/python3.10/dist-packages/torch/__init__.py", line 236, in <module>
    from torch._C import *  # noqa: F403
  ImportError: /opt/hpcx/ucx/lib/libucs.so.0: undefined symbol: ucm_set_global_opts

This also occures when i do this in the terminal:

 python -c "import torch; print(torch.__version__)

[D-1shu]

Hi @Pavidiran did you find any fix for this? I am facing the same issue.

[Pavidiran]

No. not yet

[balakumar-s]

Can you run the below script inside the docker and see if that fixes your issue?

sudo apt-get update && sudo apt-get install --reinstall -y \
  libmpich-dev \
  hwloc-nox libmpich12 mpich

[Pavidiran]

Thank you very much. This works.

[D-1shu]

@balakumar-s is it possible for you to give me insight on how exactly reinstalling these packages are fixing the torch import issue? When I am adding ros2 humble to the the same docker file (x86) and build it, on running examples I get exactly the same import error until I run this inside the docker container.

sudo apt-get update && sudo apt-get install --reinstall -y \
  libmpich-dev \
  hwloc-nox libmpich12 mpich

[anubhav-agrawal-mu-sigma]

@balakumar-s is it possible for you to give me insight on how exactly reinstalling these packages are fixing the torch import issue? When I am adding ros2 humble to the the same docker file (x86) and build it, on running examples I get exactly the same import error until I run this inside the docker container.

sudo apt-get update && sudo apt-get install --reinstall -y \
  libmpich-dev \
  hwloc-nox libmpich12 mpich

Hi, Were you able to get the answer?

[D-1shu]

Running ik_example.pygives this issue

Traceback (most recent call last):
  File "/pkgs/curobo/examples/ik_example.py", line 232, in <module>
    demo_basic_ik()
  File "/pkgs/curobo/examples/ik_example.py", line 42, in demo_basic_ik
    ik_config = IKSolverConfig.load_from_robot_config(
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/curobo/wrap/reacher/ik_solver.py", line 192, in load_from_robot_config
    arm_rollout_mppi = ArmReacher(cfg)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/curobo/rollout/arm_reacher.py", line 166, in __init__
    ArmBase.__init__(self)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/curobo/rollout/arm_base.py", line 233, in __init__
    self._init_after_config_load()
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/dist-packages/curobo/rollout/arm_base.py", line 265, in _init_after_config_load
    self.bound_cost = BoundCost(self.cost_cfg.bound_cfg)
  File "/usr/local/lib/python3.10/dist-packages/curobo/rollout/cost/bound_cost.py", line 87, in __init__
    init_warp()
  File "/usr/local/lib/python3.10/dist-packages/curobo/util/warp.py", line 25, in init_warp
    wp.init()
  File "/usr/local/lib/python3.10/dist-packages/warp/context.py", line 5042, in init
    runtime = Runtime()
  File "/usr/local/lib/python3.10/dist-packages/warp/context.py", line 2956, in __init__
    if devices_without_uva:
UnboundLocalError: local variable 'devices_without_uva' referenced before assignment

[balakumar-s]

We are looking at this issue. In the mean time, install a older warp version with pip install warp-lang==0.13.0

[D-1shu]

Thank you @balakumar-s. Installating older warp version fixes the issue.

[DeclK]

Is there a way to solve this just through docker file? I tried everything, it seems it can only work in a running container...

[D-1shu]

@DeclK I couldn't find a way to solve this through docker file as well.

[DeclK]

Just found a way to do it: add the ucx path to the LD_LIBRARY_PATH would be OK! see this issue

export LD_LIBRARY_PATH=/opt/hpcx/ucx/lib:$LD_LIBRARY_PATH

[D-1shu]

awesome! this works.

[balakumar-s]

Thanks @DeclK for the fix. We will add this to the default docker file in the next release!