riscv64 build issue - OOM kill, high memory usage

[ThomasWaldmann]

Wanted to try whether borgbackup works on RISC-V, but I already failed setting up the development requirements for borg.

I tried this on Proxmox PVE, using a container with a preinstalled Ubuntu 22.04 riscv64 system image (and installed all the missing stuff manually).

• Python 3.10.6
• cryptography 40.0.1
• cffi 1.15.1
• pip 23.0.1
• setuptools 67.6.1

$ pip install -v cryptography
...
  running build_ext
  running build_rust
  cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features pyo3/extension-module pyo3/abi3-py36 -- --crate-type cdylib
      Updating crates.io index

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation/ for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq/
      4) Ensure you have a recent Rust toolchain installed:
         https://cryptography.io/en/latest/installation/#rust

      Python: 3.10.6
      platform: Linux-5.15.102-1-pve-riscv64-with-glibc2.35
      pip: n/a
      setuptools: 67.6.1
      setuptools_rust: 1.5.2
      rustc: 1.65.0
      =============================DEBUG ASSISTANCE=============================

  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features 'pyo3/extension-module pyo3/abi3-py36' -- --crate-type cdylib` failed with code -9
  error: subprocess-exited-with-error
  
  × Building wheel for cryptography (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/tw/w/borg-env/bin/python3 /home/tw/w/borg-env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpt1xmuo4v
  cwd: /tmp/pip-install-_r3y8dnc/cryptography_4e062afca8d24cbaa29fd07e87886e35
  Building wheel for cryptography (pyproject.toml) ... error
  ERROR: Failed building wheel for cryptography
Failed to build cryptography
ERROR: Could not build wheels for cryptography, which is required to install pyproject.toml-based projects

[alex]

The relevant line is:

  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path src/rust/Cargo.toml --release -v --features 'pyo3/extension-module pyo3/abi3-py36' -- --crate-type cdylib` failed with code -9

Negative value means that the process exited due to a signal, I believe -9 is SIGKILL.

Is it possible the process was killed by the OOM killer?

[ThomasWaldmann]

BTW, the debug assistance section says pip: n/a, but it definitely was available.

[ThomasWaldmann]

@alex thanks for the hint, that seems to be the problem.

I didn't expect having to give 16GB RAM to that container (and with that, it is still using the swap).

Is that normal / expected?

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                      
   2252 tw        20   0   17.1g  14.4g  29976 S   0.0  90.0   6:38.87 cargo                                                                        
   3309 tw        20   0 1356520 547856 153088 S  99.3   3.3   1:13.38 rustc                                                                        
...

[alex]

I would not have expected it to need 16GB, no. Our github actions runners have 7GB of RAM, and we build fine there.

My first guess would be that since riscv64 is less widely used, there may be issues in llvm/clang/rustc that cause increased memory usage. But that's just speculation.

[ThomasWaldmann]

OK, it worked! \o/

Looks like one needs 16GB RAM + 4GB Swap at least. cargo using most of that most of the time.

[alex]

Huh, cargo is really just coordinating shelling out to rustc, its very surprising to me that it'd be using a ton of RAM.

[Xe]

I am trying to reproduce this issue on my VisionFive 2 board. I am seeing pip install cryptography peg one core and use 128-256 MB of the system's available 8 GB of ram. What build configuration flags are you using?

[ThomasWaldmann]

I didn't do anything special. Could it be that running this in a proxmox container on a ryzen host makes a difference?

[Xe]

How are you cross-compiling from amd64 to riscv64?

[ThomasWaldmann]

Not at all, I compile inside the riscv64 container.

The userspace runs via binfmt_misc. Kernel is the proxmox (amd64) kernel.