Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] GPU container does not report GPU memory usage accurately #1071

Open
mmartial opened this issue Sep 8, 2024 · 0 comments
Open

[Bug] GPU container does not report GPU memory usage accurately #1071

mmartial opened this issue Sep 8, 2024 · 0 comments

Comments

@mmartial
Copy link

mmartial commented Sep 8, 2024

Description of the bug

The dashdot:nvidia container does not report memory usage accurately.

I am attaching screenshots of memory consumption using a ComfyUI workflow.
The workflow run uses close to 80% memory, but Dashdot reports only 30% VRAM use (despite recognizing the correct GPU model and memory).
One image has the value of nvidia-smi to show the total memory usage.

Dashdot-comfyui Dashdot-main Dashdot-nvidiasmi Dashdot-widget

After: Dashdot reports that 0% of the memory is in use when it is closer to 50%

Dashdot-main-after Dashdot-nvidiasmi-after

A few minutes after (while I type this), the numbers are still the same 0% for Dashdot, about 50% for nvidia-smi

I note that ComfyUI is also running as a GPU container.

How to reproduce

compose.yaml file:

services:
  dash:
    image: mauricenino/dashdot:nvidia
    container_name: dashdot-nvidia
    restart: unless-stopped
    privileged: true
    deploy:
      resources:
        reservations:
          devices:
            - capabilities:
                - gpu
    ports:
      - 3001:3001
    volumes:
      - /:/mnt/host:ro
    environment:
      DASHDOT_WIDGET_LIST: os,cpu,storage,ram,network,gpu

Relevant log output

No response

Info output of dashdot cli

INFO
=========
Yarn: 3.7.0
Node: v20.16.0
Dash: 5.8.6

Cwd: /app
Hash: a0e2ee941c37092804cb11acc7252ada877b9772
Platform: Linux 97d6bae71c48 6.8.0-41-generic #41-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug  2 20:41:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Docker image: nvidia
In Docker: true
In Docker (env): true
In Podman: false

What browsers are you seeing the problem on?

Firefox, Chrome

Where is your instance running?

Linux Server, Desktop PC (Linux)

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant