Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: - All VRAM of first device gets allocated when importing the lib on a gpu environment. #143

Closed
Foglia-m opened this issue Oct 26, 2023 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@Foglia-m
Copy link

Module

None

Current Behavior

When running import xplique on a gpu enabled environment, the vram of the first device is allocated at maximum (32go / 32go)

All modules from the init are concerned except from . import plots

Expected Behavior

I would expect lazy allocatino of the ram only when I load models. (is it due to the recent addition of the torch wrapper?)

Version

Version: 1.3.0

Environment

- OS: linux 
- Python version: 3.9.16
- Tensorflow version: 2.8.2

Relevant log output

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.13    Driver Version: 525.60.13    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100S-PCI...  On   | 00000000:00:07.0 Off |                    0 |
| N/A   33C    P0    38W / 250W |  31452MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100S-PCI...  On   | 00000000:00:08.0 Off |                    0 |
| N/A   33C    P0    41W / 250W |    474MiB / 32768MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

To Reproduce

import xplique in a python console / notebook

@Foglia-m Foglia-m added the bug Something isn't working label Oct 26, 2023
@AntoninPoche
Copy link
Collaborator

AntoninPoche commented Nov 9, 2023

Hello @Foglia-m, thank you for your contribution. We are going to investigate this ASAP, but a quick fix may be:

import tensorflow as tf
gpus = tf.config.list_physical_devices('GPU')
for gpu in gpus:
  tf.config.experimental.set_memory_growth(gpu, True)

import xplique

Can you try this and tell us if it helps!

Can you also tell if you had this issue on previous Xplique versions?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants