You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have previously developed JAX on cloud TPU without issues, about a week or two ago, using the same exact startup script as I had previously, performance has degraded to being slower than if I run the same code locally on CPU. It may or may not be related but I started getting this warning when running a basic flax mnist example (or anything for that matter).
(.venv310) will@xxxxxxxxxxx:~/flax/examples/mnist$ python3 main.py --workdir=/tmp/mnist \
> --config=configs/default.py \
> --config.learning_rate=0.05 \
> --config.num_epochs=5
2023-01-31 17:03:51.011383: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/lib
2023-01-31 17:03:51.641403: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/lib
2023-01-31 17:03:51.641518: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: :/usr/local/lib
2023-01-31 17:03:51.641529: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-01-31 17:18:22.501774: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303)
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I have previously developed JAX on cloud TPU without issues, about a week or two ago, using the same exact startup script as I had previously, performance has degraded to being slower than if I run the same code locally on CPU. It may or may not be related but I started getting this warning when running a basic flax mnist example (or anything for that matter).
Creating the TPU:
setup.sh
Any help would be greatly appreciated!
Beta Was this translation helpful? Give feedback.
All reactions