AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam' #1846
Comments
|
Can you share the output of ds_report after your install? Also I recently discovered a potential issue with this pre compile style (see #1840). Can you see if you get the same error installing this way: DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 pip install -e . |
|
Here is the output of ds_report DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at
|
|
Here is the command I've used for installation: TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_CPU_ADAM=1 DS_BUILD_OP_ADAM=1 DS_BUILD_UTILS=1 DS_BUILD_AIO=1 pip install -e. using makes no difference |
|
I think the root cause of the problem is this: ImportError: /media/max/Volume/GPT/finetune/DeepSpeed/deepspeed/ops/adam/cpu_adam_op.cpython-38-x86_64-linux-gnu.so: undefined symbol: curandCreateGenerator |
|
Ohh I see your comments on this issue now as well (pytorch/pytorch#69666). If you try a recent torch nightly build does it still exhibit the issue? |
|
Hmm, I've tried with the torch nightly build but I am getting still the same error message DeepSpeed C++/CUDA extension op reportNOTE: Ops not installed will be just-in-time (JIT) compiled at
|
|
I wonder if there is any hope to get this fixed or a work around. |
|
@maxmaier59, can clarify whether your intention is to use DeepSpeed CPUAdam or torch Adam optimizer? |
|
I am not sure what the difference ist. |
|
@maxmaier59, CPUAdam was created for executing optimizer computations on CPU instead of GPU. Please see this tutorial for more details. |
|
In this case I need CPUAdam |
|
Please can somebody help me to solve this problem? I wonder what is going on. It seems to me that either CPUAdam optimizer for Deepspeed has been abandoned or I am doing something wrong. If the latter is the case, can somebody please help me to find my error to fix the problem? If the first is the case I wonder why the optimizer had been dropped. Is there any alternative? |
|
@maxmaier59, apologies for the delayed response. CPUAdam is still very much an important part of DeepSpeed as our offloading technologies depend on it. I am a bit confused about whether the original issue was observed during build or during an actual run. The issue mentions an attribute error which suggests this occurred during a run, so in that case can you please repaste or point me to the stack trace? Sorry for asking you to provide this again. |
|
Many thanks for getting back to me! deepspeed --num_gpus=2 run_clm.py And here is the output: [2022-03-24 22:33:28,352] [WARNING] [runner.py:155:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. [INFO|configuration_utils.py:648] 2022-03-24 22:33:32,544 >> loading configuration file https://huggingface.co/EleutherAI/gpt-neo-2.7B/resolve/main/config.json from cache at /home/max/.cache/huggingface/transformers/3c80ef2946e1aacc6dd37cb986ea989c29c92775701655bedf14d8791825a30b.f1ede5af01beb85af6cba189a5671dbac3fe256282f737ff0fedf1db882ca729 [INFO|tokenization_utils_base.py:1786] 2022-03-24 22:33:34,931 >> loading file https://huggingface.co/EleutherAI/gpt-neo-2.7B/resolve/main/vocab.json from cache at /home/max/.cache/huggingface/transformers/d4455fdc7c8e2bcf94a0bfe134b748a93c37ecadb7b8f6b0eb508ffdd433a61e.a1b97b074a5ac71fad0544c8abc1b3581803d73832476184bde6cff06a67b6bb [INFO|modeling_utils.py:1431] 2022-03-24 22:33:36,020 >> loading weights file https://huggingface.co/EleutherAI/gpt-neo-2.7B/resolve/main/pytorch_model.bin from cache at /home/max/.cache/huggingface/transformers/0839a11efa893f2a554f8f540f904b0db0e5320a2b1612eb02c3fd25471c189a.a144c17634fa6a7823e398888396dd623e204dce9e33c3175afabfbf24bd8f56 [INFO|modeling_utils.py:1710] 2022-03-24 22:34:17,407 >> All the weights of GPTNeoForCausalLM were initialized from the model checkpoint at EleutherAI/gpt-neo-2.7B. Exception ignored in: <function DeepSpeedCPUAdam.del at 0x7fa491b11a60> |
Thanks! Can you please share the contents of |
|
Here is the ds_config.json { } |
|
BTW, there is a simpler way to reproduce the problem: DS_BUILD_OPS=1 pip install deepspeed And then run this: Error message: Traceback (most recent call last): Can you please let me know what is the official way to build DeepSpeed to be able to run the cpu_adam optimizer? To me this seems fundamentally broken. |
|
Hi @maxmaier59, so sorry you’re running into this issue. One thing I don’t recall if I’ve asked. Can you use JIT to compile cpu Adam successfully? You can try this by installing deepspeed w/o any DS_* variables or via: “DS_BUILD_OPS=0 pip install deepspeed”. After install you can force a build of cpu Adam in a Python shell via: import deepspeed You’ll need ninja installed for this to work, many setups already have this though. More info here: https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages |
|
Hello Jeff, |
|
Excellent, really glad to hear. It still concerns me that the pre-compilation method doesn't work for you but I am glad you are unblocked for now at least. I'll close this issue for now, feel free to re-open if you have further issues along this line. |
|
I am also facing a similar problem and I have detailed about it here: https://discuss.huggingface.co/t/run-translation-py-example-is-erroring-out-with-the-recommended-settings/16432 |
|
#1846 (comment) solved my problem too but I think it's a matter of concern still. |
|
@jeffra, if you remember these 2 interconnected threads:
I am pretty sure that's the cause of the problem for pre-building. If you remember torch @maxmaier59, please check if the problem goes away if you installed torch via conda. |
|
Is there a recent workaround I could refer to in case installing via |
|
JIT build is the workaround if conda is not an option. And the main thread is pytorch/pytorch#69666 For some reason the problem went away for me with could you please post the output of your: |
|
As I've mentioned above building with pip fails as well as TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 python setup.py build_ext -j8 bdist_wheel Here is my environment: Collecting environment information... OS: Ubuntu 20.04.4 LTS (x86_64) Python version: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0] (64-bit runtime) Nvidia driver version: 510.54 Versions of relevant libraries: |
|
OK, I have created a new conda env and I'm able to reproduce the problem: note, I first install no failure reported during the build. |
|
@stas00 mine is: |
|
So glad to hear it finally worked, @maxmaier59! Does it mean that you tried installing the binary wheel and it didn't work? |
|
current main branch 0.8.2+4ae3a3da gives the same error. pip installing 0.8.1 works fine |
Sorry, I also have this problem, I execute this command, the outputs as follow, but l aso have the proble "AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'", so how to fix it ?
|
|
For the record, I did not have this issue on 0.10.0 but when I upgraded to the current main (777ae39), I got the issue as well. Pre-building the cpu adam solved the issue but regardless, it seems an important issue to raise. |
|
hi @BramVanroy how did you Pre-building the cpu adam? |
You find the instructions here https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops |
|
|
|
I'm still facing the same issue. RunTimeError during finetuning:ds_reportWhat I have tried: Stack trace of Jeffra's suggestionAny help will be appreciated please! |
|
clone the git and do python setup.py install |
|
Even the latest version of deepspeed still has this problem ... |
|
I am also struggling with this problem. Summary: Hopeing for help. |
|
Either install deepspeed from source or upgrade to the newest version of
deepspeed, ```pip install --upgrade deepspeed```
…On Fri, Apr 26, 2024 at 7:19 PM Esra Lenz ***@***.***> wrote:
I am also struggling with this problem.
I installed Pytorch via pip and via conda because if I install via conda
it doesn't recognize my cuda.
As I understood so far the problem is not there if you are installing with
conda, but this is not working because than it doesn't recognize my Cuda.
I don't get the prebuilt solutions from above.
Can someone give a step by step instructions to solve this problem in an
easy way with commands I should run?
I also have no sudo rights so installing cudatoolkit should be working
with cuda (or with pip if this exists).
Summary:
Just pip installation for Pytorch (pip3 install torch torchvision
torchaudio)
cudatoolkit installation only with cuda or pip possible (no sudo rights)
Hopeing for help.
—
Reply to this email directly, view it on GitHub
<#1846 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A3WU7BYZIPRMT4E6DH5YSIDY7LVIRAVCNFSM5RDB6W42U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBYGAZDMMRUG42A>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
I face the same problem [AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'] |
|
Just had the same problem: I managed to fix it by installing |
|
Hi @yannikkellerde, @Alpha-Girl, @42elenz, @rangehow, Can you please try uninstalling deepspeed: If deepspeed was installed from source, also clean the repo: Then install w/ From source: |
|
Hi @maxmaier59, Can you please try installing the latest DeepSpeed from source since #5780 has been merged addressing this issue. Thanks, |
hello jeff! I follow your advice but I find a new mistake: The above exception was the direct cause of the following exception: Traceback (most recent call last): |
|
I’ve solved the problem here |
|
this solved: DS_BUILD_OPS=0 DS_BUILD_CPU_ADAM=1 pip install deepspeed --no-cache |
|
First you wanna check if cpu_adam is compatible with your (driver version + cuda version) pair. You can do this using
I knew that it was compatible, so I stopped fixing compatibility issues and focused on other parts of the error logs. Then I noticed a missing which fixed the problem for me |
When I try to do finetuning with Deepspeed I get the following error message:
Traceback (most recent call last):
File "/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/deepspeed/ops/adam/cpu_adam.py", line 97, in del
self.ds_opt_adam.destroy_adam(self.opt_id)
AttributeError: 'DeepSpeedCPUAdam' object has no attribute 'ds_opt_adam'
I have built Deepspeed with
git clone https://github.com/microsoft/DeepSpeed
cd DeepSpeed
DS_BUILD_CPU_ADAM=1 DS_BUILD_AIO=1 DS_BUILD_UTILS=1 pip install -e . --global-option="build_ext" --global-option="-j8" --no-cache -v --disable-pip-version-check
It seems that ds_opt_adam was not built
This is the output I've got:
/media/max/Volume/GPT/finetune/DeepSpeed
Using pip 21.2.4 from /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/pip (python 3.8)
Obtaining file:///media/max/Volume/GPT/finetune/DeepSpeed
/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/pip/_internal/commands/install.py:229: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
cmdoptions.check_install_build_global(options)
Running command python setup.py egg_info
DS_BUILD_OPS=0
Installed CUDA version 11.4 does not match the version torch was compiled with 11.5 but since the APIs are compatible, accepting this combination
Install Ops={'cpu_adam': 1, 'cpu_adagrad': False, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': 1, 'utils': 1, 'quantizer': False, 'transformer_inference': False}
version=0.6.0+a32e9b33, git_hash=a32e9b33, git_branch=HEAD
install_requires=['hjson', 'ninja', 'numpy', 'packaging', 'psutil', 'py-cpuinfo', 'torch', 'tqdm', 'triton==1.0.0']
compatible_ops={'cpu_adam': True, 'cpu_adagrad': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': True, 'transformer': True, 'stochastic_transformer': True, 'async_io': True, 'utils': True, 'quantizer': True, 'transformer_inference': True}
ext_modules=[<setuptools.extension.Extension('deepspeed.ops.adam.cpu_adam_op') at 0x7f2b7bd0e820>, <setuptools.extension.Extension('deepspeed.ops.aio.async_io_op') at 0x7f2b7bbdd790>, <setuptools.extension.Extension('deepspeed.ops.utils_op') at 0x7f2b7bb5ff70>]
running egg_info
creating /tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info
writing /tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/dependency_links.txt
writing entry points to /tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/entry_points.txt
writing requirements to /tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/requires.txt
writing top-level names to /tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '.hip' under directory 'deepspeed'
warning: no files found matching '.cc' under directory 'deepspeed'
warning: no files found matching '.tr' under directory 'csrc'
warning: no files found matching '.cc' under directory 'csrc'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-pip-egg-info-vqyrd9dj/deepspeed.egg-info/SOURCES.txt'
deepspeed build time = 0.36443185806274414 secs
Requirement already satisfied: hjson in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (3.0.2)
Requirement already satisfied: ninja in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (1.10.2.3)
Requirement already satisfied: numpy in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (1.22.3)
Requirement already satisfied: packaging in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (21.3)
Requirement already satisfied: psutil in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (5.9.0)
Requirement already satisfied: py-cpuinfo in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (8.0.0)
Requirement already satisfied: torch in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (1.11.0+cu115)
Requirement already satisfied: tqdm in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (4.63.0)
Requirement already satisfied: triton==1.0.0 in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from deepspeed==0.6.0+a32e9b33) (1.0.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from packaging->deepspeed==0.6.0+a32e9b33) (3.0.4)
Requirement already satisfied: typing-extensions in /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages (from torch->deepspeed==0.6.0+a32e9b33) (3.10.0.2)
Installing collected packages: deepspeed
Attempting uninstall: deepspeed
Found existing installation: deepspeed 0.5.9+d0ab7224
Uninstalling deepspeed-0.5.9+d0ab7224:
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/bin/deepspeed
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/bin/deepspeed.pt
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/bin/ds
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/bin/ds_elastic
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/bin/ds_report
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/bin/ds_ssh
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/deepspeed-0.5.9+d0ab7224-py3.8.egg-info
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/deepspeed/
Removing file or directory /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/op_builder/
Successfully uninstalled deepspeed-0.5.9+d0ab7224
Running setup.py develop for deepspeed
Running command /home/max/anaconda3/envs/gptneo_finetuned/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/media/max/Volume/GPT/finetune/DeepSpeed/setup.py'"'"'; file='"'"'/media/max/Volume/GPT/finetune/DeepSpeed/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' build_ext -j8 develop --no-deps
DS_BUILD_OPS=0
Installed CUDA version 11.4 does not match the version torch was compiled with 11.5 but since the APIs are compatible, accepting this combination
Install Ops={'cpu_adam': 1, 'cpu_adagrad': False, 'fused_adam': False, 'fused_lamb': False, 'sparse_attn': False, 'transformer': False, 'stochastic_transformer': False, 'async_io': 1, 'utils': 1, 'quantizer': False, 'transformer_inference': False}
version=0.6.0+a32e9b33, git_hash=a32e9b33, git_branch=HEAD
install_requires=['hjson', 'ninja', 'numpy', 'packaging', 'psutil', 'py-cpuinfo', 'torch', 'tqdm', 'triton==1.0.0']
compatible_ops={'cpu_adam': True, 'cpu_adagrad': True, 'fused_adam': True, 'fused_lamb': True, 'sparse_attn': True, 'transformer': True, 'stochastic_transformer': True, 'async_io': True, 'utils': True, 'quantizer': True, 'transformer_inference': True}
ext_modules=[<setuptools.extension.Extension('deepspeed.ops.adam.cpu_adam_op') at 0x7f41e6e48f10>, <setuptools.extension.Extension('deepspeed.ops.aio.async_io_op') at 0x7f41e6214790>, <setuptools.extension.Extension('deepspeed.ops.utils_op') at 0x7f41e6193f40>]
running build_ext
building 'deepspeed.ops.adam.cpu_adam_op' extension
building 'deepspeed.ops.aio.async_io_op' extension
creating build
creating build/temp.linux-x86_64-3.8
building 'deepspeed.ops.utils_op' extension
creating build/temp.linux-x86_64-3.8/csrc
creating build/temp.linux-x86_64-3.8/csrc
creating build/temp.linux-x86_64-3.8/csrc/adam
creating build/temp.linux-x86_64-3.8/csrc/utils
creating build/temp.linux-x86_64-3.8/csrc/aio
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/utils/flatten_unflatten.cpp -o build/temp.linux-x86_64-3.8/csrc/utils/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=utils_op -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
creating build/temp.linux-x86_64-3.8/csrc/aio/py_lib
creating build/temp.linux-x86_64-3.8/csrc/common
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/includes -I/home/max/anaconda3/envs/gptneo_finetuned/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-3.8/csrc/adam/cpu_adam.o -O3 -std=c++14 -g -Wno-reorder -L/home/max/anaconda3/envs/gptneo_finetuned/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
creating build/temp.linux-x86_64-3.8/csrc/aio/common
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_copy.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256 -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
In file included from csrc/includes/cpu_adam.h:12,
from csrc/adam/cpu_adam.cpp:1:
csrc/includes/simd.h:63: warning: ignoring #pragma unroll [-Wunknown-pragmas]
63 | #pragma unroll
|
csrc/includes/simd.h:71: warning: ignoring #pragma unroll [-Wunknown-pragmas]
71 | #pragma unroll
|
csrc/includes/simd.h:79: warning: ignoring #pragma unroll [-Wunknown-pragmas]
79 | #pragma unroll
|
csrc/includes/simd.h:87: warning: ignoring #pragma unroll [-Wunknown-pragmas]
87 | #pragma unroll
|
csrc/includes/simd.h:95: warning: ignoring #pragma unroll [-Wunknown-pragmas]
95 | #pragma unroll
|
csrc/includes/simd.h:103: warning: ignoring #pragma unroll [-Wunknown-pragmas]
103 | #pragma unroll
|
csrc/includes/simd.h:109: warning: ignoring #pragma unroll [-Wunknown-pragmas]
109 | #pragma unroll
|
csrc/includes/simd.h:115: warning: ignoring #pragma unroll [-Wunknown-pragmas]
115 | #pragma unroll
|
csrc/includes/simd.h:121: warning: ignoring #pragma unroll [-Wunknown-pragmas]
121 | #pragma unroll
|
csrc/includes/simd.h:127: warning: ignoring #pragma unroll [-Wunknown-pragmas]
127 | #pragma unroll
|
csrc/includes/simd.h:133: warning: ignoring #pragma unroll [-Wunknown-pragmas]
133 | #pragma unroll
|
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/py_lib/py_ds_aio.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/py_ds_aio.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256__ -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/deepspeed
creating build/lib.linux-x86_64-3.8/deepspeed/ops
g++ -pthread -shared -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -L/home/max/anaconda3/envs/gptneo_finetuned/lib -Wl,-rpath=/home/max/anaconda3/envs/gptneo_finetuned/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/csrc/utils/flatten_unflatten.o -L/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.8/deepspeed/ops/utils_op.cpython-38-x86_64-linux-gnu.so
csrc/adam/cpu_adam.cpp: In member function ‘void Adam_Optimizer::Step_1(float*, float*, float*, float*, size_t, half*, bool)’:
csrc/adam/cpu_adam.cpp:45:17: warning: ‘params_cast_h’ may be used uninitialized in this function [-Wmaybe-uninitialized]
45 | half* params_cast_h;
| ^~~~~~~~~~~~~
csrc/adam/cpu_adam.cpp:44:17: warning: ‘grads_cast_h’ may be used uninitialized in this function [-Wmaybe-uninitialized]
44 | half* grads_cast_h;
| ^~~~~~~~~~~~
/home/max/anaconda3/envs/gptneo_finetuned/bin/nvcc -Icsrc/includes -I/home/max/anaconda3/envs/gptneo_finetuned/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/common/custom_cuda_kernel.cu -o build/temp.linux-x86_64-3.8/csrc/common/custom_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 --use_fast_math -std=c++14 -U__CUDA_NO_HALF_OPERATORS_ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
creating build/lib.linux-x86_64-3.8/deepspeed/ops/adam
g++ -pthread -shared -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -L/home/max/anaconda3/envs/gptneo_finetuned/lib -Wl,-rpath=/home/max/anaconda3/envs/gptneo_finetuned/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/csrc/adam/cpu_adam.o build/temp.linux-x86_64-3.8/csrc/common/custom_cuda_kernel.o -L/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/lib -L/home/max/anaconda3/envs/gptneo_finetuned/lib64 -lc10 -ltorch -ltorch_cpu -ltorch_python -lcudart -lc10_cuda -ltorch_cuda_cu -ltorch_cuda_cpp -o build/lib.linux-x86_64-3.8/deepspeed/ops/adam/cpu_adam_op.cpython-38-x86_64-linux-gnu.so
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_aio.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_aio.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256 -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/py_lib/deepspeed_py_aio_handle.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_aio_handle.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256 -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/py_lib/deepspeed_aio_thread.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_aio_thread.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256 -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/common/deepspeed_aio_utils.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/common/deepspeed_aio_utils.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256__ -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/common/deepspeed_aio_common.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/common/deepspeed_aio_common.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256 -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
csrc/aio/common/deepspeed_aio_common.cpp: In function ‘void _do_io_submit_singles(long long int, long long int, std::unique_ptr<aio_context>&, std::vector<std::chrono::duration >&)’:
csrc/aio/common/deepspeed_aio_common.cpp:76:20: warning: unused variable ‘submit_ret’ [-Wunused-variable]
76 | const auto submit_ret = io_submit(aio_ctxt->_io_ctxt, 1, aio_ctxt->_iocbs.data() + i);
| ^~~~~~~~~~
csrc/aio/common/deepspeed_aio_common.cpp: In function ‘void _do_io_submit_block(long long int, long long int, std::unique_ptr<aio_context>&, std::vector<std::chrono::duration >&)’:
csrc/aio/common/deepspeed_aio_common.cpp:96:16: warning: unused variable ‘submit_ret’ [-Wunused-variable]
96 | const auto submit_ret = io_submit(aio_ctxt->_io_ctxt, n_iocbs, aio_ctxt->iocbs.data());
| ^~~~~~~~~~
csrc/aio/common/deepspeed_aio_common.cpp: In function ‘int regular_read(const char*, std::vector&)’:
csrc/aio/common/deepspeed_aio_common.cpp:280:16: warning: unused variable ‘f_size’ [-Wunused-variable]
280 | const auto f_size = get_file_size(filename, num_bytes);
| ^~~~~~
csrc/aio/common/deepspeed_aio_common.cpp: In function ‘bool validate_buffer(const char*, void*, long long int)’:
csrc/aio/common/deepspeed_aio_common.cpp:307:16: warning: unused variable ‘reg_ret’ [-Wunused-variable]
307 | const auto reg_ret = regular_read(filename, regular_buffer);
| ^~~~~~~
gcc -pthread -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc/aio/py_lib -Icsrc/aio/common -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/TH -I/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/include/THC -I/home/max/anaconda3/envs/gptneo_finetuned/include/python3.8 -c csrc/aio/common/deepspeed_aio_types.cpp -o build/temp.linux-x86_64-3.8/csrc/aio/common/deepspeed_aio_types.o -g -Wall -O0 -std=c++14 -shared -fPIC -Wno-reorder -march=native -fopenmp -D__AVX256 -laio -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=async_io_op -D_GLIBCXX_USE_CXX11_ABI=0
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
creating build/lib.linux-x86_64-3.8/deepspeed/ops/aio
g++ -pthread -shared -B /home/max/anaconda3/envs/gptneo_finetuned/compiler_compat -L/home/max/anaconda3/envs/gptneo_finetuned/lib -Wl,-rpath=/home/max/anaconda3/envs/gptneo_finetuned/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_copy.o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/py_ds_aio.o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_aio.o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_py_aio_handle.o build/temp.linux-x86_64-3.8/csrc/aio/py_lib/deepspeed_aio_thread.o build/temp.linux-x86_64-3.8/csrc/aio/common/deepspeed_aio_utils.o build/temp.linux-x86_64-3.8/csrc/aio/common/deepspeed_aio_common.o build/temp.linux-x86_64-3.8/csrc/aio/common/deepspeed_aio_types.o -L/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-3.8/deepspeed/ops/aio/async_io_op.cpython-38-x86_64-linux-gnu.so -laio
running develop
running egg_info
creating deepspeed.egg-info
writing deepspeed.egg-info/PKG-INFO
writing dependency_links to deepspeed.egg-info/dependency_links.txt
writing entry points to deepspeed.egg-info/entry_points.txt
writing requirements to deepspeed.egg-info/requires.txt
writing top-level names to deepspeed.egg-info/top_level.txt
writing manifest file 'deepspeed.egg-info/SOURCES.txt'
reading manifest file 'deepspeed.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/utils/cpp_extension.py:788: UserWarning: The detected CUDA version (11.4) has a minor version mismatch with the version that was used to compile PyTorch (11.5). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
warning: no files found matching '.hip' under directory 'deepspeed'
warning: no files found matching '.cc' under directory 'deepspeed'
warning: no files found matching '.tr' under directory 'csrc'
warning: no files found matching '.cc' under directory 'csrc'
adding license file 'LICENSE'
writing manifest file 'deepspeed.egg-info/SOURCES.txt'
running build_ext
copying build/lib.linux-x86_64-3.8/deepspeed/ops/adam/cpu_adam_op.cpython-38-x86_64-linux-gnu.so -> deepspeed/ops/adam
copying build/lib.linux-x86_64-3.8/deepspeed/ops/aio/async_io_op.cpython-38-x86_64-linux-gnu.so -> deepspeed/ops/aio
copying build/lib.linux-x86_64-3.8/deepspeed/ops/utils_op.cpython-38-x86_64-linux-gnu.so -> deepspeed/ops
Creating /home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/deepspeed.egg-link (link to .)
Adding deepspeed 0.6.0+a32e9b33 to easy-install.pth file
Installing deepspeed script to /home/max/anaconda3/envs/gptneo_finetuned/bin
Installing deepspeed.pt script to /home/max/anaconda3/envs/gptneo_finetuned/bin
Installing ds script to /home/max/anaconda3/envs/gptneo_finetuned/bin
Installing ds_ssh script to /home/max/anaconda3/envs/gptneo_finetuned/bin
Installing ds_report script to /home/max/anaconda3/envs/gptneo_finetuned/bin
Installing ds_elastic script to /home/max/anaconda3/envs/gptneo_finetuned/bin
Installed /media/max/Volume/GPT/finetune/DeepSpeed
/home/max/anaconda3/envs/gptneo_finetuned/lib/python3.8/site-packages/torch/utils/cpp_extension.py:788: UserWarning: The detected CUDA version (11.4) has a minor version mismatch with the version that was used to compile PyTorch (11.5). Most likely this shouldn't be a problem.
warnings.warn(CUDA_MISMATCH_WARN.format(cuda_str_version, torch.version.cuda))
deepspeed build time = 90.15858387947083 secs
The text was updated successfully, but these errors were encountered: