OpenCL - ViennaCL: FATAL ERROR: CL_INVALID_WORK_GROUP_SIZE

[fragolka]

Hi,

I've build the caffe libray using following configuration:

cmake -DUSE_GREENTEA=ON -DUSE_INTEL_SPATIAL=OFF -DUSE_ISAAC=OFF -DUSE_CLBLAS=ON -DBUILD_SHARED_LIBS=ON -DUSE_CUDNN=OFF -DBUILD_docs=OFF -DBUILD_python=ON -DBUILD_matlab=OFF -DBLAS=open ..

I'm using OpenCL 1.2, viennacl-dev 1.7.1, clBLAS 2.12, openBLAS 0.2.20.

Running the classification example, I got following error:

./build/examples/cpp_classification/classification.bin \

models/bvlc_reference_caffenet/deploy.prototxt
models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel
data/ilsvrc12/imagenet_mean.binaryproto
data/ilsvrc12/synset_words.txt
examples/images/cat.jpg
Use GPU with device ID 0
---------- Prediction for examples/images/cat.jpg ----------
ViennaCL: FATAL ERROR: Kernel start failed for 'conv_forward'.
terminate called after throwing an instance of 'viennacl::ocl::invalid_work_group_size'
what(): ViennaCL: FATAL ERROR: CL_INVALID_WORK_GROUP_SIZE
The supplied work group size is invalid. If you have set this value manually, please reconsider your choice.
If you think that this is a bug in ViennaCL, please report it at [email protected] and supply at least the following information:

• Operating System
• Which OpenCL implementation (AMD, NVIDIA, etc.)
• ViennaCL version
  Many thanks in advance!
  Aborted (core dumped)

I get the same error running cifar10 example:
...
I0215 07:16:11.737159 29837 net.cpp:280] Network initialization done.
I0215 07:16:11.737175 29837 net.cpp:281] Memory required for data: 31987608
I0215 07:16:11.737361 29837 solver.cpp:65] Solver scaffolding done.
I0215 07:16:11.737481 29837 caffe.cpp:272] Starting Optimization
I0215 07:16:11.737502 29837 solver.cpp:295] Solving CIFAR10_quick
I0215 07:16:11.737519 29837 solver.cpp:296] Learning Rate Policy: fixed
I0215 07:16:11.748381 29837 solver.cpp:354] Iteration 0, Testing net (#0)
ViennaCL: FATAL ERROR: Kernel start failed for 'conv_forward'.
terminate called after throwing an instance of 'viennacl::ocl::invalid_work_group_size'
what(): ViennaCL: FATAL ERROR: CL_INVALID_WORK_GROUP_SIZE
The supplied work group size is invalid. If you have set this value manually, please reconsider your choice.
If you think that this is a bug in ViennaCL, please report it at [email protected] and supply at least the following information:

• Operating System
• Which OpenCL implementation (AMD, NVIDIA, etc.)
• ViennaCL version
  Many thanks in advance!
  *** Aborted at 1518678971 (unix time) try "date -d @1518678971" if you are using GNU date ***
  PC: @ 0x0 (unknown)
  *** SIGABRT (@0x748d) received by PID 29837 (TID 0xffff73f34000) from PID 29837; stack trace: ***
  @ 0xffff7b3e06c0 ([vdso]+0x6bf)
  @ 0xffff77b579b0 gsignal
  ./examples/cifar10/train_quick.sh: line 7: 29837 Aborted (core dumped) $TOOLS/caffe train --solver=examples/cifar10/cifar10_quick_solver.prototxt $@

Any idea where is the problem? Many thanks. Lucy

[naibaf7]

@fragolka Please post the result of the following command: clinfo and ./build/tools/caffe device_query

[fragolka]

Sure:

clinfo
Platform #0
Name: Vivante OpenCL Platform
Version: OpenCL 1.2 V6.2.4.138003

Device #0
Name: Vivante OpenCL Device GC7000XSVX.6009.0000
Type: GPU
Version: OpenCL 1.2
Global memory size: 256 MB
Local memory size: 32 kB
Max work group size: 1024
Max work item sizes: (1024, 1024, 1024)

Device #1
Name: Vivante OpenCL Device GC7000XSVX.6009.0000
Type: GPU
Version: OpenCL 1.2
Global memory size: 256 MB
Local memory size: 32 kB
Max work group size: 1024
Max work item sizes: (1024, 1024, 1024)

./build/tools/caffe device_query
I0216 10:04:33.404959 31588 common.cpp:438] Total devices: 2
I0216 10:04:33.405869 31588 common.cpp:439] CUDA devices: 0
I0216 10:04:33.405902 31588 common.cpp:440] OpenCL devices: 2
I0216 10:04:33.405926 31588 common.cpp:464] Device id: 0
I0216 10:04:33.405949 31588 common.cpp:466] Device backend: OpenCL
I0216 10:04:33.405972 31588 common.cpp:468] Backend details: Vivante Corporation: OpenCL 1.2 V6.2.4.138003
I0216 10:04:33.406059 31588 common.cpp:470] Device vendor: Vivante Corporation
I0216 10:04:33.406093 31588 common.cpp:472] Name: Vivante OpenCL Device GC7000XSVX.6009.0000
I0216 10:04:33.406119 31588 common.cpp:474] Total global memory: 268435456
I0216 10:04:33.406168 31588 common.cpp:464] Device id: 1
I0216 10:04:33.406203 31588 common.cpp:466] Device backend: OpenCL
I0216 10:04:33.406236 31588 common.cpp:468] Backend details: Vivante Corporation: OpenCL 1.2 V6.2.4.138003
I0216 10:04:33.406278 31588 common.cpp:470] Device vendor: Vivante Corporation
I0216 10:04:33.406316 31588 common.cpp:472] Name: Vivante OpenCL Device GC7000XSVX.6009.0000
I0216 10:04:33.406350 31588 common.cpp:474] Total global memory: 268435456

[naibaf7]

@fragolka Ok that just won't work. Vivante's OpenCL implementation is quite bad and I can't fix any bugs with their GPUs since I don't have one of them myself. Sorry. If you want to get this fixed you'd either have to do it yourself and submit a pull request (PR) to OpenCL Caffe or provide me with a Vivante GPU.

[fragolka]

Thank you for your answer. Is there a way how to get kernel source string that is created?

[naibaf7]

@fragolka You can read the kernels from here:
https://github.com/BVLC/caffe/tree/opencl/src/caffe/greentea/cl_kernels
and the header files you need to use here:
https://github.com/BVLC/caffe/tree/opencl/src/caffe/greentea/cl_headers
(either 32 bit or 64 bit plus header.cl need to be added to the sources in order to compile them)

[fragolka]

I've found the problem - local work group size depends on CL_KERNEL_WORK_GROUP_SIZE that can be retrieved from clGetKernelWorkGroupInfo function. In my case, CL_KERNEL_WORK_GROUP_SIZE is 224. So after changing the local work group size to 8 it worked.

One more thing.... not sure where it comes from but i had a "function 'atomicAdd' hasn't the corresponding declaration" error. I solved that by deleting the ifelse statement here

[naibaf7]

@fragolka
I'll have a fix for this issue implemented in the next big OpenCL Caffe release. Thanks.
I'll close the issue when it is merged.

[soulslicer]

Can you tell me what you change

[fabiosammy]

@soulslicer I know it's old, but use the fe2a1102cefafb968904f6bce0a2b47a87112fc6 commit, i can build and run the caffe opencl version.

But now i have an issue in #6711