python: Relink error in GPU image

[RafalSkolasinski]

It seems that there is a problem with GPU image that in certain scenarios lead to the following error.

$ docker run --rm -it seldonio/seldon-core-s2i-python3-tf-gpu:0.18 \
   python -c 'import seldon_core.proto.tensorflow.core.framework; import grpc' 
python: Relink `/usr/local/lib/python3.6/dist-packages/grpc/_cython/cygrpc.cpython-36m-x86_64-linux-gnu.so' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'

$ docker run --rm -it seldonio/seldon-core-s2i-python3-tf-gpu:1.2.0 \
   python -c 'import seldon_core.proto.tensorflow.core.framework; import grpc'      (kind-kind/seldon)
python: Relink `/usr/local/lib/python3.6/dist-packages/grpc/_cython/cygrpc.cpython-36m-x86_64-linux-gnu.so' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'

Problem goes away if one either

• remove tensorflow-gpu with pip uninstall tensorflow-gpu
• add tensorflow, e.g. pip install tensorflow==1.13.1 (or other tensorflow-gpu matching version)

Problem is also solved if conda is installed and used as a base Python.
This means the issue will be solved once #1972 is merged.

As it seems to be of interest to have GPU image without tensorflow for some use cases we will probably for 1.3 have a GPU conda-based image without the tensorflow / tensorflow-gpu installed.

[RafalSkolasinski]

Side note: The import seldon_core.proto.tensorflow.core.framework; import grpc import order is just the easiest way to reproduce the problem.

This problem when container starts seldon-core-microservice <- it took me some time to go through the imports order and find what was the root of the problem.

[RafalSkolasinski]

Solved by #1972