-
Notifications
You must be signed in to change notification settings - Fork 831
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extra python dependencies #942
Extra python dependencies #942
Conversation
/uncc @gipster |
/assign @axsaucedo @cliveseldon |
/cc @jklaise @cliveseldon |
@jklaise @cliveseldon @axsaucedo This solution to make the |
Perhaps the changes in the JX file had this effect? You could test by reverting the java piece, but perhaps it's the server |
@axsaucedo I think that one doesn't affect the tests. I've seen the binding error on the Java logs before. I haven't digged into it, but it doesn't make the tests fail. In this case it failed because an error on the python tests: _____________________ ERROR collecting tests/test_utils.py _____________________
Traceback (most recent call last):
File "/workspace/source/python/seldon_core/proto/tensorflow/core/framework/__init__.py", line 8, in <module>
from tensorflow.core.framework import (
ModuleNotFoundError: No module named 'tensorflow'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspace/source/python/tests/test_utils.py", line 6, in <module>
import seldon_core.utils as scu
File "/workspace/source/python/seldon_core/utils.py", line 11, in <module>
from seldon_core.proto import prediction_pb2
File "/workspace/source/python/seldon_core/proto/prediction_pb2.py", line 17, in <module>
from seldon_core.proto.tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
File "/workspace/source/python/seldon_core/proto/tensorflow/core/framework/__init__.py", line 18, in <module>
logger.notice(notice)
AttributeError: 'Logger' object has no attribute 'notice' I was using It's been a really good idea to make Jenkins X test the build path without |
@axsaucedo @cliveseldon the tests are now added and fixed. I had to change how I was checking if I've also removed a few no-longer used lines on the |
Looks good from my side. @cliveseldon are we good for landing? |
/lgtm |
/lgtm |
/hold |
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: axsaucedo, cliveseldon, jklaise The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/hold remove |
This PR partly addresses #564
Changes
tensorflow
is now an extra package, which can be installed as$ pip install seldon_core[tensorflow]
tensorflow
is not present, thetftensor
data type won't be usable. This is because thetftensor
type still requires a couple oftensorflow
methods:tf.make_ndarray()
andtf.make_tensor_proto()
.seldon_core.proto.prediction_pb2
protobuf still needs the definitions for thetensorflow
tensor protobuffs:seldon-core/python/seldon_core/proto/prediction_pb2.py
Lines 16 to 17 in 68c2596
As a workaround, if
tensorflow
is present, then we don't do anything and we just use the definitions fromtensorflow.core.framework.tensor_pb2
. However, iftensorflow
is not present, we patch that import path onseldon_core/proto/__init__.py
so that it points to our own compiledtensorflow.core.framework.tensor_pb2
protobuffs. Patchingsyspath
is not very clean and can cause problems down the line. I'm still not 100% sure it's a good choice to do this. Suggestions are more than welcome.Makefile
now compiles thetensorflow.core.framework.tensor_pb2
protobuffs in case we need to patch the import path.Pillow
has also been moved to the test dependencies, since it's only used on the tests and it's a pretty heavy dependency.Notes
As mentioned above, the solution we have implemented is patching
syspath
to point to our own set of pre-compiledtensorflow.core.framework.tensor_pb2
protobuffs whentensorflow
is not present. This is required because our ownseldon_core.proto.prediction_pb2
has a hard dependency on them.A different approach we tried was to keep our set of pre-compiled
tensorflow
protobuffs on a different package which doesn't shadow thetensorflow.core.framework.tensor_pb2
import path (e.g. a newtf_proto
package). However, iftensorflow
is present, it will also try to import its own set of protobuffs and these will then conflict since they have the same ID.Renaming the package of our pre-compiled
tensorflow.core.framework.tensor_pb2
protobuffs to avoid this conflict doesn't help either, since in that case thetf.make_ndarray()
andtf.make_tensor_proto()
methods stop working because the types are different.Before moving forward, we should consider the pros and cons of overriding
syspath
to maketensorflow
an optional dependency and perhaps evaluate different alternatives (like writing our ownmake_ndarray
andmake_tensor_proto
methods and getting rid of thetensorflow
dep altogether).