Skip to content

Commit

Permalink
Fix tiff lib and remove stub ffmpeg libraries for macOS support
Browse files Browse the repository at this point in the history
The video ops used to be linked to a stub (empty) library
so that in compile time it is possible to build tensorflow-io
without actually providing ffmpeg dynamic library.
This causes issues in bazel's sandboxed mode because sandboxed
one need to know all files in advance. So far we have been
using standalone mode to address this issue. It is still
desirable to use sandbox mode as there might be some unintended
effects. For example, when we compile tiff we assume
jpeg is not supported, but in Linux it is almost certain it will
include a header from system library. Which is one of the
reasons causeing MacOS to fail (include path different from linux).

In short, we should use sandboxed mode for Bazel.

This PR fixes the TIFF issue, and also uses dlopen(RTLD_GLOBAL)
to resolve ffmpeg symbols in run time, effectively avoids
the stub (empty) libary we have.

This PR is part of the effort to spport macOS for tensorflow-io.

Signed-off-by: Yong Tang <[email protected]>
  • Loading branch information
yongtang committed Feb 25, 2019
1 parent 4b3e018 commit 8a4a895
Show file tree
Hide file tree
Showing 16 changed files with 10 additions and 42 deletions.
1 change: 0 additions & 1 deletion .travis/build.test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ fi
bazel build \
--noshow_progress \
--noshow_loading_progress \
--spawn_strategy standalone \
--verbose_failures \
--test_output=errors -- \
//tensorflow_io/...
Expand Down
1 change: 0 additions & 1 deletion .travis/python.release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@ fi
bazel build \
--noshow_progress \
--noshow_loading_progress \
--spawn_strategy standalone \
--verbose_failures \
--test_output=errors -- \
//tensorflow_io/...
Expand Down
43 changes: 6 additions & 37 deletions tensorflow_io/video/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,6 @@ licenses(["notice"]) # Apache 2.0

package(default_visibility = ["//visibility:public"])

# Note: stub files are generated from Ubuntu 14.0 with
# `echo '' | g++ -shared -fPIC -x c++ - -o $@`
filegroup(
name = "stub.so",
srcs = glob(["kernel/stub/*"]),
)

cc_binary(
name = 'python/ops/_video_ops_ffmpeg_3.4.so',
srcs = [
Expand All @@ -19,21 +12,13 @@ cc_binary(
],
includes = ["."],
linkshared = 1,
linkopts = [
"-Ltensorflow_io/video/kernels/stub",
"-l:libavformat.so.57",
"-l:libavcodec.so.57",
"-l:libavutil.so.55",
"-l:libswscale.so.4",
],
linkopts = [],
deps = [
"@local_config_tf//:libtensorflow_framework",
"@local_config_tf//:tf_header_lib",
"@ffmpeg_3_4//:ffmpeg",
],
data = [
":stub.so",
],
data = [],
copts = ["-pthread", "-std=c++11", "-D_GLIBCXX_USE_CXX11_ABI=0", "-DNDEBUG"]
)

Expand All @@ -47,21 +32,13 @@ cc_binary(
],
includes = ["."],
linkshared = 1,
linkopts = [
"-Ltensorflow_io/video/kernels/stub",
"-l:libavformat-ffmpeg.so.56",
"-l:libavcodec-ffmpeg.so.56",
"-l:libavutil-ffmpeg.so.54",
"-l:libswscale-ffmpeg.so.3",
],
linkopts = [],
deps = [
"@local_config_tf//:libtensorflow_framework",
"@local_config_tf//:tf_header_lib",
"@ffmpeg_2_8//:ffmpeg",
],
data = [
":stub.so",
],
data = [],
copts = ["-pthread", "-std=c++11", "-D_GLIBCXX_USE_CXX11_ABI=0", "-DNDEBUG"]
)

Expand All @@ -75,20 +52,12 @@ cc_binary(
],
includes = ["."],
linkshared = 1,
linkopts = [
"-Ltensorflow_io/video/kernels/stub",
"-l:libavformat.so.54",
"-l:libavcodec.so.54",
"-l:libavutil.so.52",
"-l:libswscale.so.2",
],
linkopts = [],
deps = [
"@local_config_tf//:libtensorflow_framework",
"@local_config_tf//:tf_header_lib",
"@libav_9_20//:libav",
],
data = [
":stub.so",
],
data = [],
copts = ["-pthread", "-std=c++11", "-D_GLIBCXX_USE_CXX11_ABI=0", "-DNDEBUG"]
)
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libavcodec.so.54
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libavcodec.so.57
Binary file not shown.
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libavformat.so.54
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libavformat.so.57
Binary file not shown.
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libavutil.so.52
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libavutil.so.55
Binary file not shown.
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libswscale.so.2
Binary file not shown.
Binary file removed tensorflow_io/video/kernels/stub/libswscale.so.4
Binary file not shown.
7 changes: 4 additions & 3 deletions third_party/libtiff.BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,11 @@ cc_library(
exclude=[
"libtiff/tif_win32.c",
]),
hdrs = [
"libtiff/tiff.h",
"libtiff/tiffconf.h",
hdrs = glob([
"libtiff/*.h",
]) + [
"libtiff/tif_config.h",
"libtiff/tiffconf.h",
],
copts = [
"-D_GLIBCXX_USE_CXX11_ABI=0",
Expand Down

0 comments on commit 8a4a895

Please sign in to comment.