Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libvmaf] New port #41152

Merged
merged 8 commits into from
Sep 25, 2024
Merged

[libvmaf] New port #41152

merged 8 commits into from
Sep 25, 2024

Conversation

bc-lee
Copy link
Contributor

@bc-lee bc-lee commented Sep 24, 2024

Fixes #41129.

  • Changes comply with the maintainer guide.
  • The name of the port matches an existing name for this component on https://repology.org/ if possible, and/or is strongly associated with that component on search engines.
  • Optional dependencies are resolved in exactly one way. For example, if the component is built with CMake, all find_package calls are REQUIRED, are satisfied by vcpkg.json's declared dependencies, or disabled with CMAKE_DISABLE_FIND_PACKAGE_Xxx.
  • The versioning scheme in vcpkg.json matches what upstream says.
  • The license declaration in vcpkg.json matches what upstream says.
  • The installed as the "copyright" file matches what upstream says.
  • The source code of the component installed comes from an authoritative source.
  • The generated "usage text" is accurate. See adding-usage for context.
  • The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
  • Only one version is in the new port's versions file.
  • Only one version is added to each modified port's versions file.

@bc-lee
Copy link
Contributor Author

bc-lee commented Sep 24, 2024

@microsoft-github-policy-service agree

@bc-lee
Copy link
Contributor Author

bc-lee commented Sep 24, 2024

For naming this package (vmaf or libvmaf), Repology lists vmaf as the original project name. However, I chose libvmaf (though I'm open to changing it to vmaf if reviewers suggest) because we're shipping only the library, not the tools.

For example, on my Fedora 40 system, vmaf is split into three packages:

$ rpm -qa | grep vmaf
libvmaf-2.3.0-7.fc40.x86_64
vmaf-models-2.3.0-7.fc40.noarch
vmaf-2.3.0-7.fc40.x86_64

vmaf provides /usr/bin/vmaf, libvmaf provides /usr/lib64/libvmaf.so, and vmaf-models provides pre-trained models like /usr/share/model/vmaf_float_b_v0.6.3.json. So, I believe libvmaf is more appropriate.

@bc-lee bc-lee marked this pull request as ready for review September 24, 2024 18:54
@bc-lee bc-lee marked this pull request as draft September 24, 2024 18:55
@LilyWangLL LilyWangLL self-assigned this Sep 25, 2024
@LilyWangLL LilyWangLL added the category:new-port The issue is requesting a new library to be added; consider making a PR! label Sep 25, 2024
{
"name": "libvmaf",
"version": "3.0.0",
"port-version": 1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"port-version": 1,

New port doesn't need add port-version.

@bc-lee
Copy link
Contributor Author

bc-lee commented Sep 25, 2024

Android and iOS compilation is disabled due to weird errors when compiling the arm-neon-android triplet. It seems that vcpkg/meson wrongly sets the sysroot, causing functions like fseeko, which aren't supported in Android API 21, to be used. According to the Android documentation (https://android.googlesource.com/platform/bionic/+/main/docs/32-bit-abi.md):

Android does not require the _LARGEFILE_SOURCE macro to be used to make fseeko and ftello available. Instead they're always available from API level 24 where they were introduced, and never available before then.

I need some more time to investigate this issue, and I don't think there are many people who want to run VMAF on Android or iOS (at least, not me).

[53/55] /Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ --target=armv7-none-linux-androideabi21 -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-security -frtti -fexceptions -fPIC -fno-limit-debug-info --sysroot=/Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -Isrc/libvmaf.a.p -Isrc -I../src/v3.0.0-2bc8652704.clean/libvmaf/src -Iinclude -I../src/v3.0.0-2bc8652704.clean/libvmaf/include -I../src/v3.0.0-2bc8652704.clean/libvmaf/src/feature -I../src/v3.0.0-2bc8652704.clean/libvmaf/src/feature/common -I/Users/redacted/src/workspace/test-vcpkg2/build/vcpkg_installed/arm-neon-android/include -fdiagnostics-color=always -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_FAST -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c++11 -g --target=armv7-none-linux-androideabi21 -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-security -frtti -fexceptions -fPIC -fno-limit-debug-info --sysroot=/Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -fPIC -pthread -pedantic -DOC_NEW_STYLE_INCLUDES -MD -MQ src/libvmaf.a.p/svm.cpp.o -MF src/libvmaf.a.p/svm.cpp.o.d -o src/libvmaf.a.p/svm.cpp.o -c ../src/v3.0.0-2bc8652704.clean/libvmaf/src/svm.cpp
FAILED: src/libvmaf.a.p/svm.cpp.o 
/Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang++ --target=armv7-none-linux-androideabi21 -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-security -frtti -fexceptions -fPIC -fno-limit-debug-info --sysroot=/Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -Isrc/libvmaf.a.p -Isrc -I../src/v3.0.0-2bc8652704.clean/libvmaf/src -Iinclude -I../src/v3.0.0-2bc8652704.clean/libvmaf/include -I../src/v3.0.0-2bc8652704.clean/libvmaf/src/feature -I../src/v3.0.0-2bc8652704.clean/libvmaf/src/feature/common -I/Users/redacted/src/workspace/test-vcpkg2/build/vcpkg_installed/arm-neon-android/include -fdiagnostics-color=always -D_LIBCPP_HARDENING_MODE=_LIBCPP_HARDENING_MODE_FAST -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -std=c++11 -g --target=armv7-none-linux-androideabi21 -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D_FORTIFY_SOURCE=2 -march=armv7-a -mthumb -Wformat -Werror=format-security -frtti -fexceptions -fPIC -fno-limit-debug-info --sysroot=/Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -fPIC -pthread -pedantic -DOC_NEW_STYLE_INCLUDES -MD -MQ src/libvmaf.a.p/svm.cpp.o -MF src/libvmaf.a.p/svm.cpp.o.d -o src/libvmaf.a.p/svm.cpp.o -c ../src/v3.0.0-2bc8652704.clean/libvmaf/src/svm.cpp
In file included from ../src/v3.0.0-2bc8652704.clean/libvmaf/src/svm.cpp:11:
/Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1/fstream:869:9: error: no member named 'fseeko' in the global namespace; did you mean 'fseek'?
  869 |   if (::fseeko(__file_, __width > 0 ? __width * __off : 0, __whence))
      |       ~~^

@bc-lee bc-lee marked this pull request as ready for review September 25, 2024 04:51
@dg0yt
Copy link
Contributor

dg0yt commented Sep 25, 2024

It seems that vcpkg/meson wrongly sets the sysroot, causing functions like fseeko, which aren't supported in Android API 21, to be used.

vcpkg doesn't set the wrong sysroot. But many packages do configuration wrong, or make wrong assumptions. And the NDK really contributes to confusion.

@dg0yt
Copy link
Contributor

dg0yt commented Sep 25, 2024

Note that vcpkg builds with API level 21 by default. You can test locally with a higher API level e.g. using

vcpkg install libvmaf:arm64-android --cmake-args=-DVCPKG_CMAKE_SYSTEM_VERSION=24

If it builds successfully, the CI failure shall be recorded in scripts/ci.baseline.txt, similar to:

# needs android-24
libimobiledevice-glue:arm-neon-android=fail
libimobiledevice-glue:arm64-android=fail
libimobiledevice-glue:x64-android=fail

Some packages need even higher levels.

"description": "libvmaf is a library developed by Netflix to compute the VMAF (Video Multi-Method Assessment Fusion) metric.",
"homepage": "https://github.com/Netflix/vmaf",
"license": "BSD-2-Clause-Patent",
"supports": "!windows & !android & !ios",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to their GHA, windows is supported.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to their GHA, windows is supported.

However, it is only tested under msys2. The libvmaf codebase extensively uses POSIX-specific headers such as <pthread.h> and <unistd.h>. I don't think it will build cleanly on Windows with MSVC. Patching a lot of code to make it work on Windows doesn't seem like a good idea, as this should be addressed in the upstream project, not in vcpkg.

@bc-lee
Copy link
Contributor Author

bc-lee commented Sep 25, 2024

The libvmaf library itself doesn't use fseeko symbols. That's why I find this issue strange. The file /Users/redacted/android/sdk/ndk/27.1.12297006/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1/fstream is clearly part of NDK's libc++, which is designed to handle various API levels. However, for some reason, it assumes fseeko is available when it's actually not. So I don't think limiting the API level is a long-term solution to this issue.

LilyWangLL
LilyWangLL previously approved these changes Sep 25, 2024
@LilyWangLL LilyWangLL added the info:reviewed Pull Request changes follow basic guidelines label Sep 25, 2024
@bc-lee
Copy link
Contributor Author

bc-lee commented Sep 25, 2024

Some packages need even higher levels.

The issue is with meson, not vcpkg. I added a workaround for mesonbuild/meson#3049 as a patch. If the API level is changed to 24 or above in the future, or if meson is fixed to no longer define _FILE_OFFSET_BITS=64 for Android targets, the workaround can be removed.

@JavierMatosD JavierMatosD merged commit d1e8683 into microsoft:master Sep 25, 2024
16 checks passed
@bc-lee bc-lee deleted the feature/libvmaf branch September 25, 2024 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:new-port The issue is requesting a new library to be added; consider making a PR! info:reviewed Pull Request changes follow basic guidelines
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[New Port Request] libvmaf
4 participants