Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[openimageIO] doesn't built with manifest - fails to compile jasper #41124

Closed
p0las opened this issue Sep 23, 2024 · 19 comments
Closed

[openimageIO] doesn't built with manifest - fails to compile jasper #41124

p0las opened this issue Sep 23, 2024 · 19 comments
Assignees
Labels
category:question This issue is a question

Comments

@p0las
Copy link

p0las commented Sep 23, 2024

Describe the bug
openImageIO builds ok on windows without a manifest file, using all the latest ports (9352613).
however when a manifest is present, with the same base and no constraints it fails.

I have a manifest as follows:

{
  "name": "project",
  "version-semver": "1.0.0",
  "dependencies": [
    "boost",
    "ffmpeg",
    "tiff",
    "opencolorio",
    "libraw",
    "libpng",
    "python3"
  ],
  "builtin-baseline":"935261370b7817db94e62c891c0eb6514634788c"
 
}

without the manifest I manually added the required libraries to vcpkg. with the manifest, I wanted to simplify the build for the future and have the ability to add constraints (python 3.10).
my assumption was that there should be no difference between those two builds as both should use the latest ports and build the same.

Environment

  • OS: windows10
  • Compiler: Visual Studio 17 2022

To Reproduce
Steps to reproduce the behavior:

  1. clone vcpkg repo (In my case c:/dev/vcpkg
  2. get OpenColourIO repo: https://github.com/AcademySoftwareFoundation/OpenImageIO/releases/tag/v2.5.14.0
  3. create manifest like above in the root of opencolourIO repo
  4. cd to build directory
  5. cmake -DCMAKE_TOOLCHAIN_FILE=c:/DEV/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -DCMAKE_PREFIX_PATH=c:\DEV\vcpkg\installed\x64-windows -DUSE_PYTHON=1 -DUSE_QT=0 -DOIIO_BUILD_TESTS=0 ..

Repro code when

Expected behavior
I expected the builds with the manifest and without in this case to be identical

Failure logs

C:\DEV\vcpkg\buildtrees\versioning_\versions\jasper\d7f51c26899ba9433a5a3ab92fc5b5887d5c764c: info: installing overlay port from here
-- Using cached jasper-software-jasper-402d096b39f4f618ad9e6ff2b4fc1b5eb260a2e4.tar.gz.
-- Cleaning sources at C:/DEV/vcpkg/buildtrees/jasper/src/5eb260a2e4-ca02e25be6.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/DEV/vcpkg/downloads/jasper-software-jasper-402d096b39f4f618ad9e6ff2b4fc1b5eb260a2e4.tar.gz
-- Applying patch no_stdc_check.patch
-- Applying patch fix-library-name.patch
-- Using source at C:/DEV/vcpkg/buildtrees/jasper/src/5eb260a2e4-ca02e25be6.clean
-- Found external ninja('1.11.0').
-- Configuring x64-windows
-- Building x64-windows-dbg
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:134 (message):
    Command failed: "C:/Program Files/CMake/bin/cmake.exe" --build . --config Debug --target install -- -v -j13
    Working Directory: C:/DEV/vcpkg/buildtrees/jasper/x64-windows-dbg
    See logs for more information:
      C:\DEV\vcpkg\buildtrees\jasper\install-x64-windows-dbg-out.log

Call Stack (most recent call first):
  C:/DEV/opencolorio/OpenImageIO-2.5.14.0/build/vcpkg_installed/x64-windows/share/vcpkg-cmake/vcpkg_cmake_build.cmake:74 (vcpkg_execute_build_process)
  C:/DEV/opencolorio/OpenImageIO-2.5.14.0/build/vcpkg_installed/x64-windows/share/vcpkg-cmake/vcpkg_cmake_install.cmake:16 (vcpkg_cmake_build)
  buildtrees/versioning_/versions/jasper/d7f51c26899ba9433a5a3ab92fc5b5887d5c764c/portfile.cmake:38 (vcpkg_cmake_install)
  scripts/ports.cmake:147 (include)


error: building jasper:x64-windows failed with: BUILD_FAILED

Additional context
Add any other context about the problem here.

@WangWeiLin-MV WangWeiLin-MV added the category:question This issue is a question label Sep 23, 2024
@WangWeiLin-MV
Copy link
Contributor

5. cmake -DCMAKE_TOOLCHAIN_FILE=c:/DEV/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows -DCMAKE_PREFIX_PATH=c:\DEV\vcpkg\installed\x64-windows -DUSE_PYTHON=1 -DUSE_QT=0 -DOIIO_BUILD_TESTS=0 ..

In manifest mode with CMake integration, the ports installation path is vcpkg_install in CMake build directory instead of installed in vcpkg instance. Just remove the argument -DCMAKE_PREFIX_PATH=c:\DEV\vcpkg\installed\x64-windows

@dg0yt
Copy link
Contributor

dg0yt commented Sep 23, 2024

Describe the bug openImageIO builds ok on windows without a manifest file, using all the latest ports (9352613). however when a manifest is present, with the same base and no constraints it fails.

I have a manifest as follows:

 ...
  "builtin-baseline":"935261370b7817db94e62c891c0eb6514634788c"
 
}

Consider updating builtin-baseline.

The chosen builtin-baseline is from "last year", according to Github. This means vcpkg will install old versions of the ports of interest, unless version constraints and overrides demand something else. So you effectively ask for bugs which were resolved a long time ago.

@p0las
Copy link
Author

p0las commented Sep 23, 2024

DCMAKE_PREFIX_PATH

thank you. that of course makes sense.

The chosen builtin-baseline is from "last year",

that was my botched attempt of getting python 3.10. I did test with the latest SHA too. the extra flag I passed to the command line was messing it all up.

thank you everyone for help! appreciate it.

@p0las p0las closed this as completed Sep 23, 2024
@p0las
Copy link
Author

p0las commented Sep 23, 2024

it compiles ok without overrides with a simple manifest like this:

{
  "name": "project",
  "version-semver": "1.0.0",
  "dependencies": [
    "boost",
    "ffmpeg",
    "tiff",
    "openexr",
    "opencolorio",
    "libpng",
    "python3",
    "pybind11"

  ],
  "builtin-baseline":"98aa6396292d57e737a6ef999d4225ca488859d5"
 
}

the moment however I try any python 3.10 it breaks.

{
  "name": "project",
  "version-semver": "1.0.0",
  "dependencies": [
    "boost",
    "ffmpeg",
    "tiff",
    "openexr",
    "opencolorio",
    "libpng",
    "python3",
    "pybind11"

  ],
  "builtin-baseline":"98aa6396292d57e737a6ef999d4225ca488859d5",
  "overrides": [
        { "name": "python3", "version": "3.10.5" }
    ]
 
}

it seems it has problems compiling an older version of python and it is looking for files not in the build vcpkg but in the generic one on my system
LINK : fatal error LNK1181: cannot open input file 'FFI_RELEASE-NOTFOUND.obj' [C:\DEV\vcpkg\buildtrees\python3\x64-windows-rel\v3.10.5-a56883f72e.clean\PCbuild\pythoncore.vcxproj]

EDIT: fixed python version in the manifest

@p0las p0las reopened this Sep 23, 2024
@dg0yt
Copy link
Contributor

dg0yt commented Sep 23, 2024

Can please clarify which port is failing - openimageio, jasper, or python3 ?

The last manifest selects python 3.10.7 via overrides, but the error shows a python 3.10.5 source dir. This data doesn't belong together.

"FFI_RELEASE-NOTFOUND" points from python3 to libffi. You may need to choose a version of libffi of the same age as python3. Only few combinations of versions are tested together, determined by PRs and merges.

@p0las
Copy link
Author

p0las commented Sep 23, 2024

I removed libraw and jasper from the process to simplify it. I'm trying to build openimageIO with python 3.10 bindings. I tried 3.10.7 and 3.10.5 hence the difference between the manifest and the error. it fails in either case. From the manual I thought that constraining one version in the overrides would make the system find matching corresponding versions of the dependencies automatically. To be honest I have never tried to control versions this way.
I also tried to roll back vcpkg repo to the last commit with python 3.10 and use this as the base (and no overrides) but this didn't work either. At this stage, I'm out of ideas. There must be a way to compile openimageIO with python 3.10.

@WangWeiLin-MV
Copy link
Contributor

From the manual I thought that constraining one version in the overrides would make the system find matching corresponding versions of the dependencies automatically.

Indirect dependencies in manifest mode also need to be explicitly specified. https://learn.microsoft.com/en-us/vcpkg/consume/boost-versions#:~:text=that%20only%20the%20version

I also tried to roll back

You could try this last commit of Python 3.10 6da00e1 before c6928df

@p0las
Copy link
Author

p0las commented Sep 23, 2024

I tried a few older ones but they all failed with:
error: no version database entry for opencolorio at 2.1.2.
I also tried older openimageIO but so far I failed to hit the right lucky combination.

@WangWeiLin-MV
Copy link
Contributor

WangWeiLin-MV commented Sep 23, 2024

error: no version database entry for opencolorio at 2.1.2.

The version of 2.1.2 is broken and deleted in 768619a#diff-427f36b18901d5fabb35e4a6ec900ee811e8eecb667bf5d5556cb9b755c78902

Maybe you can add overrides to specify opencolorio to 2.1.1 or other versions.

@p0las
Copy link
Author

p0las commented Sep 23, 2024

I thought this was a brilliant idea and I tried it. old SHA with python 3.10 and an override to existing opencolorio

  "builtin-baseline":"6da00e1160d0ded7b8df06d214a513d5dec9825c",
  "overrides": [
        
        {"name": "opencolorio", "version": "2.1.1"}
    ]

(this is not the full manifest, just changes)
unfortunately this didn't work either. this time boost is missing...

-- Using Boost::filesystem
-- clang-format not found.
--
-- * Checking for dependencies...
-- *   - Missing a dependency 'Package'?
-- *     Try cmake -DPackage_ROOT=path or set environment var Package_ROOT=path
-- *     For many dependencies, we supply src/build-scripts/build_Package.bash
-- *   - To exclude an optional dependency (even if found),
-- *     -DUSE_Package=OFF or set environment var USE_Package=OFF
--
-- Boost_COMPONENTS = thread;filesystem
-- Could NOT find Boost (missing: Boost_DIR)
-- Boost library not found
--     Try setting Boost_ROOT ?
CMake Error at src/cmake/checked_find_package.cmake:184 (message):
  Boost is required, aborting.

I think I'm going to give up on this exercise as I see no way to use vcpkg to compile it in this case. I can see boost in the vcpkg folder and this was a clean run with fresh checkout of vcpkg. no idea why vcpkg thinks it is missing. I'm assuming it might be looking for a different version than the one that is present but this is unclear.

@WangWeiLin-MV
Copy link
Contributor

@p0las I've tried this that the last commit before opencolorio 2.1.2 with python 3.10.5 687f4aa, there seems to be no problem.

@p0las
Copy link
Author

p0las commented Sep 24, 2024

@WangWeiLin-MV thank you. I was pushing to get the latest possible commit that was using python 3.10 but the trick seems to go back in time even more to find a commit with working opencolourIO and python 3.10 and use no overrides. I was hoping that overrides would have more or less the same effect but I think I expected too much from this system.

did you use a manifest or manually check out this commit in vcpkg? it fails for me with the manifest like this:

{
  "name": "project",
  "version-semver": "1.0.0",
  "dependencies": [
    "boost",
    "ffmpeg",
    "tiff",
    "openexr",
    "opencolorio",
    "libpng",
    "python3",
    "pybind11"

  ],
  "builtin-baseline":"687f4aab11df9b1a854d0d7207c558da545b4cc9"
  
}

will experiment with it more. thank you.

@WangWeiLin-MV
Copy link
Contributor

Yes, and it still running since there are a lot of ports. How's the error when you run it?

@p0las
Copy link
Author

p0las commented Sep 24, 2024

after a very long build it complained that boost was missing like in my other case above

-- Using Boost::filesystem
-- clang-format not found.
--
-- * Checking for dependencies...
-- *   - Missing a dependency 'Package'?
-- *     Try cmake -DPackage_ROOT=path or set environment var Package_ROOT=path
-- *     For many dependencies, we supply src/build-scripts/build_Package.bash
-- *   - To exclude an optional dependency (even if found),
-- *     -DUSE_Package=OFF or set environment var USE_Package=OFF
--
-- Boost_COMPONENTS = thread;filesystem
-- Could NOT find Boost (missing: Boost_DIR)
-- Boost library not found
--     Try setting Boost_ROOT ?
CMake Error at src/cmake/checked_find_package.cmake:184 (message):
  Boost is required, aborting.

@WangWeiLin-MV
Copy link
Contributor

I was hoping that overrides would have more or less the same effect

This is the case when some port are tightly coupled with dependencies' version. And vcpkg only tests compatibility between the ports' latest versions.

@p0las
Copy link
Author

p0las commented Sep 24, 2024

it seems that in this old checkout some ports cannot be downloaded any more. I manually switched vcpkg to the old comment with python 3.10.
then I simply ran:
vcpkg install --triplet x64-windows boost
it eventually failed trying to build bzip2 because it wasn't able to download the required dependency from any of the mirrors:

Error: Failed to download from mirror set:
https://repo.msys2.org/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
https://www2.futureware.at/~nickoe/msys2-mirror/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
https://mirror.yandex.ru/mirrors/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
https://mirrors.tuna.tsinghua.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
https://mirrors.ustc.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12007
https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12007
https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12007
https://mirror.bit.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: WinHttpSendRequest() failed: 12007
https://mirror.selfnet.de/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
https://mirrors.sjtug.sjtu.edu.cn/msys2/mingw/i686/mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst: failed: status code 404
Call Stack (most recent call first):
  scripts/cmake/vcpkg_download_distfile.cmake:249 (z_vcpkg_download_distfile_show_proxy_and_fail)
  scripts/cmake/vcpkg_acquire_msys.cmake:26 (vcpkg_download_distfile)
  scripts/cmake/vcpkg_acquire_msys.cmake:116 (z_vcpkg_acquire_msys_download_package)
  scripts/cmake/vcpkg_find_acquire_program.cmake:530 (vcpkg_acquire_msys)
  scripts/cmake/vcpkg_fixup_pkgconfig.cmake:151 (vcpkg_find_acquire_program)
  ports/bzip2/portfile.cmake:47 (vcpkg_fixup_pkgconfig)
  scripts/ports.cmake:147 (include)


error: building bzip2:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest port files with `git pull` and `vcpkg update`.
Then check for known issues at:
    https://github.com/microsoft/vcpkg/issues?q=is%3Aissue+is%3Aopen+in%3Atitle+bzip2
You can submit a new issue at:
    https://github.com/microsoft/vcpkg/issues/new?template=report-package-build-failure.md&title=[bzip2]+Build+error
Include '[bzip2] Build error' in your bug report title, the following version information in your bug description, and attach any relevant failure logs from above.
    vcpkg-tool version: 2022-07-14-fb7e3b1bb3b1ca185fffe2be36b2fbc53198360f
    vcpkg-scripts version: 687f4aab1 2022-07-18 (2 years, 2 months ago)

@WangWeiLin-MV
Copy link
Contributor

"FFI_RELEASE-NOTFOUND" points from python3 to libffi. You may need to choose a version of libffi of the same age as python3.

By using @dg0yt 's solution, you could use this json.

Installation passed on x64-windows

{
  "name": "project",
  "version-semver": "1.0.0",
  "dependencies": [
    "boost",
    "ffmpeg",
    "tiff",
    "openexr",
    "opencolorio",
    "libpng",
    "python3",
    "pybind11"
  ],
  "overrides": [
    {
      "name": "libffi",
      "version": "3.4.4",
      "port-version": 2
    },
    {
      "name": "python3",
      "version": "3.10.7",
      "port-version": 7
    }
  ],
  "builtin-baseline": "f176b58f35a75f9f8f54099cd9df97d2e2793a2e"
}

@dg0yt
Copy link
Contributor

dg0yt commented Sep 24, 2024

it seems that in this old checkout some ports cannot be downloaded any more.

Those errors were for (very) old msys2 binary packages (not ports) which were removed from msys2 download servers. Well, old checkouts bring old scripts. Just don't use old checkouts. If you need old port version, use manifest mode and version constraints. Or use overlay ports. Or add your own registry.

@p0las
Copy link
Author

p0las commented Sep 24, 2024

@WangWeiLin-MV omg that worked! thank you.

@p0las p0las closed this as completed Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:question This issue is a question
Projects
None yet
Development

No branches or pull requests

3 participants