[usd] 23.05 Runtime plugins of OpenUSD installed incorrectly

[LittleCoinCoin]

Describe the bug
The .exe using OpenUSD delivered by vcpkg systematically crashes at runtime with message `Failed to find plugin for ArDefaultResolver``

Configuration and build of the port itself terminates without errors but it seems the install of OpenUSD (port name usd in vcpkg) by vcpkg seem to cause problems to find runtime plugins of OpenUSD.

The same issue was mentioned at the end of the PR for port update to version 23.02 of OpenUSD: #29688 (comment)

This issue seem to be related to the tree of directories, and the solution when doing a manual install of OpenUSD (i.e., without vcpkg) is discussed in this issue: PixarAnimationStudios/OpenUSD#1788

Environment

• OS: Windows
• Compiler: MSVC 14.37.32822

To Reproduce

• Create CMake project called USDTest
• Add vcpkg as a submodule in path/to/project/USDTest/.
• Add path/to/project/USDTest/vcpkg.json:

{
  "name": "usd-test",
  "version": "0.1.0",
  "dependencies": [
    {
      "name": "tbb"
    },
    "usd"
  ],
  "builtin-baseline": "61f610845fb206298a69f708104a51d651872877",
  "overrides": [
    {
      "name": "tbb",
      "version-string": "2020_U3#8"
    }
  ]
}

• In path/to/project/USDTest/CMakeLists.txt;

# ./CMakeList.txt
cmake_minimum_required (VERSION 3.20)
project ("USDTest")
add_subdirectory ("USDTest")

• In path/to/project/USDTest/USDTest/CMakeLists.txt:

add_executable (USDTest

"src/main.cpp"

)

if(WIN32)
	message("Building for Windows: disabling min/max macros from minwin.h which collide with OpenUSD max member function in robin_growth_policy.h")
	add_compile_definitions(NOMINMAX)
endif()

set_property(TARGET USDTest PROPERTY CXX_STANDARD 20)

find_package(TBB CONFIG REQUIRED)
target_link_libraries(USDTest PRIVATE TBB::tbb)

find_package(pxr CONFIG REQUIRED)
target_link_libraries(USDTest PRIVATE

ar
arch
gf
js
kind
ndr
pcp
plug
sdf
sdr
tf
trace
usd
usdGeom
usdHydra
usdLux
usdMedia
usdPhysics
usdRender
usdShade
usdSkel
usdUI
usdUtils
usdVol
vt
work
)


target_include_directories(USDTest
							PUBLIC
							"./headers"
)

• Add path/to/project/USDTest/USDTest/headers/main.h:

#include "pxr/usd/usd/stage.h"
#include "pxr/usd/usdGeom/mesh.h"

• Add path/to/project/USDTest/USDTest/src/main.cpp:

#include "main.h"

int main()
{
	// Create a new stage
	pxr::UsdStageRefPtr stage = pxr::UsdStage::CreateInMemory();

	return 0;
}

• The resulting project folder layout should be:

path/to/project/USDTest
    /USDTest
        /headers
            main.h
        /src
            main.cpp
        CMakeLists.txt
    /vcpkg
        -- all the stuffs from vcpkg --
    CMakeLists.txt
    vcpkg.json

Expected behavior
I expect the the exe to find the plugins at runtime and the .exe to exit without error messages.

Failure logs
When running the .exe:
Coding Error: in _GetPluginForType at line 147 of C:path\to\project\vcpkg\buildtrees\usd\src\v23.05-16c1648942.clean\pxr\usd\ar\resolver.cpp -- Failed to find plugin for ArDefaultResolver

Additional context
No other context at this time.

[JonLiu1993]

@LittleCoinCoin, Thanks for your issue, I will reproduce your issue locally and fix it.

[dg0yt]

I expect the the exe to find the plugins at runtime and the .exe to exit without error messages.

This takes two things:

• Ensure that the program use a relocatable approach to find the plugins. A relative location might work, but absolute paths from build time won't.
• Actually deploying the plugins. There is no general way to detect which plugins must be deployed along other binaries.

[LittleCoinCoin]

Hello @dg0yt,
Thank you for dropping by! Your input or reviews I've seen in the other issues are always informative.

If I understand correctly, you are suggesting that it's up to me to indicate in CMake which plugins I want to grab from where they are installed by default by vcpkg, and bring them somewhere my .exe can find them?

If so, I've thought about it, but ignoring the fact that I don't know how to write that in CMake, I got confused by the work done in the portfile.cmake of the usd port at lines:

vcpkg/ports/usd/portfile.cmake

Lines 87 to 92 in c5da4ba

	# fix plugInfo.json for runtime
	file(GLOB_RECURSE PLUGINFO_FILES ${CURRENT_PACKAGES_DIR}/lib/usd/*/resources/plugInfo.json)
	file(GLOB_RECURSE PLUGINFO_FILES_DEBUG ${CURRENT_PACKAGES_DIR}/debug/lib/usd/*/resources/plugInfo.json)
	foreach(PLUGINFO ${PLUGINFO_FILES} ${PLUGINFO_FILES_DEBUG})
	file_replace_regex(${PLUGINFO} [=["LibraryPath": "../../([a-zA-Z0-9_]+).dll"]=] [=["LibraryPath": "../../../bin/\1.dll"]=])
	endforeach()

The port file already does some work to fix the plugin's json, so I started thinking that the default intended behaviour of the portfile was to add every found plugins in the loop. I suppose I am wrong? 🤔

[dg0yt]

So the first bullet point is probably solved.
You still have to deal with the second bullet point.
For Qt plugins, an example was given recently in #37898 (comment). It might be generalized as:

add_custom_command(TARGET ${your_executable}
    POST_BUILD
    COMMAND ${CMAKE_COMMAND} -E make_directory "$<TARGET_FILE_DIR:${your_executable}>/<plugin-subdir>"
    COMMAND ${CMAKE_COMMAND} -E copy_if_different "<plugin-from-vcpkg-install>" "$<TARGET_FILE_DIR:${your_executable}>/<plugin-subdir>"
)

(untested)

[LittleCoinCoin]

Hello,

I got it to work. Before I explain, I set up a public template repo for anyone who wants to jumpstart from the project that worked for me: https://github.com/LittleCoinCoin/OpenUSD-setup-vcpkg-template
For now, I only checked windows but I'll do the work to check for other OS (though I expect less troubles). Also, my goal for the template project is that it can reproduce all the tutorials at https://openusd.org/release/tut_usd_tutorials.html.

Thank you @dg0yt for pointing me in the right direction. I'm still very skeptical about the fact that portfile.cmake for OpenUSD goes through the trouble of updating the dll search paths in the various plugInfo.json, but doesn't bring over the plugins at a path implied by the changes it made. It's like doing half the job.

• Anyway, the various plugins must be copied somewhere in our build directory where ../../../bin is correct. So, now, my path/to/project/USDTest/CMakeLists.txt looks like:

# ./CMakeList.txt
cmake_minimum_required (VERSION 3.20)

project ("USDTest")

# Because the USD plugin file directory resolution expects the dlls to be in a "/bin" folder
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin" CACHE STRING "" )

add_subdirectory ("USDTest")

So that all dynamic libraries built by CMake are copied in a /bin directory.

• Then, to bring over OpenUSD plugins, at the end of path/to/project/USDTest/USDTest/CMakeLists.txt, you can add:

  1.  add_custom_command(TARGET USDTest
      # Create the directory where the plugins information files will be copied over.
      POST_BUILD COMMAND ${CMAKE_COMMAND} -E make_directory "${CMAKE_BINARY_DIR}/lib/usd/"
      # Easy solution: copy all directories from "/lib/usd" of OpenUSD installed by vcpkg to the "/lib/usd" of your application.
      COMMAND ${CMAKE_COMMAND} -E copy_directory_if_different  "${CMAKE_BINARY_DIR}/vcpkg_installed/x64-windows/lib/usd" "${CMAKE_BINARY_DIR}/lib/usd"
      )

  2.  add_custom_command(TARGET USDTest
      # Create the directory where the plugins information files will be copied over.
      POST_BUILD COMMAND ${CMAKE_COMMAND} -E make_directory "${CMAKE_BINARY_DIR}/lib/usd/"
      
      #base plugin file
      COMMAND ${CMAKE_COMMAND} -E copy_if_different "${CMAKE_BINARY_DIR}/vcpkg_installed/x64-windows/lib/usd/plugInfo.json" "${CMAKE_BINARY_DIR}/lib/usd/"
     
      #for the default asset resolver plugin
      #required even for the very basic action of creating a new stage
      COMMAND ${CMAKE_COMMAND} -E make_directory "${CMAKE_BINARY_DIR}/lib/usd/ar"
      COMMAND ${CMAKE_COMMAND} -E copy_directory_if_different "${CMAKE_BINARY_DIR}/vcpkg_installed/x64-windows/lib/usd/ar" "${CMAKE_BINARY_DIR}/lib/usd/ar"
     
      #for the scene description format plugin
      #required even for the very basic action of creating a new stage
      COMMAND ${CMAKE_COMMAND} -E make_directory "${CMAKE_BINARY_DIR}/lib/usd/sdf"
      COMMAND ${CMAKE_COMMAND} -E copy_directory_if_different "${CMAKE_BINARY_DIR}/vcpkg_installed/x64-windows/lib/usd/sdf" "${CMAKE_BINARY_DIR}/lib/usd/sdf"
     
      #for many things (judging by the number of plugin information given in the )
      #required even for the very basic action of creating a new stage
      COMMAND ${CMAKE_COMMAND} -E make_directory "${CMAKE_BINARY_DIR}/lib/usd/usd"
      COMMAND ${CMAKE_COMMAND} -E copy_directory_if_different "${CMAKE_BINARY_DIR}/vcpkg_installed/x64-windows/lib/usd/usd" "${CMAKE_BINARY_DIR}/lib/usd/usd"
      )

Case i. is the easy solution where you bring everything over.
Case ii. is the "annoying" solution where you micro-manage the plugins you want. But note that the three plugins in this example are actually mandatory for the super basic task of creating a new USD stage (i.e. pxr::UsdStageRefPtr stage = pxr::UsdStage::CreateInMemory();). I'd argue these plugins should be added by default all the time.

I'll close this issue. Thank you for your support ☺️