Make USD built with python and without python C++ ABI compatible

[ARFoisy]

Description of Issue

To run USD in specialized processing environments, we would like a minimal USD runtime that has a lighter footprint.  In particular, the minimal USD runtime is compiled with option “—no-python”.

At the same time, the code compiled and linked against minimal USD should be able to load and run against a full USD runtime that has been compiled with option “—python”, for example when the environment is a DCC.

Currently the Minimal USD (option —no-python) and the full USD (option —python) are not ABI compatible. When building USD with “—python” and “—no-python”, we would like that the resulting libraries be ABI compatible with respect to the non-python C++ related code.

Steps to Reproduce

To reproduce we built two USD versions, the Minimal USD version with —no-python and the Full USD version with —python. Our internal code is compiled and linked against Minimal USD and then it loaded and ran in an environment that uses Full USD.

Work Description

We will be working on this issue and submitting a PR that will attempt to regularize vtables, member definitions, and other ABI issues between builds with “—python” and “—no-python” options.

In particular, code guarded by #ifdef PXR_PYTHON_SUPPORT_ENABLED that affects address alignment will be exposed to both “—python” and “—no-python” options and the resulting code definitions will be ajusted according the active option.

The above exposes some python related classes when compiling with “—no-python”. The affected classes will have two implementations with static_assert safeguards to check for size and alignment. For example (some includes and comments have been removed to show the pattern):

#ifdef PXR_PYTHON_SUPPORT_ENABLED

PXR_NAMESPACE_OPEN_SCOPE

// We define the empty stub for ABI compatibility even if Python support is 
// enabled so we can make sure size and alignment is the same.
class TfPyObjWrapperStub
{
public:
    static constexpr std::size_t Size = 16;
    static constexpr std::size_t Align = 8;

private:
    std::aligned_storage<Size, Align>::type _stub;
};

/// \class TfPyObjWrapper
///
/// Boost Python object wrapper.
/// …
#ifdef PXR_PYTHON_SUPPORT_ENABLED
class TfPyObjWrapper
    : public boost::python::api::object_operators<TfPyObjWrapper>
{
    typedef boost::python::object object;

public:

    TF_API TfPyObjWrapper();

    TF_API TfPyObjWrapper(object obj);

    object const &Get() const {
        return *_objectPtr;
    }

    TF_API PyObject *ptr() const;

    friend inline size_t hash_value(TfPyObjWrapper const &o) {
        return (size_t) o.ptr();
    }

    TF_API bool operator==(TfPyObjWrapper const &other) const;

    TF_API bool operator!=(TfPyObjWrapper const &other) const;

private:

    // Befriend object_operators to allow it access to implicit conversion to
    // boost::python::object.
    friend class boost::python::api::object_operators<TfPyObjWrapper>;
    operator object const &() const {
        return Get();
    }

    // Store a shared_ptr to a python object.
    std::shared_ptr<object> _objectPtr;
};

static_assert(sizeof(TfPyObjWrapper) == sizeof(TfPyObjWrapperStub),
              "ABI break: Incompatible class sizes.");
static_assert(alignof(TfPyObjWrapper) == alignof(TfPyObjWrapperStub),
              "ABI break: Incompatible class alignments.");

#else // PXR_PYTHON_SUPPORT_ENABLED

class TfPyObjWrapper : TfPyObjWrapperStub
{
};

#endif // PXR_PYTHON_SUPPORT_ENABLED

Alternatives

We considered fully isolating the python implementation by having zero python types in the .h files. Hiding the python-aware code is more involved. For example, some template functions that use python types in low-level libraries are being instantiated with types which are only defined in downstream libraries.

The above could be “future work”. At this moment we are working to have the same address alignment of declarations with properly stubbed definitions when compiling with “—python” and “—no-python”.

System Information (OS, Hardware)

Linux CentOS 7

Package Versions

USD 21.11

Build Flags

• Minimal USD version built with —no-python
• Full USD version built with —python

Additional Information

Ref.: Previous discussions on USD Group: https://groups.google.com/g/usd-interest/c/BQmL9BwwlZo

[jilliene]

Filed as internal issue #USD-7079

[sunyab]

This was addressed by #1729 and merged in the latest 22.05 release. Closing this out, thanks!