render delegate: loading mtoa while render delegate active crashes maya

[pmolodo]

Pretty much as the title says:

• start maya, make sure mtoa is not loaded / not set to auto-load
• start maya-to-hydra, set it as the viewport renderer, set it's renderer plugin to arnold
• load mtoa
• crash!

[sirpalee]

This problem is not something we can quickly solve. It happens because Arnold still lacks full support of multiple universes. Every call has to take place between a pair of AiBegin and AiEnd calls, and plugins are limited to this scope as well. Meaning, the render delegate sets up the global playground for Arnold, loads the shaders, plugins etc. but later this is overtaken by MtoA, and MtoA closes this playground once it finished initialising the plugin.

While we can sort this specific issue by making sure MtoA is always loaded before Maya to Hydra, this approach is just a quick patch on one of many problems. Any subsequent renders for the hypershade, or renders in the viewport (or evaluating AiStandins!) will have the same effect. While we can push AD to tackle this problem in the future, it's not something we can expect in the near term.

I can see two options to handle this problem.
1, rendering in an external process. This approach will also solve the current LD_PRELOAD issues we see on Linux, but it adds complexity and a small overhead due to serialising/deserialising commands. Note, rendering happens in a separate process in Houdini to Arnold and Katana to Arnold, so the cost could be small enough not to notice. And my previous experience with rendering in an external process confirms that for most interactive events the overhead won't be excessive.
2, loading the Arnold library into a private namespace and using the functions from there. It won't solve our LD_PRELOAD issues, but has minimal overhead and doesn't require significant changes in the API. We have to provide an alternative implementation of the AtString class, that uses our functions to Hash strings but has the same memory layout as the original struct. One thing I'm unsure, how the side loading of libraries is possible on Windows. While I had prototypes confirming the viability of this approach, those were limited to Linux. (and potentially behaving similarly to OSX)