Introducing SpectatorView into MRTK

[danielescudero]

Hi Everyone,
Lighthouse London guys here (part of MCS Mixed Reality Studios). After some initial conversations with @timGerken , @keveleigh and @davidkline-ms we've finished the first version of SpectatorView for the MRToolkit. This new system allows iOS ARKit-enabled devices to witness and record what HoloLens users are experiencing, just like spectator view, but cheaper and faster. SpectatorView takes care of handling multiple sessions, automatic networking, recording and syncing space so phones see where holograms are.

The files are ready to be integrated into MRTK (2017.2.1.3). In order to do so we'd like to understand where in the toolkit this section of work should sit. Currently we have unity component and a native code component, which has a dependency on the OpenCV libraries.

Also, we have had issues building against the toolkit on the mac due to the GLTF component (inside MRTK) not being compatible with Mac. The GLTF component needs to be platform safe, and it's currently the only module inside MRTK that's not.

We'd love to know how to proceed.
Best
Dan Escudero

[keveleigh]

Looking forward to seeing this make its way in!

What error do you see with GLTF when building for Mac?

[david-c-kline]

Just spoke with @danielescudero,

He will be creating an ARCA folder underneath HoloToolkit for their components, scripts and documentation.

The OpenCV dependency will be documented and tested to insure that the lack of OpenCV does not break building the toolkit or any functionality (other than ARCA becoming a noop).

A new issue will be opened re: the GLTF script(s) not being platform safe and will recommend their clients remove it from their ios builds.

An example scene is ready and will be added in the PR :)

[danielescudero]

@keveleigh All scripts using GLTF causing issues only when building for iOS. For example: Assets/HoloToolkit/Utilities/Scripts/GLTF/Scripts/Cache/MeshCacheData.cs(2,7): error CS0246: The type or namespace name `GLTF' could not be found. Are you missing an assembly reference?

[david-c-kline]

Updated initial issue comment to match the feature name.

[keveleigh]

@danielescudero If you check the include platforms to include iOS, do you still see that?

[danielescudero]

@keveleigh
It stopped complaining but now throws this error when iOS is included.
ArgumentException: The Assembly Newtonsoft.Json is referenced by GLTFSerialization ('Assets/HoloToolkit/Utilities/Scripts/GLTF/Plugins/GLTFSerialization/GLTFSerialization.dll'). But the dll is not allowed to be included or could not be found.

[keveleigh]

Does adding the same checkbox on the non-UWP Newtonsoft DLL fix that?

[danielescudero]

Nope, throws:
TypeLoadException: Could not load type 'GLTF.Math.Matrix4x4' from assembly 'GLTFSerialization, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.
UnityEditor.Build.BuildPipelineInterfaces.InitializeBuildCallbacks (BuildCallbacks findFlags) (at /Users/builduser/buildslave/unity/build/Editor/Mono/BuildPipeline/BuildPipelineInterfaces.cs:115)
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

Error building Player: TypeLoadException: Could not load type 'GLTF.Math.Matrix4x4' from assembly 'GLTFSerialization, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'.

[david-c-kline]

This was released in the 2017.4.0.0 release candidate

[ilyavitek]

Trying to build and test an example of Spectator View, hololens is ok, but during iOS building in Unity I've received some errors (1st screenshot), added iOS platform for Newtonsoft.Json.dll (2nd screenshot) and after that, I get some more problems (3rd screenshot). Maybe somebody knows how to fix it?

[timGerken]

@TobyGilbert-ms @JoseSantano-ms

[ChauNTran]

@ilyavitek In player settings> other settings, change api compatibility level to from .NET 2.0 subset to .NET 2.0. It solved the same issue for me.

[ilyavitek]

I have built both of the versions, but problem is after iOS app show code HoloLens not responding. Should example scene show at least something when it is not connected to iPhone app (for me it's only splash screen and then nothing, without errors)?

[danielescudero]

@ilyavitek is the HoloLens camera light turning on when iPhone displays the code?

[ilyavitek]

@danielescudero nope. In HoloLens app I can see only splash at the beginning and nothing else (is it ok?). I'm trying to launch a sample scene with spectator view.

[danielescudero]

That’s not what should be happening. Your HL app should work normally independent of any iPhone joining the session, so HL app should show you your holograms. I’m sure there was a similar comment from someone some months ago. Let me look this up.

[danielescudero]

@ilyavitek What version of MRTK are you using? If you run the app from the Unity Editor, do your holograms appear? This thread solves the same issue. Look towards the end of the thread: #2438

[ilyavitek]

@danielescudero
MRTK Version: 2017.4.2.0
In Unity Editor when app switched for HoloLens build and I press play everything seems ok.
Thank you for link to thread.

[ilyavitek]

@danielescudero
It seems that my problem was that I was thinking that plugin is already in the project. But I need to build it. Now I'm struggling with building plugin. It failed at the "Install OpenCV Contrib for UWP" step :(
microsoft/vcpkg#4745

[CMLeshiba]

I am struggling to build IOS spectator view

[ghost]

Hello @CMLeshiba
The GLTF stuff isn't compatible with iOS at the moment, you can just remove it when building for it. You can find more information Here

Thanks
Jose

[CMLeshiba]

@JoseSantano-ms

How do I disable it?

[ghost]

@CMLeshiba Personally, I tend to just move it outside the Assets folder whenever I build for iOS.

[CMLeshiba]

@JoseSantano-ms thanks for prompt response. Let's say I am starting a new project, what is required for me to import into this project. I may sound like I am asking stupid questions but I am new to this hololens and unity thing.

[ghost]

Hi @CMLeshiba I recommend you have a look here and then ask any questions you may have. That is to set up Spectator View in an existing HoloLens project.

[CMLeshiba]

@JoseSantano-ms I have been looking at that but what confuses me is that:

1. Unity ARKit Plugin does not exist anymore.
2. The prefabs, do I get them from MixedRealityToolkit-Unity? if so does it mean I should bring MixedRealityToolkit-Unity into my project?

[CMLeshiba]

So this the error that I am coming across: multiple plugins with the same name "sharingclient".
@JoseSantano-ms how do I go about this?

[ghost]

Hi @CMLeshiba
That sounds like both dll are being set to be used in the Editor. if you select a dll you should see something like this in the inspector.

I assume that both dll have Editor selected and only one of then should.

[CMLeshiba]

Thanks @JoseSantano-ms so which one should be use in editor?

[ghost]

I tend to go with the x64 unless it doesn't work for you. You could try the same!

[CMLeshiba]

@JoseSantano-ms When I deploy my app on the hololens I cannot see the holograms except for unity splash screen. Am I missing anything?