Canvas video stream freezes on mobile safari

[pkpp1233]

Describe the bug
iPhone 7 and 8 for mobile safari face tracking seems to be broken.

Specifically, if you look at the 2d face paint demo, the canvas loads, shows the first few images, then freezes. Console shows the following error: WebGL: INVALID_OPERATION: texImage2D: type HALF_FLOAT_OES but ArrayBufferView is not NULL

To Reproduce
Steps to reproduce the behavior:

1. You can reproduce using an iPhone on iOS 11, mobile safari.
2. Open https://jeeliz.com/demos/faceFilter/demos/canvas2D/faceDraw/
3. Your picture will load, but video stream will freeze. You'll still be able to draw.

Expected behavior
I'd except the video stream / canvas to not freeze while tracking face. It works properly on desktop and android mobile.

Screenshots
n/a

Smartphone (please complete the following information):

• Device: iPhone7
• OS: iOS11
• Browser safari

[xavierjs]

Thank you for this detailed bug report. Fortunately I can reproduce it on my ipad pro, it freezes perfectly well ! I will fix it today.

[pkpp1233]

Awesome 👍, thanks @xavierjs.

[pkpp1233]

If helpful, I did notice when I refreshed a few times, 1 out of every few ended up not freezing. Seems like a timing issue.

[xavierjs]

This bug was pretty hard... This is a problem with IOS.
Since a recent update I cannot use a

[xavierjs]

I have pushed the modif and it works for me with Ipad pro (I tried with Safari in private navigation to avoid any browser cache problem). I hope it will work for you too !

[pkpp1233]

Taking a look now. Will report.

[pkpp1233]

Works great! Thanks for the fix. 🚀

[ashique12009]

Actually i am trying it in tensorflow JS, in there i am facing same freezing issue.
Could you tell me how you solve it @xavierjs?

Thank you.

[xavierjs]

It may be due to the GPU. I had a similar issue with an old laptop, the GPU was overheating and it triggered thermal throttling.

I don’t use tensorslow and face filter does not rely on it, but I know that tensorflow.js uses WebGL too to run.

[namklabs]

@ashique12009 Greetings! I just fixed this issue in my app. In my app, all iOS devices had problems with freezing video or black screen video; I was using the video stream from the device's rear (environment) camera. The solution to the problem was to make sure the <video> element is appended to the DOM. The code I was using created a <video> element but did not add it to the DOM. I believe this is the cause of the problem. As @xavierjs mentioned above, it sounds like you have to use CSS to creatively hide the <video> element, as simply using display: none will bring the problem back. In my app's case, the <video> element appears underneath another element so I didn't have to worry about hiding it. Good luck!

[xavierjs]

I have reopened the issue and pinned it since it seems to be still a real problem.
If you create your own <video> element and use MediaStream API by yourself, yes you have to do the trick mentioned by @namklabs

If you let the lib to request the video (like in the boilerplate and most of the included demos), then we will test if it is an apple device (Safari on IOS or desktop). If yes, we append the <video> element to the DOM and we apply this CSS style:

setTimeout(function(){
  video['style']['transform'] = 'scale(0.0001,0.0001)';
  video['style']['position'] = 'fixed';
  video['style']['bottom'] = '0px';
  video['style']['right'] = '0px';
  
  setTimeout(function () {
    video['play']();
  }, 100);

}, 80);

It is dirty but it seems to work...

[andrejustinnn]

add playsinline on video tag like this "" looks good to me, i notice that my video wasn't appearing n playing in any browsers on iphone. except, including playsinline on the tag. I believe this is the cause of the problem.