add ar2Tracking to fixing-nft

[misdake]

1. revert 5aa10bd to restore trackingMod file.
2. add ar2Tracking2dSub implementation to tracking2d.c, and add usage to trackingMod.c.
3. add detectedPage logic to ARTookKitJS.cpp.
4. load ar2Handle in setupAR2.

There is no change to the original artoolkit5 repo, because I've copied ar2Tracking2dSub/ar2CreateHandle/ar2CreateHandleSub into this repo.

note:
I built js & wasm files in windows environment based on this modified makem.js file here, based on here. This modified file is not in this pr.
I added tracking2d.c to maken.js in this pr, you should be able to rebuild with it in linux/mac environment.

EDIT: (~nicolocarpignoli) I will report original @misdake posts in other thread, with live demos:

See below for demo:
https://rgbuv.xyz/jsartoolkit/video.html (single-threaded)
https://rgbuv.xyz/jsartoolkit/video_worker.html(offload single-threaded work to a worker to keep main thread smooth)

This demo uses 320*240 canvas as process input, getting ~300ms/frame for kpmMatching and ~30ms/frame for ar2Tracking. Snapdragon 845 CPU.
screen capture: https://imgur.com/a/91NdUcy). nft: https://imgur.com/a/wXar8T2.
screen capture: https://imgur.com/a/iUYc2cj nft: https://imgur.com/a/qbhANXr
(edit: replaced nft due to possible copyright issues)

[nicolocarpignoli]

Ok, I will re-compile it on unix (I have mac) and do some tests with the file we are testing with before (nft_threejs.html and similar). My tests will be done with smartphones (android for now, then also ios), the most useful use case.

Let you know, thanks in advance for your contribution!

[nicolocarpignoli]

Unfortunately, the first test did not give good results.
There's a big lag during the camera stream. Maybe I have done something wrong?
These are my steps:

• started from this branch
• npm run build (on mac) - build is ok
• show pinball image on my mac (https://github.com/kalwalt/jsartoolkit5/blob/fixing-nft/pinball.jpg)
• start a local web server and open my smartphone (android 9) to the webpage of https://github.com/kalwalt/jsartoolkit5/blob/fixing-nft/examples/nft_threejs.html

@misdake do you have time to reproduce this very same test, with a smartphone, and tell me if you got good performances, as we saw on your videos from the other discussion (by the way, they seems sooo promising)?

thank you so much

[nicolocarpignoli]

I'm on mac, I do not have edited makem.js or anything else, just the steps above.

[misdake]

Lag comes from slow kpmmatching runs. if it is tracking, frame rate will be much higher.
take a look at the HTML and js file of the demo. video feed is shown natively as video element, and on top of it is a canvas overlay. they run asynchronously, so the camera stream will never lag.

[misdake]

@nicolocarpignoli see comment above for the lag issue.
and if you want js thread not blocked by processing, do it like the video_worker.html and main_worker.html, I offloaded all processing to a webworker. 3d stuff can run super smoothly. (remember to use transferable to send camera stream to worker, I didn't do that because of my target platform doesn't support it)

[kalwalt]

Hi @misdake thank you for this PR. I hope to test this soon.
@nicolocarpignoli could be this lag depends by the type of device?

[kalwalt]

@misdake i can see that there is a worker.js file in your example https://rgbuv.xyz/jsartoolkit/video_worker.html but i can not see in the PR...

[kalwalt]

@misdake can you update the first post with all the examples you provided?

[nicolocarpignoli]

Yea, uploading all the examples file will be great!

[misdake]

I've cleaned the code, here is the source of my demos.
demo.zip
(edit: replaced nft due to possible copyright issues)

Content also got uploaded to server, links:
camera input, single-threaded
camera input, webworker
video input, single-threaded
video input, webworker

[misdake]

Demos are updated with threejs rendering (sphere): (index)

screen capture: https://imgur.com/BciDGxo

demo: camera input, worker, threejs rendering
demo: video input, worker, threejs rendering

code: jsartoolkit.zip

[kalwalt]

Thank you @misdake i will try as soon as i can!

[nicolocarpignoli]

Thanks @misdake ! We can try and do some tests, if works great we can merge this PR, and continue testing/working on this.

I will not delay too much this merge so we can focus on just ONE version

[nicolocarpignoli]

reporting misdake implementation details by him:

Some implementation details of the demo are listed below. they may not be necessary, well, at least it works.🤔

1. I keep the canvas passed into ar.process to have aspect ratio 4:3 to fit the camera params. This canvas is called "canvas_process" . (You can see a lot of my code is just calculating width/height.)
2. Camera/video input is scaled down to boost performance. then draw the small input image onto the 4:3 "canvas_process" (black on left/right side). (also lots of code).
3. I modified camera projection matrix to fix aspect from 4:3 to canvas_draw. (see threejs_worker.js:107)
4. Transferring the matrix through postMessage is very slow for whatever reason, at least on PC. Stringify + postMessage + parse, saves you quite some time (it becomes an array-like object, see threejs_worker.js:17).
5. Size of marker in world space is calculated in threejs_worker.js:160.
6. vw/vh: video; sw/sh: render; pscale: process scale; sscale: render scale; w/h: process size w/o black; pw/ph: process size with black; ox/oy offset to draw video w/h to process pw/ph.

When testing for performance, I see 70ms of postMessage({projmatrix, worldmatrix}) delay from worker to main thread. I don't know why.
Switch to the stringify+postMessage+parse route, this delay disappears.
That's the reason for:
worker thread: postMessage({type: "found", matrixGL_RH: JSON.stringify(ev.data.matrixGL_RH), proj: JSON.stringify(cameraMatrix)}).
main thread: let world = JSON.parse(lastmsg.matrixGL_RH);

[kalwalt]

Hi @misdake the last example works very good! ( at least on Android) can you push all your examples in this PR?
Just create a folder ( for example nft_improved ) and put the example there. After we think we can merge this ( @nicolocarpignoli maybe also solving the iPhone issue?..)

Edit: maybe nft_improved_worker is better

[kalwalt]

@misdake I pushed some of the examples in this branch but not sure that there are all your changes, can you take a look at? In this way it should be easier for you adding the examples...

[kalwalt]

Many thanks @misdake! i will test/check this evening!

[commentatorboy]

HOLY SHIET! It almost works. It works better, but it has a bit hard time to show it.
https://user-images.githubusercontent.com/1793964/70346711-3813c180-185f-11ea-82d5-27fa5ec0d733.png (works)
https://user-images.githubusercontent.com/1793964/70346719-39dd8500-185f-11ea-92e2-435ae6c33175.png (does not work)
It even works on somewhat extreme angles (after the marker has been recognized).

I tested this on firefox, and it is the same in my chrome.

My smartphone is:
moto e3, android 8.0.0, firefox 68.3.0, chrome 78.0.3904.108

Like there needs to be either very great lighting and no disturbances to the picture.

[nicolocarpignoli]

@commentatorboy I found great results in androids. Yes, and i'd like to say to everybody: don't mind the initial lag. It's slow because it's loading image descriptors (aka the data that needs to recognize a specific image, when scanned). It's not a great time, that computation is heavy, in fact.
We will for sure insert an optional (active as default) loader, a nice loading page/loader that disappears as soon as the descriptors are loaded.
This is actually common for a lot of AR apps (e.g. it happens also on mobile apps when they need to recognize a room before placing objects, altough it's a different thing, it's just an example of UX hacks for AR).

Will let you know about iphones bad performances

[kalwalt]

HOLY SHIET! It almost works. It works better, but it has a bit hard time to show it.
https://user-images.githubusercontent.com/1793964/70346711-3813c180-185f-11ea-82d5-27fa5ec0d733.png (works)
https://user-images.githubusercontent.com/1793964/70346719-39dd8500-185f-11ea-92e2-435ae6c33175.png (does not work)
It even works on somewhat extreme angles (after the marker has been recognized).

I tested this on firefox, and it is the same in my chrome.

My smartphone is:
moto e3, android 8.0.0, firefox 68.3.0, chrome 78.0.3904.108

Like there needs to be either very great lighting and no disturbances to the picture.

Thanks for testing @commentatorboy ! 😄
It must load the nft marker and yes this take a lot of time, for this we are thinking to add a loader as in the nft_worker/index.html example that @nicolocarpignoli did, That not speed up the code but at least the user know that is loading.
BTW the two images are referring to the two different browsers? or what?

[kalwalt]

Didn't see your comment @nicolocarpignoli because i didn't refreshed the page... your explanantion is better than mine!

[nicolocarpignoli]

Closing this one: modifies are now on fixing-nft branch (principal one). We can continue discussin on that.

[commentatorboy]

@kalwalt No it is the same browser it was Firefox. Just showing that it some times worked