About how is the tracking mesh built?

[Luh1124]

Hello, I have a new problem: I am trying to use the video data of my head to reconstruct mesh and texture as the input and supervision of the network. I am trying to use colmap to reconstruct the point cloud and retargeting it to a public model with 7306 vertices; But the effect is very poor. Is there any method to build mesh for reference?

[Luh1124]

[stephenlombardi]

Are you trying to capture and train on just a single frame or a set of frames?

If you're trying to capture and train multiple frames then you need a topologically-consistent mesh over time. You might try a method like FLAME to achieve this.

If you're just using a single frame, you can create a hole-free mesh by using Poisson reconstruction (you can do this in meshlab). You'll also need to generate UV coordinates for the mesh (you can do this in Blender).

[Luh1124]

We collected multiple photos of the neutral head with our mobile phone, and used 3DF, Maya, and reality capture to build a texture map. But there should be a better method than colmap when reconstructing human heads from multi view images. Do you have a more recommended tracking mesh generation method to align with the effect of your paper?

[stephenlombardi]

I'm still not clear if you're trying to attempt this on a static scene or a dynamic scene. COLMAP doesn't do tracking, it just does 3D reconstruction. In the MVP paper we use a method for tracking based around Wu et al., 2018 which uses a simple region-based PCA model to do the initial tracking and then later refines that tracking. However, we have many viewpoints captured simultaneously to do this and I'm not sure if it's applicable to a mobile phone scan. I haven't tried it but like I mentioned previously FLAME may be a good place to start.

[Luh1124]

Sorry, I want to try to reconstruct the dynamic scene; I'll try it first according to the literature you mentioned in the mvp. Thank you

[Kartik-Teotia]

@stephenlombardi if we're to use a linear model like FLAME, won't we lose out on both high-fidelity geometry and lose out even more with respect to appearance/texture conditioning as you can only expect to get good texture maps if you have a good approximation of the underlying geometry, with linear models, we'll hardly recover anything outside of the face region itself. I don't believe there is a good open-source Head tracking alternative to what Meta Reality labs has built in house :D. Perhaps an alternative could be to use FLAME for primitive initialization and use images as conditioning for the encoder.