why do you project the covariance matrix? instead of projecting the eigen vectors?

[shi-yan]

in the ply file, each vertex has scaling and rotation.

in your code, you first construct the covariance matrix out of those values, and then, in the shader, you project the 3d covariance matrix to 2d and then obtain the eigen vectors again as the 2D basis to construct a rectangle for rendering.

But why don't you obtain 3D eigen vectors from scaling and rotation first and then just project the 3D eigen vectors to 2D?

is it necessary to convert things to the covariance matrix to calculate the projection?

[shi-yan]

nvm, on a secondary thought, if an eigen vector is perpendicular to the screen plane, the vector projection will become a point.

does 2d covariance matrix projection work with perspective matrix? or only orthogonal matrix?

Thank you

[mkkellogg]

right now, only a perspective projection will work -- The shader that renders the splat assumes a perspective projection and builds a Jacobian for one with the specified focal length to form a linear approximation of the projection transformation.

[shi-yan]

TBH, I also don't get the following code

          if (transform) {
                transform3x3.setFromMatrix4(transform);
                transform3x3Transpose.copy(transform3x3).transpose();
                transformedCovariance.multiply(transform3x3Transpose);
                transformedCovariance.premultiply(transform3x3);
            }

where you apply a transform to the covariance. The transform can contain translation, but when you construct a 3x3 matrix out of a 4x4 matrix, the translation will be thrown away?

[mkkellogg]

Yes that is correct. We only want to apply scale and rotation to the covariance. The position of the splat centers will be affected by the translation in the transform in calls to functions like fillSplatCenterArray() so the end result will be correct.

[shi-yan]

Thank you very much for the answers!

I'm attempting to implement gaussian splatting in WebGPU. I noticed both your repo and the huggingface implementation use the same approach.

I did a simple experiment of rendering a single gassian splat and a 3d sphere. The 3d sphere is stretched and rotated by the same scaling and rotation of the splat. I expect them to be perfectly overlapping.

My first implementation creates a 4x4 covariance matrix and applies the 4x4 perspective project matrix. I was able to get okish overlapping (but I guess this is mathematically wrong?).

I then implemented your approach too, but under certain angle, they don't overlap well. I'm not sure what's wrong and still debugging.

Image 1, under certain angle, they overlap well:

Image 2, under other angles, they don't

[mkkellogg]

So my first question would be: which projection transformation are you applying to the covariance? Is it the same perspective projection that is used by the camera, or is it the Jacobian of that projection? Secondly, to properly transform covariance, it's not as simple as a single matrix multiplication. You need to use the following formula:

C' = M * C * Mt

Where C is the covariance matrix, M is the transformation you want to apply and Mt is its transpose.

[shi-yan]

Yes, I know I need to do that, I saw it from here : https://stats.stackexchange.com/questions/484094/projecting-a-covariance-matrix-to-a-lower-dimensional-space

I think I strictly followed your code, but I still get some mis-alignment between the ellipsoid and the projected gaussian.

here is my code using your approach (3x3 covariance) (it needs a webgpu compatible browser.):

https://shi-yan.github.io/WebGPUUnleashed/#5_08_gaussian_splatting_5

it works for most of the angles, but if you view it from certain direction (for example from 10,10,10 to the origin), they mis-align badly.

This is using my approach (4x4 covariance):

https://shi-yan.github.io/WebGPUUnleashed/#5_08_gaussian_splatting_4

This is GPU radix sort + render + my 4x4 approach, the rendering quality is very bad. Still debugging

https://shi-yan.github.io/WebGPUUnleashed/#5_08_gaussian_splatting_6

(github page is a bit slow, after clicking run, it might take a few seconds for the demos to load.)

[mkkellogg]

I'm not very familiar with WebGPU, so I'm not entirely sure what is happening in your code -- Are you by any chance trying to apply to the covariance matrix to the 3D sphere? You should get the correct result if you just apply the original scale & rotation for the splat to the sphere instead, and then apply a uniform scale of roughly 2.0 * log10 of the splat's alpha.

[shi-yan]

Thank you very much! Sorry for bothering again. After set this aside for a while, I'm now trying to inspect your code closely and comparing it with my implementation. I can't make sense of some of the scaling you put in the code:

this.cameraFocalLengthX = this.camera.projectionMatrix.elements[0] *
                                          this.devicePixelRatio * renderDimensions.x * 0.45;
this.cameraFocalLengthY = this.camera.projectionMatrix.elements[5] *
                                          this.devicePixelRatio * renderDimensions.y * 0.45;

in the above code, I don't know what 0.45 is? I think it should be 0.5? half the width and height?

if this formula calculates the focal length, I think cameraFocalLengthX should be the same as cameraFocalLengthY? is there a reason we need to calculate two focal lengths? If I print them out, their values are slightly different, but I think it is due to numerical errors.

secondly, in the shader code where you perform transformations on the covariance matrix, you offset two of the entries by 0.3, I also can't understand this part:

mat3 cov2Dm = transpose(T) * Vrk * T;
cov2Dm[0][0] += 0.3;
cov2Dm[1][1] += 0.3;

what does the 0.3 mean here?

Thank you so much!

[shi-yan]

Thank you. I debugged your code thoroughly and was able to reproduce the correct result in WebGPU. I wrote what I learned here https://shi-yan.github.io/how_to_render_a_single_gaussian_splat/