When orientation_method=pca, transformation_matrix can be incorrect

[ChenYutongTHU]

Hi NeRFStudio's contributors,

Thanks for your work!
I notice that when setting ColmapDataParserConfig.orientation_method=pca, a misalignment between initial pointcloud and camera pose can happen due to the incorrect transformation_matrix

In my example, I rendered the COLMAP initialized point cloud in step 0 but found that it was misaligned with the groundtruth image. Specifically, the image is vertically flipped.

I found that the issue relates to these lines

nerfstudio/nerfstudio/cameras/camera_utils.py

Lines 577 to 580 in f86dbe6

	oriented_poses = transform @ poses
	if oriented_poses.mean(dim=0)[2, 1] < 0:
	oriented_poses[:, 1:3] = -1 * oriented_poses[:, 1:3]

'oriented_poses.mean(dim=0)[2, 1] < 0: ' seems to mean where the Camera's y-axis (up-direction) is projected to the world's z-axis. So the up direction in the image should be aligned with +z axis in the world. If not, the code here negates the y and z axes in the camera's coordinate. However, this leads to the flipping renderings, misaligned with the label images which are not flipped.

If I turn off this negation, the step=0 renderings become aligned again.

To address this issue, we need to apply the transformation to the world coordinate (Flip the objects in y and z axes) instead of the camera system. For example

        if oriented_poses.mean(dim=0)[2, 1] < 0:
            #oriented_poses[:, 1:3] = -1 * oriented_poses[:, 1:3]
            transform_plus = torch.eye(3)
            transform_plus[1, 1] = -1
            transform_plus[2, 2] = -1
            oriented_poses = transform_plus @ oriented_poses
            transform = transform_plus @ transform

This will lead to much better final results due to the correct initial point clouds.

Great thanks :)

[jb-ye]

Indeed, this seems like a bug.

Could the following change also do the same job on your dataset?

oriented_poses[1:3, :] = -1 * oriented_poses[1:3, :]

Can you send a PR to fix it? Thanks for your contribution.

[ChenYutongTHU]

Indeed, this seems like a bug.

Could the following change also do the same job on your dataset?

oriented_poses[1:3, :] = -1 * oriented_poses[1:3, :]

Can you send a PR to fix it? Thanks for your contribution.

Hi. In addition to transforming the C2W matrix, we also need to modify the transform matrix accordingly, which will be used to transform the input cloud.

            oriented_poses[1:3, :] = -1 * oriented_poses[1:3, :]
            transform[1:3,:] = -1*transform[1:3,:]

Thanks!