SFM: Known intrinsic parameters

[hcjghr]

Hi,

I was looking at the code but I was not able to find any way to introduce known intrinsic camera parameters with the images. Did I just overlooked it, or for now there isn't any efficient way to do it before computing SFM reconstruction?

Thanks

[simonfuhrmann]

Currently, there is indeed no obvious way to introduce known camera intrinsics. The focal length is obtained from the EXIF tags, and principal point is set to the image center. One would have to inject the known focal length into EXIF and use --fixed-intrinsics, but that is a crappy solution.

I have to look into what must be done to initialize the intrinsics from the views. In combination with --fixed-intrinsics, this should do the job. Then it's up to you to put your known intrinsics into the views before SfM reconstruction.

[simonfuhrmann]

This feature request raised a few design issues. Most importantly, we need to normalize feature coordinates which is a larger change in the SfM library. Also, the prebundle.sfm file might change to a matching.sfm file. This suggests that we implement this change before the release of MVE2.

[simonfuhrmann]

Prerequisite issue #126.

[simonfuhrmann]

A feature --intrinsics-from-views has been added to sfmrecon obtain intrinsic parameters from the views. Currently focal length is the only intrinsic parameter that is supported (camera.focal_length in the meta.ini file).

Is this already useful for you in the current state?

[Sudhir1990]

Hello,

I am looking for a solution to include known intrinsics(all intrinsics along with focal length).

Is it possible to include known intrinsics other than focal length as well in the EXIF info? If yes, what should be the format of exif info? Also can we include approximate positions?

Many thanks!

[simonfuhrmann]

Only the focal length can be provided in the EXIF info. A precise focal length can be provided in the meta.ini file. Radial distortion, however, can not be provided at this time. We are working on that.

[tobycollins]

A solution to handle arbitrary intrinsics could be as follows. You warp each image using its calibration intrinsics, so that the warped image has a canonical intrinsic matrix of [f, 0, imageWidth/2; 0; f; imgHeight/2; 0, 0,1]. Then you just plug f into the EXIF tag, and it's done.

[simonfuhrmann]

This will, however, still optimize the intrinsics if --fixed-intrinsics is not specified. Also, unfortuantely, EXIF only supports integer values for the focal length, IIRC.

[tobycollins]

Hi - yes I assumed --fixed-intrinsics would be set :-) Regarding the fact that EXIF only supports integer values, it's not really a problem. The canonical intrinsic matrix can have an arbitrary f (and so can be set to an integer). This could be done by rounding the original intrinsic matrix's fx value.

So the workflow could look something like:

1. Undistort the input image to eliminate effects of lens distortion. This can be done with e.g. opencv's undistort method. Let's call this undistorted image I. This has an intrinsic matrix K (which is the same as the original input image).
2. Define a canonical intrinsic matrix K2 = [f,0,w/2;0,f,h/2,0,0,1] with f = round(K(0,0))
3. Warp I using a homography H, so that the warped image's intrinsic matrix is K2. To do this you use H = K2*inv(K). Let's call the warped image I2.
4. Set the focal length in the EXIF tag to f and specify --fixed-intrinsics, then perform SfM on the warped images.

The nice thing about this is that beyond some image pre-processing there would be no significant code/design changes. One thing to point out is that when you undistort an image and warp it with a homography there can be visible borders that correspond to the perimeter of the original image. These borders may produce spurious features and it could be necessary to remove them.

Hope this helps.

[simonfuhrmann]

That's an interesting approach. If you set f slightly larger than initially, the homography H will essentially crop a little bit the image thus preventing visible borders.

[drbuckingham]

I added camera.focal_length = 20 to the meta.ini file of all the views in the scene, but every view reported "Warning: View X has zero focal length. Using fallback.". I am new to the process, so perhaps I am missing something obvious to others.

[simonfuhrmann]

First, focal_length = 20 doesn't sound right. The focal length is given in normalized format, e.g., if you have a 70mm lens on a 36mm sensor, you'd pass 1.9444. So you need to know your sensor size, which is the whole point of MVE having a camera database. (If your camera is missing from that database, feel free to add it or provide an EXIF dump.)

Second, you also have to pass --intrinsics-from-views to tell MVE to use the values you provided in the meta.ini file. Otherwise it will try to obtain the information from EXIF.

[drbuckingham]

Based on comments in the threads discussing focal length, I entered the "FocalLengthIn35mmFilm" value which is 20 (mm). I have attached an exif dump of one of the pictures. I did use the --intrinsics-from-views option as instructed in this thread, after adding the text line "camera.focal_length = 20" to each meta.ini file. The camera is a stock DJI Phantom 3 Advanced camera.
exif-dump.txt

[simonfuhrmann]

You would still have to provide 20mm/35mm = 0.6666 as focal_length.

[simonfuhrmann]

On another note, MVE should pick up on the EXIF field FocalLengthIn35mmFilm:
https://github.com/simonfuhrmann/mve/blob/master/libs/mve/image_exif.cc#L365
Do you mind checking why it's not working for you?

[drbuckingham]

Should the line added to meta.ini be "camera.focal_length = 0.6666" or "focal_length = 0.6666". I have seen it referenced both ways now.

As far as checking on the FocalLengthIn35mmFilm being ignored, how do you want me to assist you (be specific)?

[drbuckingham]

meta.zip
I have tried both "camera.focal_length = 0.6666" and "focal_length = 0.6666" added to the end of the meta.ini files, both times using --intrinsics-from-views and neither time the value was used, yielding "Warning: View 0 has zero focal length. Using fallback." What am I missing? I have attached the meta.ini file of the last test.

[simonfuhrmann]

You have to put focal_length = 0.6666 under the [camera] section.

[HelliceSaouli]

PLease change the Title to SFM: Known intrinsic parameters and camera pose; Will be nice to have this __

[simonfuhrmann]

@HelliceSaouli This issue is about SfM. If you know the camera pose, you don't need SfM. You just need a way to generate 3D features.

[HelliceSaouli]

Don't be mad at me ^^

[nitrotm]

See pull request #431 for a draft of this feature.

[jhaggle]

I understand from above that if you add intrinsics in meta.ini, then you need to run sfmrecon with "--intrinsics-from-views". But should you then also always set --fixed-intrinsics??

[simonfuhrmann]

I think if you specify --intrinsics-from-views, it'll load the intrinsics from the view instead of assuming default ones. Additionally, if you specify --fixed-intrinsics, it will not attempt to optimize them. The only parameter that is optimized during SfM is focal length. So if you're absolutely certain your focal length is correct, you can specify that flag, but otherwise I usually wouldn't, as things like focus point can change focal length slightly, and you may get better results by letting the optimizer change it.

[jhaggle]

Before running sfmrecon, the meta.ini-files contain only the following info:

[view]
id = 0
name = 4103691902_000

So I suppose I should loop over the meta.ini-files and add [camera][focal_length] and [camera][principal_point] to the files BEFORE running sfmrecon if I want to use beforehand known values? But when I do that, the added values are overwritten when sfmrecon even though I specify "--intrinsics-from-view". So how should it be done??

[simonfuhrmann]

That's right, the meta.ini file should look something like this:

[camera]
focal_length = 1.05427

[view]
id = 0
name = 0000

Note that focal_length is normalized by the image size. So if you have the focal length in pixels, you'd compute focal_length = focal_length_in_px / max(image_width, image_height). If you specify --fixed-intrinsics, the intrinsics won't change after sfmrecon. If you don't specify it, yes, it will be overwritten with the optimized focal length value. You can create "backup copy" if you want to.

[jhaggle]

This does not work for me. The meta.ini-files are overwritten during sfmrecon even though I specified --intrinsics-from-views and --fixed-intrinsics.

I start with mve.makescene -i input_dir output_dir. This creates folders with meta.ini-files, one for each image.
The content of the meta.ini-files at this points looks like this:

I then run a script to add my own intrinsics. I also add a variable test just for testing purposes. After running my script the meta.ini-files looks like this:

Next, I run the sfm pointcloud reconstruction with mve.sfmrecon scene_x/ --intrinsics-from-views --fixed-intrinsics. This overwrites the intrinsics that I added to the meta.ini-files, as you can see here: