Debugging Mapping on Custom Video

[ehratjon]

Hey everyone! Nice to see a community maintained SLAM project with active discussions :)

TL;DR:
Trying SLAM on mobile phone video. Only keypoints detected, no landmarks -> no map generated.
Any tips or ideas? Best approaches for debugging?

Description:
I am just diving into the code of this repository. I finished reading through the documentation and many discussions. The tutorial example is running on all viewers, tested on the aist_entrance_hall examples.

Now, I'd like to get the SLAM system running on a video file recorded on my mobile phone, a Samsung Galaxy A23.

./run_video_slam -v orb_vocab.fbow -c video_dir/config.yaml -m video_dir/video.mp4

shelf-iridescence_key_point_cut_sized.mp4

When running the video with estimated intrinsic parameter, no map was created. So I followed the openCV Python camera calibration turorial and got the correct focal length, camera center and distortion parameters.

Current YAML Config:

# Perspective model for Android Phone A23 

#==============#
# Camera Model #
#==============#

Camera:
 name: "Samsung A23"
 setup: "monocular"
 model: "perspective"

 fps: 30
 cols: 1080
 rows: 1920

 fx: 1765.962
 fy: 1780.006
 cx: 547.620
 cy: 928.668

 k1: 0.2889
 k2: -1.3129
 p1: -0.0002
 p2: 0.0051
 k3: 1.6594

# Half-size parameters
# cols: 540
# rows: 960

# fx: 882.21
# fy: 889.13
# cx: 271.63
# cy: 462.36

# k1: 0.2862
# k2: -1.3603
# p1: -0.0013
# p2: 0.0045
# k3: 1.8548

Camera.color_order: "RGB"

#================#
# ORB Parameters #
#================#

Feature.max_num_keypoints: 2000
Feature.scale_factor: 1.2
Feature.num_levels: 16
Feature.ini_fast_threshold: 20
Feature.min_fast_threshold: 3
Feature.mask_rectangles:
 - [0.0, 1.0, 0.0, 0.1]
 - [0.0, 1.0, 0.84, 1.0]
 - [0.0, 0.2, 0.7, 1.0]
 - [0.8, 1.0, 0.7, 1.0]

Preprocessing:
 min_size: 300

Mapping:
 backend: "g2o"
 baseline_dist_thr_ratio: 0.01
 redundant_obs_ratio_thr: 0.90
 num_covisibilities_for_landmark_generation: 10
 num_covisibilities_for_landmark_fusion: 10
 residual_deg_thr: 0.6

Tracking:
 backend: "g2o"

As seen in Feature extraction and projections #411 there are different parameters that have an influence on key points and landmarks. I tried playing with the parameters listed under Config.Mapping, as well as using a smaller video size.

While there are key points found in the video, they are not stable enough (?) to lead to landmarks and mapping.
It seems that the key points are oddly shifted to the right, even with the calibrated camera.

Questions:
Does anyone have an idea why the mapping fails and what needs to change in order for it to work?
Is there a good approach to debug such an issue?

(Currently I am exploring the code in run_video_slam and getting information about the key points and landmarks.)