Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add software to filter SfM data and to create landmarks-based masking for depth map computation #1526

Open
wants to merge 37 commits into
base: develop
Choose a base branch
from

Conversation

almarouk
Copy link
Member

@almarouk almarouk commented Sep 6, 2023

Linked to alicevision/Meshroom#2211

Description

This PR introduces the following two main features:

  • Depth maps computation can be masked using landmarks from SfM data.
  • SfM data can be filtered to efficiently be used for depth maps masking.

The SfM data filtering supports the following features (the parameters are independent):

  • Filtering landmarks over 3 steps:
    • Step 1: Remove landmarks that have an insufficient number of landmarks.
      Parameters: minNbObservationsPerLandmark
    • Step 2: Remove landmarks with dissimilar observations compared to their neighbors in 3D.
      Parameters: nbNeighbors3D, minNbObservationsPerLandmark
    • Step 3: Remove landmarks with worse resolution than their neighbors in 3D. The resolution is based on pixel size.
      Parameters: radiusScale, useFeatureScale
  • Filtering observations to select best observations per landmark and to ensure consistency with neighbors in 3D.
    The observations are scored in function of distance between corresponding camera centers and landmarks positions.
    The scores are iteratively propagated between neighboring landmarks in 3D.
    Parameters: maxNbObservationsPerLandmark, propagationEnabled, nbNeighbors3D, nbIterations, dampingEnabled, dampingFactor
  • Filtering observations (and eventually landmarks) to ensure non-sparsity in 2D projection of landmarks separately in each view.
    Mask radius for projected landmarks per view is estimated and reported in an output text file (outputRadiiFile).
    Parameters: nbNeighbors2D, percentile, maskRadiusThreshold

Future Work

  • fix duplicate landmarks' biased influence.
  • take into consideration the angle of observation when scoring the observations in addition to the distance to camera.
  • include blur estimation.
  • have a more complex relative estimation for neighborhoods (e.g. elliptic neighborhood with orientation, based on landmarks segmentation/classification for surface information...).
  • verify if modifications are required in case of multiple capturing devices.

@almarouk almarouk force-pushed the dev/maskDepthMapFilterSfm branch 13 times, most recently from 6851026 to b7ff339 Compare September 12, 2023 15:24
@almarouk almarouk force-pushed the dev/maskDepthMapFilterSfm branch 3 times, most recently from 7d09e3a to 21105d0 Compare September 25, 2023 07:53
@almarouk almarouk force-pushed the dev/maskDepthMapFilterSfm branch 6 times, most recently from 3002139 to 718224d Compare September 27, 2023 09:40
@almarouk almarouk marked this pull request as ready for review September 27, 2023 09:42
@almarouk almarouk changed the title Dev/mask depth map filter sfm Add software to filter SfM data and to create landmarks-based masking for depth map computation Sep 27, 2023
filter unstable landmarks that have an insufficient number of observations (specified as a parameter)
If a 2D projected observation in a view doesn't have enough other
neighboring observations, it will not be considered for creating
the depth map mask
- estimate the mask radius dynamically for each view
based on average distance to neighbors
- filter 2D observations based on quantile on distribution
of average distance to neighbors
a point is neighbor to itself if it participated in creating the tree
remove landmarks that don't have enough common observations with their neighbors
- add a re-distribution of observation scores at each iteration by favouring best N observations
that will be selected at the end
- fix hard-coded parameters
- update parameters and code structure for a clearer understanding of different steps
- fix bug with damping renormalization
- add comments
- resolve duplicate features issue
- do not normalize propagated neighbors scores
- fix NaN MSE issue
- fix radius estimation caused by quantile issue
- add option to propagate neighbor observations
- use Jaccard index in step 2
at the end of 3D Observations filter
- create a fake observation by projecting the landmark in the view
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant