Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[request] Add possibility to import depth information/estimation from an external source as a pipeline node #2534

Open
imDema opened this issue Sep 14, 2024 · 0 comments
Labels
feature request feature request from the community

Comments

@imDema
Copy link

imDema commented Sep 14, 2024

Problem description

I am having problems reconstructing a scene taken with a not so good camera.
The problem I would like to address is the quality of the depth estimation performed by the DepthMap node.

While trying to tune my pipeline to get a better result, I got to inspecting the depth estimation produced by the depth map node. To my surprise I found the depth estimation to be extremely noisy, to the point that it got me wondering if it were the cause of the bad reconstruction (image below)

Proposed solution

Having experience with neural network methods for depth estimation, I tried computing the same depth estimate using MiDaS and, as expected I got an incredibly better result (image below).

While I do understand that including the MiDaS or another ML based approach for depth estimation in Meshroom could lead to a set of problems and may not be desirable, I think that adding an option to outsource the depth estimation step to an external tool for advanced users could be very beneficial to the quality of the output.
This would also come in handy in cases when depth information for the image is already available thanks to ToF sensors or other, which is also a concern that has been raised in other issues.

Alternatives

  • Adding choices for integrated DepthMap algorithms
  • Adding a node to import depth map information that the user is responsible for computing given the folder
  • Adding the possibilty of inserting a custom command that gets passed as argument input and desired output folder for the depth maps / single input output files
  • Splitting the depth map node in a way that allows advanced users to compute the depth map autonomously and then compute the simMap / nmodMap using meshroom

Additional context
This issue is related to #1493, however I could not find an actual implemented/documented solution to do it even though it is marked as closed.


Here are some comparison between the meshroom output and the MiDaS output run on the same photos (i resized and reencoded them to jpeg manually for this issue)

Sample 1

Meshroom:
51780469_depthMap_Meshroom
MiDaS:
51780469_depthMap_MiDaS

Sample 2

Meshroom:
765218757_depthMap_Meshroom
MiDaS:
765218757_depthMap_MiDaS

Sample 3

Meshroom:
1940366396_depthMap_Meshroom
MiDaS:
1940366396_depthMap_MiDaS

EDIT: this is the script I used to compute the depth maps

#!/bin/bash

if [ "$#" -ne 1 ]; then
    echo "Illegal number of parameters"
fi

for file in $1/*.exr; do
        filename=$(basename "$file" .exr)
        magick "$file" -resize 50% -filter Lanczos "input/${filename}.jpg"
done
python3 run.py --model_type dpt_beit_large_512 --input input/ --output output/ --grayscale

SUFFIX=-dpt_beit_large_512.png

for file in output/*${SUFFIX}; do
        filename=$(basename "$file" ${SUFFIX})
        magick "$file" -compress Lossless "exr/${filename}_depthMap.exr"
done
@imDema imDema added the feature request feature request from the community label Sep 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request feature request from the community
Projects
None yet
Development

No branches or pull requests

1 participant