large microlens artifacts using render_view_3g and Raytrix R5

[crvogt]

With a good depth and confidence map, foregrounds in rendered images show full microlens, seemingly independent of available arguments.Tried feeding an inverted depth map as the artifacts are only in the foreground, this didn't work. Problem is specific to new view renderer as previous versions did not show these artifacts.

[PlenopticToolbox]

Thanks for the notification!
You are right. I believe the issue is in the rendering process, because I need to know the relation between lens type and position in the scene (near, middle, far).
This should be read from the .xml file as best practice I think, and it was hard coded for my case for the R29.
Can you try to change the render_SI method in rendering/render.py at line 1522?
https://github.com/PlenopticToolbox/PlenopticToolbox2.0/blob/master/python/rendering/render.py#L1521

The code is the following

    # for the R29 dataset
    range_t0 = [0.400, 1]
    range_t1 = [0, 0.200]
    range_t2 = [0.200, 0.400]

So the disparity is divided into three slices. In this case t1 is for small disparities, t2 medium and t0 for larger values. I will work on a small method that reads these values from the .xml file.
There they are in form of virtual depth.
In a typical raytrix .xml files, they are shown as

        <lens_type id="0">
		<offset units="lens">
			<x>0</x>
			<y>0</y>
		</offset>
		<depth_range units="virtual_depth">
			<min>1</min>
			<max>3</max>
		</depth_range>
	</lens_type>
	<lens_type id="1">
		<offset units="lens">
			<x>1</x>
			<y>8.74227765735e-008</y>
		</offset>
		<depth_range units="virtual_depth">
			<min>2.79999995232</min>
			<max>4</max>
		</depth_range>
	</lens_type>
	<lens_type id="2">
		<offset units="lens">
			<x>-1</x>
			<y>8.74227765735e-008</y>
		</offset>
		<depth_range units="virtual_depth">
			<min>3.79999995232</min>
			<max>100</max>
		</depth_range>
	</lens_type>

And they also have some overlapping. The mapping between virtual depth and disparity is also calibration depending, so it may be tricky. But a linear mapping could be enough to render without artifacts.
If you have time and want to try, you can change the values in the code and let me know which values are correct for the R5, in the meanwhile I will try to automate the method reading and converting from .xml files.

[crvogt]

Thanks for getting back to me! Yeah, my .xml looks pretty much identical to that. I ended up trying this:

scale_fact = 0.004
print("scale_fact: {}".format(scale_fact))
range_t0 = [0.400 * scale_fact, 1 * scale_fact]
range_t1 = [0 * scale_fact, 0.200 * scale_fact]
range_t2 = [0.200 * scale_fact, 0.400 * scale_fact]

which gave about the best results I could get, but still pretty significant artifacts. I assume this is what you meant by a linear mapping?

[PlenopticToolbox]

Hey. Sorry for getting back later.
No, I did not mean exactly that. There are a couple of things more.
First, the pairing of the lens type. The lens type 0 could be the lens focused on the "near" part of the scene on R29, but the lens focused on the "far" part of the scene on the R5. I hope (and think) this is not the case, but it may be worth checking. I assumed the grid was consistent across cameras, but otherwise that would explain the artifacts.
That could be solved by switching range_t0, t1 and t2 between them by matching the correct lens type. In the visualization folder you find a script to show the lens types.
I am not sure if the names coincide with the .xml files, now looking at that I have some doubts. I thought lens types 0 in the code is the same as lens type 0 in the .xml, but that may be a bug. In the cameras I used (R42 and R29) the grid was consistent, so I did not encountered problems. But I may need to fix it for R5.

About the scaling, I believe this is relative to the configuration.
You have the xml file telling you about virtual depth (1 to 100, but actually used is 2 to 8 more or less) and you need to convert this into disparity. So a virtual depth of 2 means that one object is visible in two horizontally adjacent lenses, so the disparity should be more or less half of the diameter of the lens. A higher virtual depth, let's say 8, relates to a very small disparity (if the object is repeated in 8 lens, the disparity between each lens must be smaller than 1/8 of the diameter). This mapping should be applied (microlens with low virtual depth are used for areas with high disparity and viceversa).

[crvogt]

I compared the lens types you assign with the output type per lens from RXLive, and they do match. However, I suppose it's possible Raytrix changed the focus distance definition per lens type, but this seems very unlikely. I'm not entirely sure what to try changing. I understand the virtual depth concept, not where it comes to play in the rendering code, though maybe the trick is to just go through the code finely once I have the time. The old code works alright, but it seems like so much work's been put in to the current version it'd be a shame to not use it.
Either way, thanks for getting back to me!

edit: do you mean scaling the disparity map?

[PlenopticToolbox]

Ok. If the microlens type match, it should work. the disparity scaling should be just a minor effect.
To clarify:
1 - we need to check if t0, t1, t2 are actually the same as raytrix (it maybe that the .xml file calls lens type 0 what we call lens type 1. Since the actual values of the focal lengths are not known, names are relative. Maybe I should write a legend). Once this is cleared,
2 - we check the disparity range

    # for the R29 dataset
    range_t0 = [0.400, 1]
    range_t1 = [0, 0.200]
    range_t2 = [0.200, 0.400]

This part assumes that lenses considered in the code as t0 (lens type 0) should be used to render the part of the image with large disparities, so with small virtual depth. So in the .xml code you should find lens type 0 with virtual depth 1 to 3 or similar. Lens type 1 is good for small disparities, so high virtual depth, so in the .xml should be virtual depth 5 to 100 or similar. Lens type 2 is in the case the midlde one, so virtual depth 3 to 5? If this is actually the case, then I don't know why this is wrong, it should work.

Can you share the image and the .xml file (if you have?), that would help

[crvogt]

proc.pdf

[crvogt]

[crvogt]

Hope that worked, assuming github doesn't apply some drastic compression. The xml looks nearly exactly like the one you posted, so I'm curious to see where you're gleaning information from/how you modify the code with that information. (proc.pdf is the xml code, github wouldn't let me upload it otherwise)

[PlenopticToolbox]

thanks. I hope there won't be compression,otherwise it may not work. I will try my best to find some time to make it work

[crvogt]

Thanks! If it does compress, let me know and I can email it to you.

[lvgeng]

Same issue here. Looking foward to a good solution!

[PlenopticToolbox]

I took a quick look, I think the size of the patch chosen is related to the maximum disparity that the algorithm used, and it should not be that way. It should be related to the lens diameter. Otherwise different executions may get different results. Sorry for that. By changing the maximum disparity value, I was able to obtain one image with almost no artifacts. But it should be corrected, and I am also not sure about the lens types. I am working on that, hopefully soon I can give you an updated code and some images.

[lvgeng]

I took a quick look, I think the size of the patch chosen is related to the maximum disparity that the algorithm used, and it should not be that way. It should be related to the lens diameter. Otherwise different executions may get different results. Sorry for that. By changing the maximum disparity value, I was able to obtain one image with almost no artifacts. But it should be corrected, and I am also not sure about the lens types. I am working on that, hopefully soon I can give you an updated code and some images.

Wait... I thought the patch size is supposed to be related to disparity (not the maximum disparity) according to the focused plenoptic camera theory?

[PlenopticToolbox]

I took a quick look, I think the size of the patch chosen is related to the maximum disparity that the algorithm used, and it should not be that way. It should be related to the lens diameter. Otherwise different executions may get different results. Sorry for that. By changing the maximum disparity value, I was able to obtain one image with almost no artifacts. But it should be corrected, and I am also not sure about the lens types. I am working on that, hopefully soon I can give you an updated code and some images.

Wait... I thought the patch size is supposed to be related to disparity (not the maximum disparity) according to the focused plenoptic camera theory?

Yes. That's why you need the disparity as input for the rendering script. I tried to explain above and in the sample page the relationship between disparity and patch size. Patch size is related to the disparity per each lens (if it would be ONLY related to the maximum disparity all patches would the same!), but there has to be a value for that relationship (disparity -> patch size) It ideally would depend on a metric calibration, but without that I have to rely on some other values, that was in this case the maximum disparity or the diameter lens.
Sorry if this was misleading, I hope you read the whole discussion and not just the last sentence. That was meant as a hint for you in case you wanted to try it on your own. I will then wait until I figured it out completely and come back with the full solution instead, if this is a better way. You may need some patience then, I will do my best.

[freerafiki]

I tried to make a notebook out of my thoughts on the problem and uploaded here, hope this helps.

Issue is not solved, but image does not have huge artifacts, only some imperfections in the upper part. I have the feeling also the image uses only a part of the depth of field, making it harder for disparity estimation and rendering.

(it may take some time to load, because there are a lot of images, but should work - maybe take a look at it online that you already have all the output and then try to use it locally)

[crvogt]

I tried to make a notebook out of my thoughts on the problem and uploaded here, hope this helps.

Issue is not solved, but image does not have huge artifacts, only some imperfections in the upper part. I have the feeling also the image uses only a part of the depth of field, making it harder for disparity estimation and rendering.

(it may take some time to load, because there are a lot of images, but should work - maybe take a look at it online that you already have all the output and then try to use it locally)

Hi thanks for that! I'm trying it on my local system. I thought I'd raise the issue that when running render_SI with sample_per_lens = 7, I get the error:

line 1380, in render_SI
patch_mask[ptc_xx2 + ptc_yy2 < hsp**2] = 1
IndexError: boolean index did not match indexed array along dimension 0; dimension is 7 but corresponding boolean dimension is 8

And when I change to sample_per_lens = 8
line 1403, in render_SI
conf_masked = conf[_x-_radius:_x+_radius+1, _y-_radius:_y+_radius+1] * mask
ValueError: operands could not be broadcast together with shapes (23,23) (24,24)

I haven't tracked it yet, but it looks like something is pulling the wrong diameter or rounding differently. Did you not encounter this?
Thanks!

[freerafiki]

it is a problem in the rounding, that I thought I already solved. I had two different ideas, and was not sure which one to follow. Maybe I got the wrong one. I will try to correct it soon, thanks for letting me know.

[freerafiki]

I pushed a little change to fix the rounding. The notebook should work fine, since now local and remote version are synchronized.
However, there are other parts of the code that are not ready or needs more work in other files. I am working on that.

[crvogt]

Awesome, thanks for doing that. I'll give it a try!