Canon R6 CR3 support

[ellenhp]

My new Canon R6's CR3 raw files seem to be parsed just fine in the RawTherapee file browser, but when I attempt to edit them I see an entirely white image. I noticed here that the program seems to just have a place where it stores information about how to decode raw files, and I'm assuming that's just missing.

I can shoot the necessary frames, but is there documentation on what to like, do with them? I know folks are probably busy and if there are docs on how to process the white frames and create the actual data on white levels for this sensor, I can probably follow them and submit a PR with the changes. Or at least publish a camconst.json so other folks with R6's can process their files without converting them to DNG first. If docs on this don't exist, maybe they should? If nothing more it might lighten the load on the developers to have drive-by contributors be able to process their own white frames. :)

[ellenhp]

I think I answered my own question by going through the camconst.json comments. I'll probably shoot those frames today and look into analyzing them. It's more complicated than I thought it would be but I'll do my best :)

[Thanatomanic]

If docs on this don't exist, maybe they should? If nothing more it might lighten the load on the developers to have drive-by contributors be able to process their own white frames. :)

The only other thing, apart from the comments in camconst.json is this wiki page: http://rawpedia.rawtherapee.com/Adding_Support_for_New_Raw_Formats
The process to determine what to put in camconst.json is not entirely trivial and there is room for some margin of error in some cases. Personally, I analyse the files completely outside of RawTherapee to determine appropriate black and white levels.

If you're interested in doing the analysis on your own and come up with a json entry, I would be happy to verify it. If so, please provide a downloadable folder with the necessary shots.

[Thanatomanic]

It seems someone else has already made the necessary shots as well. https://discuss.pixls.us/t/canon-eos-r6-photo-sets/20094
I'll try to analyse them and see what I can come up with.

[ellenhp]

Cool! I appreciate it. I looked into it a bit more and it seems like I'd have to pay for RawDigger and run it in wine, which kind of tamped down my enthusiasm for figuring this out.

[Thanatomanic]

After analysis, I get the following - quite unsurprising - results:

Regular distribution

Cumulative distribution

Some explanation
Without LENR, almost the full 14-bit range is used and the white level could be set at 16382. The LENR mode does some local magic, giving a tail in the pixel value distribution. The problem with this, is that we don't / cannot currently differentiate the white levels based on whether LENR is on or off. The usual approach therefore is to set the white level relatively low (here e.g. 15900 or 16000). My understanding is that it is worse to _over_estimate the true white level than to _under_estimate it. Activating 'highlight reconstruction' can easily recover initially unavailable highlight information, but values that are actually clipped, should remain so.
@Beep6581 could probably tell more about how camconst.json came to be and how the black and white levels have been set in the past.

[Thanatomanic]

@ellenhp The PR #5905 includes a setting for the R6. Tomorrow's nightly build will include it in RawTherapee.

[Beep6581]

Hi @Thanatomanic

What program did you use to gather the data, and to visualize it? It looks great.

This issue indicates that reading image data from Canon EOS R6 is supported, but I tested 5.8-2775-g7a2463a and the raw file neither shows up in the File Browser nor is opened when I pass it as an argument:
https://raw.pixls.us/getfile.php/4135/nice/Canon%20-%20Canon%20EOS%20R6%20-%203:2.CR3

[Thanatomanic]

I'm using Wolfram Mathematica, commercial software that I got to know during my PhD and absolutely love for data processing. It should be very similarly possible to make these plots in Python.

As for not showing up in the File Browser, did you enable the CR3 extension in the Preferences?

[Beep6581]

@Thanatomanic yes, tried passing it as an argument too. Does that file open in your RT?

ParseExtensions=3fr;arw;arq;cr2;crf;crw;dcr;dng;fff;iiq;jpg;jpeg;kdc;mef;mos;mrw;nef;nrw;orf;pef;png;raf;raw;rw2;rwl;rwz;sr2;srf;srw;tif;tiff;x3f;cr3;ori;

[Thanatomanic]

@Beep6581 The file you link to is a compressed RAW format according to ExifTool. Apparently, that is not supported yet in the CR3 decoder. ART shows the file in the browser, but outputs garbage too.

Edit: In verbose mode RT outputs the following when loading the file browser:

O:\Fotos dev\RAWTest\Canon - Canon EOS R6 - 3_2.CR3: Corrupt data near 0x60eb68

[kdm9]

@Beep6581 & @Thanatomanic It would appear that exiv2 now support the ISO BMFF, the base format for CR3s after PR Exiv2/exiv2#1229. AFAICT the main impediment to supporting R5/R6 metadata in RT (including funky whitebalance issues, #6089) is getting the rtexif -> exiv2 PR (#5889) merged. Is there anything external devs like myself (with some CV, exiv2, and C++ experience) can do to assist with this? lack of metadata support has been a significant issue for me (w/ an R5), and I'm desperate to see a fix included in the next version of RT.

[Lawrence37]

@kdm9 There are some recent comments in #5889 regarding the current status, if you want to take a look.

[Beep6581]

@kdm9 branch metadata-exiv2 needs to be tested and issues ironed out.

[freebrowser1]

Still not supported. Loading an EOS R6 (FW 1.40) .cr3 file results in a plain white image. Version is 5.8. It happens on both macOS Big Sur 10.16 and Linux Ubuntu 20.04 (both Intel 64).

[ellenhp]

There's a fork of RawTherapee called ART that does support CR3 files I think. It seemed to work fine last time I tried it, but I can't vouch for things like color accuracy. Only that it wasn't a plain white image :)

[Thanatomanic]

@freebrowser1 The R6 files can be opened perfectly fine in the nightly builds. I would advise to download the dev AppImage and use it from here https://github.com/Beep6581/RawTherapee/releases/tag/nightly or use an unofficial build http://rawpedia.rawtherapee.com/Download

[freebrowser1]

Thanks.
That worked. I can edit the .cr3, but I still see a message 'cannot read EXIF data'.