Add support for ISOBMFF Files (AVIF, HEIF, CR3)

[cryptomilk]

Describe the bug

Hello!

I've started to add support for the AVIF file format [2] in darktable using libavif [1]. I've worked with the maintainer so that it works nicely with cmake.

However I'm not able to load the image file in darktable, because exiv2 is not able to read the exif data and then fails. You can find my darktable code here [3].

If you need help or want to do some pair programming, let me know. It isn't hard to just read the exif data blob.

[1] https://github.com/AOMediaCodec/av1-avif
[2] https://github.com/AOMediaCodec/libavif/
[3] https://github.com/cryptomilk/darktable/tree/master-avif

[cryptomilk]

I've finished the implementation and can export and import AVIF images now!

[cryptomilk]

AVIF support landed in master and will be available with 3.0.1.

[boardhead]

Perhaps I'm missing something (likely), but the latest update I see to master was 20 days ago.

[cryptomilk]

I'm talking about darktable master -> https://github.com/darktable-org/darktable

[boardhead]

Ah

[novomesk]

I have a question for exiv2 developers:
Would you allow another library as dependency to support this feature?
Or the code must be completely independent, without additional libraries?
If external dependency is possible, then read-only support should be trivial. After calling avifDecoderParse() in libavif, size and metadata are available already (pixel data were not decompressed yet).

[D4N]

I have a question for exiv2 developers:
Would you allow another library as dependency to support this feature?

I have no objections to additional dependencies, provided that they are sane to build, relatively stable and don't drag us into a licensing nightmare like HEIF does.

Or the code must be completely independent, without additional libraries?

No, we have already some dependencies for reading well known formats.

If external dependency is possible, then read-only support should be trivial. After calling avifDecoderParse() in libavif, size and metadata are available already (pixel data were not decompressed yet).

Sounds good. So how is the licensing & patent situation around AVIF?

[cryptomilk]

AVIF is a royalty-free codec. The patents are free to use. I've added libavif to openSUSE and currently working to get it into Fedora.

[boardhead]

Note that AVIF is nearly identical to HEIF as far as structure and metadata is concerned.

[novomesk]

Here is an example how to print EXIF and XMP metadata from .AVIF file using libavif:
avif_metadata.cpp.txt

How to compile:
g++ avif_metadata.cpp -lavif -lexiv2 -o avif_metadata

How to use (example):
./avif_metadata Tomsk_with_thumbnails.avif

Example of the output:

[Tomsk_with_thumbnails.avif],   1280x720        8 bit   images: 1
EXIF data length: 58 bytes
Exif.Image.Copyright                         0x8298 Ascii      31  https://www.elecard.com/videos
No XMP metadata in Tomsk_with_thumbnails.avif

Test file(s) download:
https://github.com/AOMediaCodec/av1-avif/tree/master/testFiles/Microsoft

My plug-in for GIMP 2.99 is able to save XMP metadata too. Windows installer here: https://www.reddit.com/r/GIMP/comments/f6ro3q/gimp_2991_for_windows_64bit_with_avif_support/

[clanmills]

I'll take ownership of this for Exiv2 v0.27.3

@D4N has expressed concern about patent infringement if we use a library to read heif files. We can totally avoid patent infringement by writing our own code to read/write the metadata. This has be done for many file formats and typically requires about 500 lines of code.

For Exiv2 v0.28 and later, the team may prefer to use a library.

[cryptomilk]

I think it is a good idea to not require a library for that!

[D4N]

We can totally avoid patent infringement by writing our own code to read/write the metadata

That's unfortunately not how patents work.

[clanmills]

You cannot have a patent on an ISO standard. We are reading an ISO standard file. When I have the code ready, I will get legal advice from the person who advised us about the commercial license.

[phako]

You cannot have a patent on an ISO standard. We are reading an ISO standard file.

I think ISO/IEC 13818 (MPEG-2) and ISO/IEC 14496-10 (H.264/MPEG4-10) tell a different story.

[clanmills]

Is it impossible to support HEIF? Everybody is being so negative. Is there a solution, or should I give up?

[novomesk]

I believe you can parse the ISOBMFF container and find location of the metadata but you should not decompress the h265-hevc data.
If someone is super scared of renamed QuickTime's container from 2001, it is known fact that Apple joined Alliance of Open Media, and the AOM members granted usage of their patents (in AOM related software).
Many people are able to find metadata using sequential search (I usually found them at the end of AVIF files I worked with)

[clanmills]

Thank You @novomesk. You are confirming my assertion. It is legal to parse an ISOBMFF container which is a linked list of self contained boxes.

The boxes which contain metadata are written in their native format. XMP is written in XML, Exif in Tiff, IPTC and ICC are binary blocks.

Exiv2 has no code to deal with pixels. Preview images in metadata are normally stored as embedded JPEGs (which do not have embedded metadata). Exiv2 simply does a binary copy of those previews without any further parsing.

I don't recall hearing of AOM. I believe ISO will never ratify a standard which includes technology which must be licensed from the owner (even if the owner requires no payment). I believe you are saying that AOM put the QuickTime file format into the public domain and ISOBMFF is the ISO Standardization of that format.

It's entirely possible for an ISOBMFF format to include boxes which have data encoded in a proprietary format such a h265-hevc. Decoding such a box may be a patent infringement, and Exiv2 will have no such code. The boxes of interest to Exiv2 are encoded using public standards.

My plan is to implement my own ISOBMFF parser to read/write metadata boxes. All other boxes will be skipped on reading, and binary copied to output when writing.

When I have the code written and tested, I will get a professional legal opinion about what I intend to publish.

I would very much appreciate help with testing from Team Exiv2 and the wider open-source community.

[phako]

I'm not saying to not support it. I'm saying make it completely disablable so that those who care about not stepping into mucky waters or care about not supporting any non-free formats can safely remove the support.

[clanmills]

This project is cancelled.

[clanmills]

This project is cancelled until somebody sorts out the legal challenges. I have expressed my point of view about this and been met with hostility. This hostility has not been backed by evidence or reasoned argument. I have offered to get professional advice before publishing. Exiv2 v0.27.3 is on track for RC1 and will not have support for ISOBMFF.

[AurelC2G]

I'm really confused by this: HEIF itself is just a container, you are fine to read the ISOBMFF boxes (such as the ones containing exif metadata) as long as you don't decode the encoded image bytes.

AVIF is just a HEIF file containing images encoded using the AV1 codec (as opposed to HEIC, which are HEIF files containing images encoded with HEVC). Decoding metadata from either would be fine, the problems only start once you decode the HEVC data itself (av1, on the other hand, is open and there's no problems at all for AVIF: https://aomedia.org/the-alliance-for-open-media-statement/)

[1div0]

Depeche Mode: Wrong!

[cryptomilk]

Please stop spreading FUD!

[johnny-bit]

In what jurisdiction is the legal advice needed? I can personally pay for legal advice regarding ISOBMFF files it that would mean possibility to read data from them. I have very little patience for FUD spreading, patent trolling etc.

[clanmills]

I also offered to get legal advice. I will never work on ISOBMFF.

I'll be 70 in January. I'm working on a book Image Metadata and Exiv2 Architecture. When that is finished, I'm done with Exiv2. I am willing be the release engineer for one more release. Either Exiv2 v0.28 or Exiv2 v0.27.4.

[clanmills]

Here's the summary of the Sunday's meeting on Zoom. #318 (comment)

[ruxandra-valcu]

Tried a bit of Karen-ness in Canon support, let's see if it works.

Hello!
I'm considering upgrading to an EOS R6 (from my current 800D) but I won't be able to use this, or any other camera with CR3 format, until my favourite photo editor, Darktable, supports it on Linux. Currently, progress on this support is stalled by people being worried by legal issues (see darktable-org/rawspeed#121 , and #1066 for the legal issues in the decoder library), I understand you may not have the resources to create an open SDK for the CR3 format, but could you at least confirm to these guys that it's fine to parse the CR3 format as they're parsing CR2 or anything else? As things stay now, people aren't buying Canon mirrorless because of lack of full Linux support.

P.S: I'm aware of the Adobe DNG converter but that introduces unnacceptable brittleness in my workflow and is therefore not a workable solution.

[StefanOltmann]

@ruxandra-valcu Did you ever get a response from Canon?

[ruxandra-valcu]

@ruxandra-valcu Did you ever get a response from Canon?

nope