Add support for ISOBMFF Files (AVIF, HEIF, CR3) #1229
Comments
|
Please add support for CR3 and HEIF/HEIC image formats, at least to support reading the image/EXIF informations located in the container. I've read #1066 , #978 and the comment here darktable-org/rawspeed#121 (comment). I tried to understand the concerns about legal issues when reading data from patented ISOBMFF, but I could not see any evidence in this concerns. As far as I've read, there already have been contributions submitted to support these file fomats. Please add them to your library. Thank you in advance! |
|
This would now be in Exiv2 v0.27.3 without this show stopper from @D4N #1066 (comment) It's too late to develop the code required to implement this feature for v0.27.3 which is now at RC2. #1018 (comment) I will support and mentor/assist anybody willing to work on this issue. |
Thank you for your answer. But I still don't understand exactly what the goal is. There have been 2 contributions done to bring HEIF support to exiv2, maybe these could be reopened? |
|
I'm working on a book "Image Metadata and Exiv2 Architecture". Here's today's version: https://clanmills.com/exiv2/book/ (and/or https://clanmills.com/exiv2/book/IMaEA.pdf). The book is coming along quite steadily. The purpose behind the book is to help Exiv2 survive and be developed in future. It was not only Dan's comment, there was another very negative comment: #1066 (comment) I didn't understand the comment because it refers to a case about which I know nothing. I don't think we need to use another library to do this. We already have a CR2 and JP2000 parsers. We can probably modify that and we're done. And there's another brainless approach. Exif metadata is written as an embedded TIFF which has the signature However, as Dan said "Patents don't work that way!", and I took that to mean "no matter what solution you propose, you may be in legal trouble!". A couple of years ago, we needed legal advice and I got free help from an expert in intellectual property licensing. I offered to consult with this person before publish any code for ISOBMFF. There was no response to my offer. Exiv2 v0.27.3 will ship as scheduled on 2020-06-30. I'm going to finish the book and have offered to do "one final release" which will be 0.27.4 or 0.28. Again, no feedback from the team. We could have a team meeting (on Zoom or something) to discuss this. I can't be bothered. |
|
@clanmills Thank you for all your work with exiv2 and please do everyone a big favor and go ahead with the initial plan of adding ISOBMFF support to your last "one final release". A few words about software patents and multimedia formats Software patents are put in place to ensure competition advantages of the companies who implement them. Companies like Canon do not want to see a rival bring a camera in the market using their own format and/or proprietary software to process their raw images. And this is the main obstacle for open sourcing their formats. Nevertheless, what the end user may or may not do with the raw files generated with a legally obtained camera, is not Canon's concern and it can't be, because any files generated with a camera are always the property of the camera's end user and not the manufacturer's. I am not sure to what Dan referred to when said "Patents don't work that way!" I am not a lawyer but following @johnny-bit and @clanmills initiative I discussed this issue with one. The above comments can be considered a short summary of this discussion. |
|
The exiv2 project has no money at all. None. Not a penny. So why would Canon sue somebody with no money? And if the project has no money, Canon would find it difficult to explain in court why reading metadata (written to a public standard) caused them to loose money. As I have pointed out, we could find the metadata by hunting for the markers. After I started to work on v0.27.3, my wife said "Am I becoming an Exiv2 widow again? You're in your office 12 hours every day. When is this going to stop?". So when Dan and Jens started being so negative, I thought: "I don't need to spend 4 weeks developing and testing ISOBMFF.". I'm not going to do anything at the moment. Exiv2 v0.27.3 RC2 was published as scheduled last Sunday on 2020-05-31. I hope there will be no code changes and it will be released as 0.27.3 on 2020-06-30. If no-one else volunteers, I'll reopen #1066 for the next release (0.27.4 or 0.28). However, I hope somebody will undertake this project. I want to pursue other hobbies such as playing the Euphonium and Piano, running, the garden and spending time with my wife.
|
|
Apparently they wouldn't, but I guess we will have to wait and see what's next. I only wish I had the necessary skills to get involved in a more productive manner other than expressing my support. On the bright side, your music room makes a clear statement about pursuing other hobbies. Although euphonium, running and gardening are not my things, the piano is a serious temptation 👍 and reasonably deserves for some daily attention, not to mention the wife of course. |
|
There's another reason that I forgot to mention. I'd like to solve the "Lens Recognition" problem. I'm going to make a proposal to the community at LGM in Rennes in May 2021. #1202 (comment) In order to work on M2Lscript (pronounced MillsScript), I have to escape from Exiv2 to give me the time. I hope the community will find somebody to take on the maintenance of Exiv2. I intend to run a work-shop at LGM based on my book. I'm doing all I can to ensure the long-term success of Exiv2. |
|
Robin Mills <[email protected]> writes:
The exiv2 project has no money at all. None. Not a penny. So why
would Canon sue somebody with no money?
It's not about us, it is about the consumers of the exiv2
library. Because they might not have the luxury of a lawyer being able
to tell them that continuing to use libexiv2 is ok. Or their
lawyer/legal advisor tells them that it is too risky.
Given that a big chunk of the Linux Desktop stack depends on exiv2 makes
me feel a little uneasy about supporting patent encumbered formats, as
this is nothing that can be solved via compile time switches.
|
|
@D4N - There is literally nothing legally speaking in the way of loading any file on your disk into memory, reading bytes from it and interpreting those. be it heif, mp3, jpg2k or whatever. Also if data is available within file in know standard, there's nothing legally barring you from interpreting that data according to that standard (meaning exif, xmp etc...). As I mentioned - you are stomping on legal minefield if you try to decrypt encrypted data. You are violating law if you're using unlicenced proprietary decoder to read encoded data. You are threading near some borders if you do cleanroom-analysis of data within file and write your own decoder. I assume than none of the above matches description of what's needed to read exif data from ISOBMFF container. Legal-expertise wise: I asked my legal advisor for legal advice and it's cost. I'll setup a meeting with him and ask for cost of legal expertise. I can cover at least part of the cost associated. I also would be needing description of way of ow exiv2 reads files, does it use any proprietary software or any "problematic" parts I've mentioned above. |
|
@D4N Nikon's NEF is a proprietary format, Canon's CRW & CR2 are proprietary formats too, so could you clarify what's the difference with Canon's CR3? They all have patents filed to maintain advantage over their competitors. Even DNG has a patent license called "Digital Negative (DNG) Specification Patent License" although they 've stated about 5 years later (in 2009) that "there were no known intellectual property encumbrances or license requirements for DNG". (sourced from wikipedia and Adobe's website) AFAIK if one runs into trouble for reading proprietary raw formats, then this trouble should be here onto all of us very long ago. If I remember correctly dcraw started reading such files by reverse engineering them almost two decades ago. To my knowledge, there has been no issue with any opensource software reading proprietary raw files all these years. On the other hand, if your concern arise from the latest developments with EU legislation and the rumors/concerns that could potentially harm other open source projects like VLC for reading other media formats (audio or video), this is an entirely different story. Regarding VLC and the like, there are legal grounds because those files are (potentially) not legally acquired or/and they still have certain restrictions such as CDs, tapes, vinyl etc. for example you need extra permission to publicly play a music CD even if you bought lawfully. And there is a lot concern about online piracy and all that. Most importantly, when you buy some music on a CD and mp3 file, you own the medium not the intellectual content it is saved in there. But this is never the case with our camera's raw files, any content saved in a raw format with a camera I take a picture with belongs to me including any copyrights I am willing to introduce to my photos. This the main reason why companies like Canon, Nikon etc never bothered coming after you or every other person like Robin who have devoted significant time into making these formats accessible to the open source community. The other reason is explained in my previous post (nothing to gain). They still charge (possibly) Adobe for some readily available API so they can include CR£ support in their latest LR version, which they charge end users for a lot of money, and they do this because they can (Adobe don't mind some premium for faster support for their software, they take for free from the opensource community a lot more, which is under LR's hood anyway). But you can add this to the reasons why Canon and the like are quite hesitant to provide a free API for reading their raw files. |
I am not sure how this directly relates to the matter of this thread. Quite interesting reading though. As for the purposes of reverse engineering and reading a proprietary format for interoperability purposes, the following US Law makes it pretty clear IMO and also explains why StarOffice, OpenOffice, & LibreOffice had no legal troubles reading MS Word formats before Microsoft's Open Specification Promise over a decade ago. Not to mention about dcraw, exiftools, exiv2, darktable, etc., which are our concern in this thread. "Sec. 103(f) of the DMCA (17 U.S.C. § 1201 (f)) says that a person who is in legal possession of a program, is permitted to reverse-engineer and circumvent its protection if this is necessary in order to achieve "interoperability" — a term broadly covering other devices and programs being able to interact with it, make use of it, and to use and transfer data to and from it, in useful ways. A limited exemption exists that allows the knowledge thus gained to be shared and used for interoperability purposes." $ |
|
Topic: Exiv2 and ISOBMFF Support Join Zoom Meeting Meeting ID: 821 3673 0279 |
Hey. Euphonium was my instrument too, but I haven't played since high school. :) |
|
I played Trumpet in the school orchestra and haven't played it since. I decided to move down an octave and delighted by the experience. I got the instrument in September 2018 and expect to pass Grade-6 in November. I attend a "virtual" trumpet school in Australia. The teacher gives us a video/lesson every day on Facebook for $30/month! (20 in the class). I have a brass tutor who comes to my house every other Wednesday. Since lockdown, we meet in a virtual class room. I've played piano all my life. We purchased a digital piano last year. It's amazing. It's beautiful to play. It'll accompany me when I can get a midi file. I purchased ScanScore which does scan (or PDF) to music. So, if I can't find a midi file, ScanScore can "read the dots.". Have a look at my book. It's coming on well: https://clanmills.com/exiv2/book/ In particular, please read this: https://clanmills.com/exiv2/book/#future |
|
I've had a totally new thought about this. I was talking to Jens (the gexiv2 engineer) and he was talking about introspection. I couldn't get the gnome introspection code to build. However, it sure smells like COM (I'm so old, I remember COM/DCOM being new). If libheif (and other libraries) used introspection, I think we can call other shared libraries without linking their code at all. It's 100% fixed at run time. If we do a metadata->read() on a HEIF, we create "Mr Heif Object" and ask him to give us the Exif Tiff embedded in this file (or XMP, or IPTC, or ICC). By using introspection, we don't have a single line of libheif in our code base. No header files. No linking. We don't need a special build option for Heif. Nothing. It's dealt with at run-time. |
|
AVIF is an open standard, CR3 is/should be freely readable. JPEG2000 is already in the lib. only issue: HEIF. Still, reading bytes from file and interpreting known values doesn't touch file patents that I know of. Using introspection and dynamic linking would've in edge cases circumvent any problems related to using totally free lib in patent minefield. Same thing would be if it were filetype libraries connection to exiv2 not other way around and announcing things they can provide. Then it'd be simple wine-brick story, only in "don't install libPatentedFileFormat along with exiv2 or you might've been able to read exif, xmp and ICMP data from PatentedFileFormat". |
|
@johnny-bit You're right. If we use a library which has code that infringes the patent, we could be in trouble, even if we don't use the part of the library that infringes the patent. I'm also mistaken about having no libheif code. We have to know the interface (a Incidentally, @cgilles submitted a PR today to link libheif #1233. It only involves changing about 25 existing files and adding 53 new files. The patch has 130,000 lines and cannot be applied to 'master'. The PR is only as simple as this because it is readonly, and doesn't include any test code at all. Clearly it's ready to ship (not). We could return to my idea to write as little code as it takes to read the file and locate the 4 blocks of interest. I was originally looking at Dave Coffman's parse.c which reads all manner of files in 1000 lines of C. |
Holy cow! This book looks like it will be amazingly comprehensive. Great work so far!! |
|
I don't know with github PR make a garbage. this is a "git diff HEAD > heif.patch.txt". 22 Kb file with limited in-deep code change. It very simple in fact... |
|
Patch V2. I forget a block change in a CMakeLists.txt script: For an HEIF unit test, it's very simple. Take an HEIF image taken with an iphone for ex, which contain metadata an try to use exiv2 CLI tool to display Exif|Iptc|Xmp. libheif parse file to extract metadata chunk and pass byte-array as well to exiv2 core. No more. The rest is standard Exiv2 code unchanged... This is a sample HEIF file with metadata: |
|
Here's the summary of the Sunday's meeting on Zoom. #318 (comment) |
|
Can you please outline the steps necessary, in your opinions, to get support merged into exiv2? You say you're worried about the consumers of the exiv2 library, and the linux desktop at large, but we are the consumers of the exiv2 library-- darktable, digiKam, elementary photos, GNOME Photos, etc etc. |
All I said was give people (i.e. individuals and distributions) who care about such things as non-free and potentially patent-proned formats an opt out. Nothing more. Because doing that later is awful and much easier if you design the whole feature with it from the start. |
|
As the upstream author of GNOME Photos, a GNOME contributor in general, and a Fedora contributor, I totally agree with @phako here. One approach that's used by some frameworks (eg., GStreamer) is to use loadable (via |
|
Alright, thanks! I really hope the ISOBMFF support is now being green-lighted. |
|
Robin and I have discussed what productive steps might be taken next, and think that in order to protect Exiv2 and its users, we should consider joining the Open Invention Network. See https://openinventionnetwork.com/ OIN is a consortium of major technology players including Redhat, Google, and IBM, with over 3300 members all up. You can read about the benefits of joining OIN at https://openinventionnetwork.com/about-us/member-benefits/ They have a license (https://openinventionnetwork.com/joining-oin/join-now/license-agreement/ ) which Exiv2 could sign which would commit us to:
There's no fee for joing OIN. There are potential benefits in the event of any legal issue arising with Exiv2, possibly even with existing code, mostly due to the membership of some very large industry leaders. Furthermore, they have a concept of a Linux System Definition, which includes core Linux and adjacent open source technologies. See https://openinventionnetwork.com/linux-system/ Note that currently included in the Linux System Definition are imaging-related packages such as gimp, gd, ImageMagick, and digikam, so I expect Exiv2 would be a prime candidate. |
|
Thank you @dhoulder for working with me on this issue. On 2021-01-18 (9 weeks on Monday), I will retire. My book will be complete. I hope somebody will step forward to maintain Exiv2 and I assure them of my support and cooperation. I hope my successor will get Exiv2 into OIN and the Linux System Definition. Concerning ISOBMFF, here's my summary of the situation.
If somebody offers a PR for ISOBMFF support, I will accept it provided:
I've personally found working on Exiv2 to be interesting and mostly enjoyable. It has been a surprise that more contributors haven't joined in the fun. On page 2 of the book, I acknowledge: Exiv2 contributors (in alphabetical order): Abhinav, Alan, Andreas (both of them), Arnold, Ben, Dan, Gilles, Kevin, Leo, Leonardo, Luis, Mahesh, Michał, Mikayel, Miloš, Nehal, Neils, Phil, Rosen, Sridhar, Thomas, Tuan .... and others who have contributed to Exiv2. I hope the C19 situation fades in spring and LGM goes ahead in Rennes in 2021. I look forward to seeing you there. |
|
Thanks, Robin! So I will reload and restart my efforts to get HEIF support into the Exiv2 and then darktable with full metadata I/O. Cheers! Happy WE. |
|
The tvisitor.cpp code in my book (svn://dev.exiv2.org/svn/team/book) reads the metadata in JP2, AVIF, HEIF and CR3 (and many more formats). I have documented and explained in the book how it works. The tvisitor.cpp code is less than 3000 lines and decodes Exif (including MakerNotes), IPTC, XMP and ICC metadata in every format supported by Exiv2, plus ISOBMFF, BigTiff. It's a single file of code. If you wish to decode PNG you need to link libz. My aim is to demystify how metadata is encoded in all the common image formats. |
|
@1div0 Thanks for your email. My reply bounced on your mail server. I've fixed the "double free" in class CRW on Linux. I've also fixed a Linux compiler warning in dmpf.cpp I've look at your "NEOWISE" image and see: I don't know why you referred me to that image, however I'm sure it'll all make sense when we discuss it some more. I'm looking forward to working with you - together we'll move some little mountains! |
|
hi everyone, happy new year to all of you and I hope that all of you are safe and healthy. As the new Canon format CR3 is still an open issue for tools such as darktable, rawtherapee and many more, there was the idea to form a little task force from different projects in order to get this done. Within this the meeting we should discuss what are the blockers, e.g. legal questions / joining the open-invention network, what can be tackled by whom etc... I hope that the time slot on the weekend fits most people. Topic: Support for ISOBMFF Files (AVIF, HEIF, CR3) Join Zoom Meeting Meeting ID: 821 6888 5673 Cheers, |
|
We have a horrible communications setup here. There are different discussions about this taking place here, discuss.pixls.us, Wire, private email. Can we discuss this here: https://app.element.io/#/room/#exiv2-chat:matrix.org. @LeoHsiao1 Could you participate in this please. I'm aiming to ship Exiv2 v0.27.4 with ISOBMFF support on 2021-04-30 and that will include your python test code (and the death of the bash scripts). |
|
@clanmills |
|
Thanks, @LeoHsiao1 I think it's unlikely that you will be expected to do anything. We will have to update the test suite of course, however that's unlikely to require your input. I'll let you know if we need your help. The good news here is that Exiv2 v0.27.4 is likely to ship at the end of April 2021 and will include your test code. What happens to Exiv2 after v0.27.4 is not my concern and hope that the joint project with darktable will lead to a bright future for Exiv2. |
|
Agenda for the meeting: |
|
Exiv2 0.27.4 Progress is being documented here: #1018 (comment) |
|
Fixed. #1475. Seven Engineers have contributed to this. @alexvanderberkel @LeoHsiao1 @hassec @clanmills @kmilos @piponazo @1div0. And I must mention David Houlder for his help with the legal investigation. And Gordon Laing for test images. Pat and his folks at discuss.pixls.us who will set up a test image server. By far the largest team of contributors in the history of the Exiv2 project. |
|
Thank you @clanmills and all contributors that made it possible! |
|
Big Thanks to @clanmills and everyone else involved in adding support for CR3 !! |
Is your feature request related to a problem? Please describe.
Exiv2 does not support ISOBMFF format, therefore doesn't support AVIF, HEIF, CR3 and possibly other filetypes using this format
Describe the solution you'd like
Ideally - ISOBMFF format support, at least reading data from it :)
Describe alternatives you've considered
ExifTool and couple other open source projects already support ISOBMFF based files, however exiv2 still doesn't.
Additional context
Iam fully aware of #1066 - I asked my legal counsel and gethered much needed advice. According to all info I managed to gather, there's no legal issue in creating and using a tool capable of reading data and metadata stored in ISOBMFF containers. Worth noting is the fact that JPEG2000 IS an ISOBMFF based and support for it is present in the library without any issue.
Aditionally, already existing libraries like libavif, libheif, libraw and others already work with ISOBMFF files without legal problems.
The ONLY problematic issue would be not reading but decrypting encrypted (not encoded) data or using proprietary code, which I am sure is not the issue here.
The text was updated successfully, but these errors were encountered: