Tamron SP 70-200mm f/2.8 Di VC USD A009 not correctly identified with lens #255

[ariznaf]

This is not really a but. But I could not find a better place, and was said to report it to exiv2 developers.
I am using darktable, which happens to use exiv2 to read exif data.
Tamron SP 70-200 f2.8 Di VC USD (G1) lens is not recognized with canon cameras, at least not with my canon 80D and under Windows 10 64bits.

Lens is identified as "255" lens.
But the lens is listed in lensfun database.

After some research I have been able to config exiv2 to get it working configuring exiv2.ini
under C:\Users\USERNAME\exiv2.ini
This is the entry I created to get it working

[canon]
255=Tamron SP 70-200mm f/2.8 Di VC USD A009

This is the thread where we discussed about the solution
https://discuss.pixls.us/t/my-tamron-lens-is-not-automatically-detected/17939

I suppose that the 255 code is not linked to this lens under canon or something like that.

I report it in case it can be added to the exiv2 lens database for other users in future versions.
If it can be included in next update it would be great, if not, may be it helps other windows users of darktable with this lens.

[clanmills]

This isn't a bug and you've discovered how to deal with it. It's a pity the darktable sent you here instead of explaining how to deal with it.

About 50% of the issues reported to Exiv2 concern lens recognition. The reason is because the lens isn't stored in the metadata by the manufacturer, so we have to guess the lens by inspecting other metadata. Even knowing the lensID (in your case 255 = -1 (uint8_t)) is of limited help, because Canon might have dozens of lenses with that lensID. We know of at least two:

1920 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance $ grep 255 src/canonmn_int.cpp 
        {  255, "Sigma 24-105mm f/4 DG OS HSM | A"                          },
        {  255, "Sigma 180mm f/2.8 EX DG OS HSM APO Macro"                  }, // 1
1921 rmills@rmillsmbp:~/gnu/github/exiv2/0.27-maintenance $

So for Exiv2 v0.26, I added the "configuration file" feature to enable you to "fix" this for your-self.

There is an open-source conference for the Linux Graphics Community called LGM (Libre Graphics Meeting). The next one is in Rennes, France in May 2021. I believe we're going to have a metadata work-shop one afternoon for everybody with an interest to discuss how we can "solve" this permanently.

[ariznaf]

Yes it is a pity I have been directed here instead of explaining me what the problem was.

I am not a conscient user of exiv2, just happen to be used by darktable internally (i am using windows).
But this problem gave me the ocassion to know about exiv2 and its interesting functionality.

Let me ask one more thing please, before closing it for ever.

If I understand you well 255 just means the lens has not been correctly recognized by exiv2 as it is not correctly identified by canon exif data (due to lens being a tamron i guess).
So it will be assigned to whatever lens exiv2 cannot identify.
For example my tokina lens for canon.

Your say I can add a configuration file, are you meaning using 255= Tamron SP 70-200...?
May you point me to instructions about correctly identifying the lens?

If I use the 255 then any other lens not identified will be uncorrectly mapped to tamron lens (like may tokina).

[clanmills]

Canon add data using the "MakerNotes" feature of the Exif specification. This enables the manufacturer to store up to 65000 bytes of binary data. Exiv2 knows how to decode/encode Canon's MakerNote (and other camera manufacturers). Down in the MakerNote, Canon store a lensID. For your lens it is 255. That's probably Canon's way of saying "I haven't a clue about this lens".

However the number 255 isn't special. There are other lensIDs (eg 137) is used by 17 Sigma and Tamron lenses. In order to know which of the 17 is being used, we have to inspect other metadata such as min/max Aperture, min/max Focal Length and other data.

It's possible that you could have 2 lenses that both have lensID = 255. Nobody has ever reported this.

In the man page for exiv2, I called the file ~/.exiv2 (or %USERPROFILE%\exiv2.ini) the configuration file.

I intend to make a proposal next year at LGM concerning this matter. Both exiv2 and ExifTool can format the metadata in JSON format. I'm going to propose to implement a program to read the JSON and return the Lens. That program will have an embedded programming language with the rules to identify the lens. The scripts will be ascii files which can be updated. It will be called M2Lscript (MetaData to Lens Script), pronounced "MillsScript". The MillsScript interpreter will be available as a command-line program, a perl module (for ExifTool) and a C++ library (for linking into exiv2).

In this way, new lens definitions can be added to "MillsScript" without touching anything in Exiv2.

If you're interested in participating in this project, I'll be happy to recruit you as a contributor.

[ariznaf]

Thank you a lot.

Well I suppose canon uses 255 (unknown) in their makernotes for any non canon lens.
Canon does not apply lens corrections or any other correction to the jpegs or if you use their software to develope images, for any lens, just for their lenses.
I had expected that canon simply don't care about the lens with any lens that is not manufacture by them.
But if you say there are lenses from sigma that are assigned an ID, it seems that is just a problem with some lenses.

Does exiv2 use just makernotes and LensType?

There are other exif data in my images that describe this lens correctly.
For example, exiv2 provides this other exif fields that identify the lens:
0x0016 CanonCs LensType Short 1 255
0x0095 Canon LensModel Ascii 74 TAMRON SP 70-200mm F/2.8 Di VC USD A009
0xa434 Photo LensModel Ascii 40 TAMRON SP 70-200mm F/2.8 Di VC USD A009

I had expected to be able to report that info somewhere to help fix the problem with this lens.

I suppose that is not so easy, may be each manufactures uses different fields and makernotes fields..

Cannot I use one of those fields to map them to the correct lens instead just 255?

Well for now it is working for me (meanwhile I do not have other lens that is not correctly identified and return 255 as ID too).

I have basic programming knowledge, and am using windows (but also can run debian in windows for tests, if it is required).
If I can help providing a MillsScript script for the lenses and cameras I own I will be happy to help in that, if you point me to the correct place where I could learn about doing it.
Is there a beta exiv2 using millsScriptI can download to make tests?

[clanmills]

MillsScript doesn't exist. It's going to be proposal to LGM in 2021. This is the first time I've mentioned it in public.

The metadata names in Exiv2 are of the form: Family.Group.Tag Family is simple: it's either Exif, or IPTC, or Xmp. Group depends where the tag was found. If its Canon, CanonCs - it came from the Canon MakerNote. The MakerNote is complex, so it has different sub-sections: hence Canon, CanonCs and others.

I don't believe I can answer any of your other questions. The Community is aware that this matter needs attention and I intend to offer a proposal to solve this. I'm not only offering the design, I will implement this if the proposal is accepted.

Up until now, the community have been saying "It's all the fault of Exiv2". This is quite incorrect. The source of the problem is that the Exif specification does not define how to store the lens in the metadata. I believe that has been addressed in the latest Exif 2.32 spec (2018). In the meanwhile, there are millions of camera and billions of photographs which do not have the lens in the metadata.

[ariznaf]

Sorry you are right that may be exif data from my viewer.
This is what exiv2 provide as lens info (fields containing lens in their name):
Exif.CanonCs.LensType Short 1 255
Exif.CanonCs.Lens Short 3 70.0 - 200.0 mm
Exif.Canon.LensModel Ascii 74 TAMRON SP 70-200mm F/2.8 Di VC USD A009
Exif.Photo.LensSpecification Rational 4 70/1 200/1 0/1 0/1
Exif.Photo.LensModel Ascii 40 TAMRON SP 70-200mm F/2.8 Di VC USD A009
Exif.Photo.LensSerialNumber Ascii 11 0000000000

OK thank you for your explanations.
As a user I had expected it was only a matter of a lens that had not yet been reported to the developers with this camera o manufactures.

I see now the problem is more complex and it is knows, so for now I will keep using the 255 to name lens map (it is a useful patch).
If I can provide useful info about my lenses for the future or try to provide a millsscript for them

[clanmills]

Sure @ariznaf. You can be my alpha tester for M2Lscript.

You are not alone in thinking "oh, we only have to ask the Exiv2 people to add one line of code". It's much more than that. Lens recognition was added to Exiv2 before I joined the project in 2008. It's been a curse. If Exiv2 hadn't attempted to solve that, the community would have better understood the issue.

I added the "Configuration File" (~/.exiv2) code in Exiv2 v0.26 and that has eased the situation for most users.

I have been thinking about M2Lscript for a few years as I have written several interpreters over the years. In fact, I would like to retire from Exiv2 and work on that instead. I retired from Exiv2 in October 2019 with the intention to have a break and make the proposal at LGM this month. Then along came covid-19. LGM has been postponed by a year. I've come out of retirement to do Exiv2 v0.27.3. #1018 (comment)

I'm 69, retired from Adobe in 2014. I'd like to escape from open-source. Probably never will!

Maybe M2Lscript will appear next year. We'll see.

[ariznaf]

Well I begun reporting to darktable developers (you have to understand end users don't know exactly how a program works).

Yes, really I have thought it was just a matter of giving the correct codes or exif fields for the lens in this camera or camera manufacturer, it is at it should be if exif worked as expected and defines a way of identify a lens with a unique id.

But it seems exif itself hast its flaws and is not as easy as that.

I want to express my grattitude to you and other developers.
It may seem that users are always complaining about software, because we complain when there are errors or report issues.

Opensource software is difficult to maintain and it is difficult to find the balance between user support and having time to develope, I know.

That is why I want to explicitly express my grattitude for the time you have take to answer.

[clanmills]

Here's the bill: Please visit OpenHub and give kudos and other kindness such as a review or 5 star rating. https://www.openhub.net/p?ref=homepage&query=Exiv2