Detect near identical images

[raphael0202]

Problem

It seems we have an increasing number of duplicated images in Open Food Facts database: images with different binary content, but that are however almost identical.

Example:

• https://world.openfoodfacts.org/images/products/317/328/910/5708/2.jpg
• https://world.openfoodfacts.org/images/products/317/328/910/5708/4.jpg

We want to detect these almost identical images to remove them. It enables us to save disk space and make the work of contributors easier.

Proposed solution

Use fingerprinting techniques to assign a single hash to each image. See this blog post for more information about image fingerprinting. Explore the recall/precision trade-off for each hashing techniques.

The documentation of the undouble library is available here.

You can download all images of a selected subset of products using Open Food Facts Images dataset, and detect quasi-similar images. A manual analysis of results should be performed to assess which technique is the most robust for our use case and the precision/recall/accuracy metrics.

Related issues

openfoodfacts/openfoodfacts-server#8445