feat: add script to export products data and images for docker dev #6010
Conversation
stephanegigandet
added
the
Data export
|
Added an option to get a random sample of products, based on their creation date: ./export_products_data_and_images.pl --sample-mod 10000,0 --products-file /srv/off/html/exports/products.random-modulo-10000.tar.gz --images-file /srv/off/html/exports/products.random-modulo-10000.images.tar This is so that we can make a random sample of products instead of using the top scanned products (which typically have hundreds of images, so it's not very suitable for a dev environment). |
| # e.g. if the tar command failed and the script was stopped | ||
| wget -O products.tar.gz https://static.openfoodfacts.org/exports/products.random-modulo-10000.tar.gz 2>&1 | ||
| tar -xzvf products.tar.gz -C /mnt/podata/products | ||
| rm products.tar.gz |
There was a problem hiding this comment.
FYI we have /mnt/podata/mnt which, in docker, is a tmpfs mount. It may be a more appropriate location.
(it also applies for following lines).
There was a problem hiding this comment.
ah I was wondering why we had this /mnt/podata/mnt empty directory, thanks for the explanation
Co-authored-by: Alex Garel <[email protected]>
Co-authored-by: Alex Garel <[email protected]>
Co-authored-by: Alex Garel <[email protected]>
|
Kudos, SonarCloud Quality Gate passed!
|
New script to export product data and images, for instance to load into a development install of Product Opener.
Related to #6009
Can be run like this:
~/openfoodfacts-server$ docker-compose run --rm backend /opt/product-opener/scripts/export_products_data_and_images.pl --query ingredients_tags=en:salt --products-file /opt/product-opener/html/exports/test-products.tar.gz --images-file /opt/product-opener/html/exports/test-products-images.tar