Export Unusably Slow

[MenooMeow]

Describe the bug

I am exporting 317 images that are 5204x3472 at AVIF. I'm running Linux Mint Cinammon 21.3, 32GB of RAM, 13th Gen Intel Core i9-13900H x 14. Overall CPU use appears to be less than 20% at all times. Darktables process averages only 5.10% CPU consistently.

I don't know how else to say this since I like Darktable. I can't use Darktable for any kind of work because 10 hrs so far and it's only on 32 of 317. At this rate, it will take 4.1 days to complete if running 24 hrs a day. Since I do graphic art work, there's no way this can ever work for me. I'm used to processing this number of images in 5 minutes.

Steps to reproduce

1. Edit a good number of RAW images.
2. Export all images to AVIF.

Expected behavior

Process all images in 15 minutes.

Logfile | Screenshot | Screencast

No response

Commit

No response

Where did you obtain darktable from?

downloaded from www.darktable.org

darktable version

4.8.1

What OS are you using?

Linux

What is the version of your OS?

Mint 21.3

Describe your system?

Linux Mint Cinammon 21.3, 32GB of RAM, 13th Gen Intel Core i9-13900H x 14

Are you using OpenCL GPU in darktable?

I dont know

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

JPEG export is many orders of magnitude faster.

[ralfbrown]

darktable uses various libraries to do the image reading and writing; most of them are single-threaded, and particularly at high compression effort can take longer than the actual export processing (the default PNG compression level for export was lowered quite a bit for this very reason a few years ago). AVIF uses a very compute-intensive compression algorithm, and the libavif library seems to be limited to 8 threads (I have a 32-core processor), with some single-threaded processing at the end.

I would suggest using something other than AVIF if at all possible for such large output images. Exporting to JPEG XL runs 16 times as fast (5 seconds versus 80 for a quick test I just did with four 24 MP images) and produces smaller files.... (700k-900k versus 1000-1300k).

[Munsen]

I have same issue.
Convert a jpeg image(24MP) to avif using avifenc took 2 seconds.
Export same file using darktables takes few minutes.

CPU Ryzen 3700X (8 cores)
OS Arch
libavif 1.1.1
darktable 4.8.1

[viba1]

Hi here,
I encounter the same issue on my Ryzen CPU, when I try to export files to .png format.
It seems some external libs doesn't handle multithreading, which is not darktable's problem but a consequence.

But I believe darktable could also benefit from parallelizing batch exports.
Currently, batch exports are processed sequentially (exportation 1/n…), meaning each file is processed one by one, even if the machine has multiple cores available.
This can be a bottleneck for exporting large quantities of files. It would be possible to parallelize batch exports based on the number of cores on the machine, using separate threads or processes for each file to be exported.
This would allow processing multiple files in parallel and improve darktable's overall performance.
However, dependencies between files and system resources would need to be taken into account to avoid conflicts and synchronization issues.
I think this could be a good opportunity to improve darktable's performance and process batch exports more efficiently.
What do you think?

[ralfbrown]

That's been discusssed before. Parallelizing the exports would be technically very difficult and would increase darktable's already high memory requirements even further. The actual image processing during export is already maximally parallel, using every core/thread if you don't have a GPU (and all of your GPU's compute units with OpenCL).

The only thing which could be done that would actually speed up the overall export times is to do the image saving separately from the image processing. Complicated to avoid race conditions and other errors resulting from parallel processing, and increases memory consumption as the uncompressed previous images (at 16 bytes per pixel) need to be kept in memory while processing the next. And gives next to no improvement for the most common output formats (JPEG and uncompressed TIFF)....

[harskish]

Where does the parameter AVIF_SPEED_DEFAULT come from on this line?
Building and debugging on my MacBook (M1 Pro), the default seems to be -1, which results in very slow exports. Overriding the value in the code with something more reasonable results in much faster exports.

For a 24MP ARW exported as a 4000 × 2661 HLG Rec2020 avif at quality 70:
-1: 24.58 seconds (this is the default)
0: 24.53s
2: 12.24s
5: 3.10s
6: 0.64s (avifenc default)
10: 0.17s (maximum speed)

I feel like the encoding speed parameter should be exposed in the UI, perhaps as the reciprocal and renamed to effort.

[harskish]

Seems related to this: AOMediaCodec/libavif#481

[kmilos]

Where does the parameter AVIF_SPEED_DEFAULT come from on this line?

It comes from libavif itself: https://github.com/AOMediaCodec/libavif/blob/v1.1.1/include/avif/avif.h#L108

Btw, none of the other image libraries have put their default to an extreme value...

E.g. zlib used in PNG (and TIFF) maps their Z_DEFAULT_COMPRESSION (incidentally also defined as -1) to level 6 (of 9).

But I suppose we'd better expose this now (not as reciprocal though, it should correspond to a parameter in the codec documentation/API), as we do w/ JPEG XL, PNG, TIFF... Thanks for narrowing it down.

[MenooMeow]

Where does the parameter AVIF_SPEED_DEFAULT come from on this line?

It comes from libavif itself: https://github.com/AOMediaCodec/libavif/blob/v1.1.1/include/avif/avif.h#L108

Btw, none of the other image libraries have put their default to an extreme value...

E.g. zlib used in PNG (and TIFF) maps their Z_DEFAULT_COMPRESSION (incidentally also defined as -1) to level 6 (of 9).

But I suppose we'd better expose this now (not as reciprocal though, it should correspond to a parameter in the codec documentation/API), as we do w/ JPEG XL, PNG, TIFF... Thanks for narrowing it down.

Thanks. It needs exposure. I just converted 233 pictures to AVIF from RAW and it took 5 days! There's no way this can be used for work with this level of handicap. I think this would go a long way to making Darktable an option against Lightroom.