reduce size limit for scanline files; prevent large chunkoffset allocations

[peterhillman]

This change introduces a new function into the API
Specially crafted OpenEXR files can cause large amounts of memory to be allocated by the library when reading code, even if the file is very small. This PR proposes two ways to mitigate this:

• regular (non-deep) scanline files are now limited to less than 2GB of uncompressed data per chunk of scanlines, Previously the maximum size was 2GB of compressed data per scanline chunk, and 2GB uncompressed data per scanline
• if the computed chunkoffsettable size in scanline or multipart images is very large, a seek followed by temporary read of the last element in the chunk table is attempted before allocating memory. This may cause slightly slower initialization of OpenEXR files containing huge images. This helps to prevent very small OpenEXR files being used to cause a target machine to allocate large amounts of memory, which may form part of a (very obscure) denial of service attack. With this change, memory is only allocated if the file really does contain enough data to store the declared size of chunk table, which means large memory allocations for the table would require an even larger file to be transferred to the target machine

It is not known if any legitimate EXRs exist that have more than 2GB of uncompressed data per scanline chunk. If so, they will rely on effective compression to prevent the compressed data size exceeding 2GB. Such files will have many channels (in which case using multiple parts would be more appropriate) or very wide scanlines (in which case tiled images would be more appropriate). Uncompressed files, and zip-single compressed files, are not affected by this change.

This PR moves int numLinesInBuffer(Compression comp) into ImfCompressor.h and makes it part of the API. It previously was in an anonymous namespace.

Addresses the following oss-fuzz issues:

• https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24573
• https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24857
• https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25002

Signed-off-by: Peter Hillman [email protected]

[meshula]

lgtm

[peterhillman]

Moving the test earlier in the function should also address https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25156

[cary-ilm]

Since this is declared in ImfCompressor.h, does it need to be declared here, too?

[cary-ilm]

What do you mean "file is big enough to contain the file"? And is the trick here that the read with throw an exception if the size is off? If so, it would be good to state that's the expectation.

Same comment in ImfScanLineInputFile below, too.

[peterhillman]

good catches: Dumb typing errors. It should make more sense now

[cary-ilm]

LGTM