[1.5] 16bit compression failure

[gcode-importer]

Originally reported on Google Code with ID 62

What steps will reproduce the problem?
1. image_to_j2k -i tmp.raw -o tmp.j2k -F 512,512,1,16,u

if I compress using the signed flag, the compression goes well.

tmp.raw is attached here 7zipped.

What is the expected output? What do you see instead?
- codec crashes on freeing 

What version of the product are you using? On what operating system?
1.4, Win7 x64, library built 32bits

Reported by oclandi on 2011-01-31 08:38:33

• Blocked on: Heap corruption in j2k encoder #5

---

- _Attachment: [tmp.7z](https://storage.googleapis.com/google-code-attachments/openjpeg/issue-62/comment-0/tmp.7z)_

[gcode-importer]

see this thread : https://groups.google.com/d/topic/openjpeg/DLVrRKbTeI0/discussion

Hi,

The problem is the following:
- 'cblk->data' contains the compressed code-stream of a code-block
- In JPEG 2000, a code-block has a maximum of 4096 coefficients
- As a maximum of 16-bits precision is accepted, it seemed enough to developers to
allocate (4096*2) bytes to store compressed data in a code-block (assuming that the
compression ratio for a code-block is > 1, which is a good assumption).
- The problem not taken into account is : when converting pixels in the wavelet domain,
16-bits precision pixels might become 19-bits precision coefficients. 
- For very hard-to-compress code-blocks, it might therefore lead to buffer overflow.

In the present case, the original image is a *signed* image. If we try to compress
it as an *unsigned* image, the compressor is very inefficient and leads to the scenario
described above. This explains why it works when allocating bigger buffer (actually
19*4096/8 is enough). The blurred image described by Winfried is normal as it is originally
a *signed* image.

Now the solutions : 
1) Replace '8192' by '9728' when allocating cblk->data in tcd_malloc_encode and tcd_init_encode.
2) Initialize the buffer to a size suited for most compression scenarios and reallocate
it in case we face a potential overflow. Btw, it seems to be done that way for decoding
images (see t2.c, l571) : 

cblk->data = (unsigned char*) opj_realloc(cblk->data, (cblk->len + seg->newlen) * sizeof(unsigned
char*));

Solution 2 preferred of course. Solution 1 might be a temporary fix but uses a lot
more memory.

NB : strangely, boths signed and unsigned scenarios work fine on macosx, without any
fix.

Cheers,

Antonin

Reported by detonin on 2011-02-17 20:08:39

• Status changed: Started

[gcode-importer]

Reported by malaterre on 2012-05-29 12:44:24

• Labels added: Milestone-Release2.0

[gcode-importer]

Reported by malaterre on 2012-05-29 17:12:42

• Blocked on: #

[gcode-importer]

antonin the code should now read:

cblk->data = (unsigned char*) opj_realloc(cblk->data, (cblk->len + seg->newlen) * sizeof(unsigned
char));

see r1322. Make sure your openjpeg copy is up to date.

Reported by malaterre on 2012-05-29 17:25:33

[gcode-importer]

This is still not fixed in v2.

The malloc is now :
p_code_block->data = (OPJ_BYTE*) opj_malloc(OPJ_J2K_DEFAULT_CBLK_DATA_SIZE);

with
#define OPJ_J2K_DEFAULT_CBLK_DATA_SIZE 8192

Reported by jf.pambrun on 2013-12-15 18:21:05

[gcode-importer]

My issue was in part cause by the assumption that raw input is big-endian while my input
file was little-endian. In the incorrect byte order, the wavelet transform is extremely
inefficient and may require more than 16bits per coefficient. With the default codeblock
size of 64x64 and buffer size of 8192 ,this may cause buffer overflows.

The documentation should clearly indicate that the raw input format assumes big-endian
ordering.

Reported by jf.pambrun on 2013-12-16 00:29:38

[gcode-importer]

This issue was closed by revision r2411.

Reported by malaterre on 2014-02-25 10:37:07

• Status changed: Fixed

[gcode-importer]

Reported by malaterre on 2014-02-25 16:05:56

• Blocking: lossy 16bit compression changes black to white #141