Fix an integer overflow issue

[trylab]

Prevent an integer overflow issue in function opj_pi_create_decode of
pi.c.

Hi, I found an integer overflow issue which could lead to OOB read and OOB write. I'd like to submit the proof-of-concept file and vulnerability report privately. Please give me an email address to report this issue. Thanks.

Regards,
Ke Liu of Tencent's Xuanwu LAB

[detonin]

@trylab could you update the PR by replacing UINT_MAX by (1<<15)-1, hence avoiding inclusion of limits.h ? Thks

[stweil]

Why do you want to avoid limits.h? IMHO using UINT_MAX is better than using an expression, especially if it is not correct (1 << 15 is a much too low limit)?

As opc_calloc takes size_t arguments, using stdint.h and SIZE_MAX would be even better:

if (l_step_l && l_tcp->numlayers < SIZE_MAX / l_step_l - 1)

[trylab]

Hi detonin and stweil, thanks for your reviewing. So which form is more preferable?
I agree that 1<<15 is too small. I'll change the code of this pull request once we come to a conclusion.

[stweil]

SIZE_MAX or ((size_t)-1) would be the real limit, because obj_calloc takes size_t parameters. On 64 bit platforms size_t is larger than OBJ_UINT32.

[trylab]

Infact using SIZE_MAX or ((size_t)-1) is wrong here. Please see the details at #825

[trylab]

Thanks stweil. I'll use SIZE_MAX which was defined in opj_malloc.c.
I think there was something wrong with the travis build system so the check was not passed.

$ export CC=clang-3.9
$ clang-3.9 --version
/home/travis/build.sh: line 45: clang-3.9: command not found
......
./tools/travis-ci/run.sh: line 100: clang-3.9: command not found

BTW, the proof-of-concept file and report have been sent via email on 22nd Aug.

[detonin]

OpenJPEG Integer Overflow in function opj_pi_create_decode of pi.c

[mayeut]

Sorry for my late comment.
The check with sizeof(OPJ_INT16) is unnecessary since it will be done by opj_calloc (through a call to calloc)
We can get rid of the check of l_step_l against 0 since l_tcp->numlayers + 1U > 0U and l_tcp->numlayers + 1U can't overflow.

The check can thus become:
if (l_step_l <= ((SIZE_MAX) / (l_tcp->numlayers + 1U)))

[trylab]

Thanks. I've already updated the code.

[mayeut]

https://pdfium.googlesource.com/pdfium/+/e865ed12c4a476a4c74bf1ae97d3a6fa8ca06f0a/third_party/libopenjpeg20/0002-packet-iterator.patch