integer overflow in sixel_frame_resize in frame.c

[sleicasper]

In function sixel_frame_resize, width and height can be specified by user. Line 503 has an integer overflow. If width and height are very large numbers, allocation will fail.

poc:
poc.zip

result:

./img2sixel -w 1000000 -h 100000 ./0.png
=================================================================
==85426==ERROR: AddressSanitizer: requested allocation size 0xffffffffd964b800 (0xffffffffd964c800 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
    #0 0x493c3d in malloc /tmp/final/llvm.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:145:3
    #1 0x4c6a67 in rpl_malloc /home/casper/targets/struct/libsixel/source/BUILD/converters/malloc_stub.c:45:20
    #2 0x4d6ba6 in sixel_allocator_malloc /home/casper/targets/struct/libsixel/source/BUILD/src/allocator.c:155:12
    #3 0x4f2f9f in sixel_frame_resize /home/casper/targets/struct/libsixel/source/BUILD/src/frame.c:504:37
    #4 0x4d29a2 in sixel_encoder_do_resize /home/casper/targets/struct/libsixel/source/BUILD/src/encoder.c:637:18
    #5 0x4d1141 in sixel_encoder_encode_frame /home/casper/targets/struct/libsixel/source/BUILD/src/encoder.c:962:18
    #6 0x4d0b73 in load_image_callback /home/casper/targets/struct/libsixel/source/BUILD/src/encoder.c:1673:12
    #7 0x4ff4a8 in load_with_builtin /home/casper/targets/struct/libsixel/source/BUILD/src/loader.c:943:14
    #8 0x4fddc2 in sixel_helper_load_image_file /home/casper/targets/struct/libsixel/source/BUILD/src/loader.c:1392:18
    #9 0x4d0857 in sixel_encoder_encode /home/casper/targets/struct/libsixel/source/BUILD/src/encoder.c:1737:14
    #10 0x4c66c8 in main /home/casper/targets/struct/libsixel/source/BUILD/converters/img2sixel.c:457:22
    #11 0x7f186b53eb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

==85426==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: allocation-size-too-big /tmp/final/llvm.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:145:3 in malloc
==85426==ABORTING

[carnil]

CVE-2019-20205 has been assigned for this issue.

[saitoha]

Fixed on v1.8.5. Thanks!