Integer-overflow in void scale_coefficients_internal<unsigned short>

[bobfriesenhahn]

GraphicsMagick oss-fuzz testing has discovered another issue (integer overflow) in libheif (or libde265). Please make me aware if the report is misdirected, although it seems that the same author maintains libde265.

This is the reported call stack:

/src/libde265/libde265/transform.cc:479:21: runtime error: signed integer overflow: -29431 * 73728 cannot be represented in type 'int'
#0 0x55c09793b549 in void scale_coefficients_internal(thread_context*, int, int, int, int, int, int, bool, bool, int) libde265/libde265/transform.cc:479:21
#1 0x55c097939df8 in scale_coefficients(thread_context*, int, int, int, int, int, int, bool, bool, int) libde265/libde265/transform.cc:650:5
#2 0x55c09792639e in decode_TU(thread_context*, int, int, int, int, int, int, PredMode, bool) libde265/libde265/slice.cc:0
#3 0x55c097924cdd in read_transform_unit(thread_context*, int, int, int, int, int, int, int, int, int, int, int, int) libde265/libde265/slice.cc:3703:9
#4 0x55c097927114 in read_transform_tree(thread_context*, int, int, int, int, int, int, int, int, int, int, int, PredMode, unsigned char, unsigned char) libde265/libde265/slice.cc:3973:5
#5 0x55c097926edf in read_transform_tree(thread_context*, int, int, int, int, int, int, int, int, int, int, int, PredMode, unsigned char, unsigned char) libde265/libde265/slice.cc:3950:5
#6 0x55c097929939 in read_coding_unit(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4585:9
#7 0x55c097921578 in read_coding_quadtree(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4648:7
#8 0x55c0979215ba in read_coding_quadtree(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4651:7
#9 0x55c0979215ba in read_coding_quadtree(thread_context*, int, int, int, int) libde265/libde265/slice.cc:4651:7
#10 0x55c09792bae8 in decode_substream(thread_context*, bool, bool) libde265/libde265/slice.cc:4751:5
#11 0x55c09792d1bf in thread_task_ctb_row::work() libde265/libde265/slice.cc:5001:3
#12 0x55c0979362dd in worker_thread(thread_pool*) libde265/libde265/threads.cc:164:11
#13 0x55c097936b6a in std::__1::__invoke_result_impl<void, void ()(thread_pool), thread_pool*>::type std::__1::__invoke[abi:ne220000]<void ()(thread_pool), thread_pool*>(void (&&)(thread_pool), thread_pool*&&) /usr/local/include/c++/v1/__type_traits/invoke.h:87:27
#14 0x55c097936b6a in void std::__1::__thread_execute[abi:ne220000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, void ()(thread_pool), thread_pool*, 0ul, 1ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, void ()(thread_pool), thread_pool*>&, std::__1::__integer_sequence<unsigned long, 0ul, 1ul>) /usr/local/include/c++/v1/__thread/thread.h:159:3
#15 0x55c097936b6a in void* std::__1::__thread_proxy[abi:ne220000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_deletestd::__1::__thread_struct>, void ()(thread_pool), thread_pool*>>(void*) /usr/local/include/c++/v1/__thread/thread.h:167:3
#16 0x7f47e53d2608 in start_thread /build/glibc-B3wQXB/glibc-2.31/nptl/pthread_create.c:477:8
#17 0x7f47e52f7352 in __clone /build/glibc-B3wQXB/glibc-2.31/sysdeps/unix/sysv/linux/x86_64/clone.S:95

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior libde265/libde265/transform.cc:479:21

These are the two test cases which were provided:

clusterfuzz-testcase-minimized-coder_HEIC_fuzzer-6627776152272896.gz

clusterfuzz-testcase-coder_HEIC_fuzzer-6627776152272896.gz

[farindk]

Should be fixed with strukturag/libde265@3f9bd7d.
The overflow does not matter since it is clamped to a valid integer value anyways and there is no "correct" way to decode an "incorrect" stream.

[bobfriesenhahn]

Oss-fuzz has detected that this issue is fixed.