src/common_utils.c: bound reorder copy in WavpackGetChannelLayout

[aizu-m]

Turned up while fuzzing the file-open path under ASan: a crafted 2-channel file declares 255 channels in its NEW_CONFIG layout word, and WavpackGetChannelLayout copies (channel_layout & 0xff) bytes from channel_reordering. That count is taken straight from the metadata in read_new_config_info and is never checked against the real channel count.

Per the discussion, the proper fix lives at open time. read_new_config_info can't do the check itself, but by the point WavpackOpenFileInputEx64 finalises num_channels (including the MONO_FLAG fallback for files with no CHANNEL_INFO block) the channel count is known, so the layout count is verifiable there. A layout claiming more channels than the file has is invalid, so the file is rejected with "invalid channel layout in file!".

The bound added in WavpackGetChannelLayout stays as a backstop.

Fuzzer change reverted: the harness sizes its reorder buffer by the layout count, matching the library docs and the command-line tools.

Regression file fuzzing/regression/new-config-channel-reorder-overflow.wv (255-channel layout on a 2-channel file) is now rejected at open; all valid corpus files still open.

[dbry]

Thanks for the bug report! I have dug into this a little and have the following observations:

1. The libwavpack documentation specifically says the number of channels in the layout reordering string should be obtained from the layout word, and every call in the command-line programs verifies that this is less than or equal to the channel count before allocating the reordering string. It also says that it probably should match the number of channels in the file, but it does not suggest that it must, and it's perfectly valid for it to be less than the channel count. I don't know why you say that real callers would just allocate based on the channel count without checking the layout first and I don't think it's appropriate to change the fuzzer to do that.
2. You also suggest that read_new_config_info() cannot do the check because the channel count is not known then. If you look at send_general_metadata() in src/pack.c, it's clear that the NEW_CONFIG metadata block is written last, so it should be possible to flag the error there instead, although it's a little tricky because a standard stereo file does not have the simpler CHANNEL_INFO block (the channel count is inferred later). However, I still feel that this would be a more appropriate fix, although your fix might serve as an additional backstop.

[aizu-m]

Both points make sense, thanks for digging in.

1. You're right, I conflated two things. The docs do say the reorder length comes from the layout word, and the tools allocate against that. Reverted the fuzzer hunk so it sizes the buffer by the layout count again.

2. Followed your steer to flag it at open. read_new_config_info still can't check (num_channels isn't final yet), but WavpackOpenFileInputEx64 sets the count just after parsing, including the MONO_FLAG fallback for the stereo case with no CHANNEL_INFO block. Added the check right there: if (channel_layout & 0xff) > num_channels the file is rejected. Left the bound in WavpackGetChannelLayout as the backstop you mentioned.

Traced it back through: the regression file (255-channel layout on a 2-channel file) now fails to open with "invalid channel layout in file!", and the 19 valid files in the seed corpus and regression set all still open clean.