Origin: https://github.com/webmproject/libvpx/commit/61c4d556bd03b97d84e3fa49180d14bde5a62baa From 61c4d556bd03b97d84e3fa49180d14bde5a62baa Mon Sep 17 00:00:00 2001 From: Wan-Teh Chang Date: Fri, 12 Apr 2024 15:48:04 -0700 Subject: [PATCH] Fix a bug in alloc_size for high bit depths I introduced this bug in commit 2e32276: https://chromium-review.googlesource.com/c/webm/libvpx/+/5446333 I changed the line stride_in_bytes = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s * 2 : s; to three lines: s = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s * 2 : s; if (s > INT_MAX) goto fail; stride_in_bytes = (int)s; But I didn't realize that `s` is used later in the calculation of alloc_size. As a quick fix, undo the effect of s * 2 for high bit depths after `s` has been assigned to stride_in_bytes. Bug: chromium:332382766 Change-Id: I53fbf405555645ab1d7254d31aadabe4f426be8c (cherry picked from commit 74c70af01667733483dc69298b8921779f5f6ff3) --- vpx/src/vpx_image.c | 1 + 1 file changed, 1 insertion(+) --- a/vpx/src/vpx_image.c +++ b/vpx/src/vpx_image.c @@ -95,6 +95,7 @@ static vpx_image_t *img_alloc_helper(vpx s = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s * 2 : s; if (s > INT_MAX) goto fail; stride_in_bytes = (int)s; + s = (fmt & VPX_IMG_FMT_HIGHBITDEPTH) ? s / 2 : s; /* Allocate the new image */ if (!img) {