Commit 8f9f42c1 authored by Martin Storsjo's avatar Martin Storsjo

Don't try to shift more bits than the variable length

Shifting by more than (or equal to) the variable length is
undefined in C.

This caused the quantized spectrum values to remain unshifted,
causing loud beeps in some samples.

The exact same change was originally applied in
65745694 (for AOSP) and in 24021f19 (in the separate fdk-aac
repo), but was overwritten and silently reverted by the new
upstream code drop for Android 4.2 in 381d6984. After the code
drop, I chose not to reapply this change since the crashes that
it fixed had been fixed in other ways upstream and I was unable to
reproduce them after the new code drop.
parent b966edf7
......@@ -127,7 +127,10 @@ static void FDKaacEnc_quantizeLines(INT gain,
accu = fMultDiv2(FDKaacEnc_mTab_3_4[tabIndex],FDKaacEnc_quantTableE[totalShift&3]);
totalShift = (16-4)-(3*(totalShift>>2));
FDK_ASSERT(totalShift >=0); /* MAX_QUANT_VIOLATION */
accu>>=totalShift;
if (totalShift < 32)
accu>>=totalShift;
else
accu = 0;
quaSpectrum[line] = (SHORT)(-((LONG)(k + accu) >> (DFRACT_BITS-1-16)));
}
else if(accu > FL2FXCONST_DBL(0.0f))
......@@ -140,7 +143,10 @@ static void FDKaacEnc_quantizeLines(INT gain,
accu = fMultDiv2(FDKaacEnc_mTab_3_4[tabIndex],FDKaacEnc_quantTableE[totalShift&3]);
totalShift = (16-4)-(3*(totalShift>>2));
FDK_ASSERT(totalShift >=0); /* MAX_QUANT_VIOLATION */
accu>>=totalShift;
if (totalShift < 32)
accu>>=totalShift;
else
accu = 0;
quaSpectrum[line] = (SHORT)((LONG)(k + accu) >> (DFRACT_BITS-1-16));
}
else
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment