Set rc_qsquish=1 in AVContext when b_strict is used. This uses a nice continous function to limit qscale within qmin/qmax.