Previously, if the absolute delay dropped by a longer interval than the
packetization time, we could end up repeatedly waiting into the future,
and never dequeue anything.