If too much frames have been dropped, then only decode I frames for a while and see if you can catch up with decoding other frames as well.