Apparently ffmpeg doesn't like how we were feeding the SPS/PPS to it. At least this way it works. There is still something wrong. Instead of directly starting to work after the SPS/PPS is received (from MPEG-TS), it waits 1 extra I frame. If I just output the SPS/PPS directly instead of reinserting it, it works directly after the SPS/PPS is received on the I frame directly following them.