return AVERROR(ENOMEM) if alloc failed, set count after alloc, use variable when reallocing, based on patch by Netgem

git-svn-id: file:///var/local/repositories/ffmpeg/trunk@17509 9553f0bf-9b14-0410-a0b8-cfaf0461ba5b