avcodec: return -ENOMEN when there is no pictures

In case of ENOMEN, the current avcodec worker thread will be stopped.

As advised by lu_zero.

modules/codec/avcodec/video.c

@@ -1097,7 +1097,7 @@ static int lavc_GetFrame(struct AVCodecContext *ctx, AVFrame *frame, int flags)
 
     pic = decoder_GetPicture(dec);
     if (pic == NULL)
-        return -1;
+        return -ENOMEM;
 
     if (sys->p_va != NULL)
         return lavc_va_GetFrame(ctx, frame, pic);