Do not malloc 900Mb, for example...

Close #5689
(cherry picked from commit ef466bbdae537748a588bac3eb55bbaa6a989a88)
Signed-off-by: Jean-Baptiste Kempf <jb@videolan.org>