stream_out: rtp: don't use VLA for user controlled data

It should fix a possible invalid memory access

When streaming ogg-files via rtp, an ogg-file can trigger an invalid
write access using an overly long 'configuration' string.

The original code attemps to allocate space to hold the string on the stack
and hence, cannot verify if allocation succeeds. Instead, we now allocate the
buffer on the heap and return if allocation fails.

In detail, rtp_packetize_xiph_config allocates a buffer on the stack at (1) where
the size depends on the local variable 'len'. The variable 'len' is
calculated at (0) to be the length of a string contained in a specially
crafted Ogg Vorbis file, and therefore, it is attacker-controlled.
Signed-off-by: Jean-Baptiste Kempf <jb@videolan.org>
(cherry picked from commit 204291467724867b79735c0ee3aeb0dbc2200f97)
Signed-off-by: Jean-Baptiste Kempf <jb@videolan.org>

modules/stream_out/rtpfmt.c

@@ -557,7 +557,11 @@ int rtp_packetize_xiph_config( sout_stream_id_sys_t *id, const char *fmtp,
     char *end = strchr(start, ';');
     assert(end != NULL);
     size_t len = end - start;
-    char b64[len + 1];
+
+    char *b64 = malloc(len + 1);
+    if(!b64)
+        return VLC_EGENERIC;
+
     memcpy(b64, start, len);
     b64[len] = '\0';
 
@@ -567,6 +571,7 @@ int rtp_packetize_xiph_config( sout_stream_id_sys_t *id, const char *fmtp,
     int i_data;
 
     i_data = vlc_b64_decode_binary(&p_orig, b64);
+    free(b64);
     if (i_data <= 9)
     {
         free(p_orig);