stream_out: rtp: don't use VLA for user controlled data

It should fix a possible invalid memory access

When streaming ogg-files via rtp, an ogg-file can trigger an invalid
write access using an overly long 'configuration' string.

The original code attemps to allocate space to hold the string on the stack
and hence, cannot verify if allocation succeeds. Instead, we now allocate the
buffer on the heap and return if allocation fails.

In detail, rtp_packetize_xiph_config allocates a buffer on the stack at (1) where
the size depends on the local variable 'len'. The variable 'len' is
calculated at (0) to be the length of a string contained in a specially
crafted Ogg Vorbis file, and therefore, it is attacker-controlled.
Signed-off-by: Jean-Baptiste Kempf <jb@videolan.org>

modules/stream_out/rtpfmt.c

@@ -559,7 +559,11 @@ int rtp_packetize_xiph_config( sout_stream_id_sys_t *id, const char *fmtp,
     char *end = strchr(start, ';');
     assert(end != NULL);
     size_t len = end - start;
-    char b64[len + 1];
+
+    char *b64 = malloc(len + 1);
+    if(!b64)
+        return VLC_EGENERIC;
+
     memcpy(b64, start, len);
     b64[len] = '\0';
 
@@ -569,6 +573,7 @@ int rtp_packetize_xiph_config( sout_stream_id_sys_t *id, const char *fmtp,
     int i_data;
 
     i_data = vlc_b64_decode_binary(&p_orig, b64);
+    free(b64);
     if (i_data <= 9)
     {
         free(p_orig);