Remove module will not free L1 memory used which caused by
memory access after free.  This patch fixes it.
Signed-off-by: Meihui Fan <mhfan@hhcn.com>
Signed-off-by: Bryan Wu <cooloney@kernel.org>