Based upon a suggestion from Christoph Hellwig.

This fixes various races in module load/unload handling
too.
Signed-off-by: David S. Miller <davem@davemloft.net>