Thanks to tip form ARM folks and Russell King.

If flush_dcache_page() occurs on a swapin it will have a mapping
and we'll try to defer the flush by setting the dirty bit.

But when it hits update_dcache_page() we won't flush because the
page won't have a mapping any more.  So remove the mapping
requirement in flush_dcache().
Signed-off-by: David S. Miller <davem@davemloft.net>