Change remove code so that we first detach
the driver from userspace, then clean up the
clock and then clean up the memory we allocated.
Signed-off-by: Wim Van Sebroeck <wim@iguana.be>