At least tusb implements set_clock(), and in that
case clk_disable() gets called twice. Do not call
clk_disable() if set_clock() is implemented.
Signed-off-by: Tony Lindgren <tony@atomide.com>