At gadget module remove function, it closes the clock, but
at platform code, it still visits register. In fact,
The PHY has already been low power mode when driver's remove
before platform code is called.
Signed-off-by: Peter Chen <peter.chen@freescale.com>