Extract out a lot of the x.port. uses and also show up where there are
things left to be isolated that prevent use using the port helpers in the
serial layer at this point
Signed-off-by: Alan Cox <alan@linux.intel.com>