Voters haven't always loved floor-crossers