i think thats how the real controls work...
it might be a bit annoying when you need to manoeuvre but if you use shift instead of precision toggle you can release shift and quickly switch to the other side and then hold it again to steer more precize