It's probably the exact same thing that makes us pull and push putts.
I'm sure it's down to our eyes seeing too many problems and we just don't feel comfortable over the ball. It's never easy aiming left when your stock shot is a draw. Quite often I see people aiming left but they don't then get the ball in the right position accordingly, it's still sitting where it would be if they were straight (or further back).
