Orbits, distance, and the mechanics of going anywhere.
Earth hands you 29 kilometres per second sideways, and every burn you can afford is added to it — so the tilt you can buy runs out long before the Sun's poles.
Plot a real transfer from Earth to any planet, and see why you never aim at the target.