At the moment, travel time and cost are roughly correlated to distance from the ground. So fans in Maidstone, Cambridge, MK, Reading, will all have roughly the same burden. Fans in Cheshunt, Barnet, Loughton - similar burden to one another. Draw a circle round Tottenham with radius of x miles, and all the fans living on that circle are in roughly the same situation, and the bigger that circle gets, the fewer Spurs fans there are. (Yes this ignores the distorting effect of London, but it's broadly true.)
Now consider moving the ground to MK. Suddenly, the many thousands of fans living within 10-15 miles of Tottenham are facing a big journey to get to matches, while the lucky few who live in and around MK have struck gold. As for the poor fans in Kent/Sussex who are used to already making a big journey to the Lane - they might as well forget it if we go to MK.
Ultimately it comes down to the fact that our match-going fanbase is concentrated mainly around north east London, Herts and Essex - and most of them will be shafted by us moving to MK. The average travel distance to matches would increase drastically, and loads of people suddenly wouldn't be able to afford to go to every match any more.