I remember this debate. The reason I don't agree with you is that it limits a clubs potential based on a name. Everton, for instance, should only be able to build a new stadium in the Everton area, but Liverpool have the whole city to build one. Villa must remain in the Aston area, yet Birmingham have 100 square miles to find a site.FC Hotspur. Don't see how we could remain Tottenham Hotspur if we don't play in Tottenham, might have involved a name change ala Woolwich, MK Dons.
I could go on. The clubs roots would always be in Tottenham, but the vast majority of our fans have no links to the area of Tottenham. Anyway, hypothetical argument now.