Yes you're right it would cause all kinds of problems. The same as, when I see an Woolwich fan on the tube I will instinctively dislike him/her intensely; if I felt that way about people who were another race/gender/sexuality from me I'd be a massive racist/misogynist/homophobe.I would agree that football and sport in general is a great outlet for tribal instincts.
It's just where you said you give him the benefit of the doubt just because he's a fellow yid. No other reason other than he supports spurs. Just imagine if everyone in every kind of group thought like this, the problems in society it would cause.
I just don't see the need to think along the lines of 'if I felt the way I do at football in other group contexts then that would be bad, therefore I'll try and change the way I feel at football'. I think it's fine to be one way at football and a different way in other contexts.