I think I might be at the same point, at the risk of a few disagrees!
I would never advocate a legal ban on the word and it is up to our fans about using it, I think eventually in the longer term it will probably stop, either due to the reality that we don't have a huge Jewish following or the context just doesn't feel right any more.
There is a bit of a counterpoint which we ignore because of 2 reasons. The first reason is that our use of the word comes from a positive place and therefore there is a sense of pride in it's use comes from a good intention.
The second is that we focus on the fact that we are being castigated by fans of Chelsea, who have some serious fan base issues with racism, particularly anti semitic nature which means that our defence is that some of their fans use of the word is absolutely abhorrent, the problem is that when Baddiel makes his comments we can't separate the fact he's both Jewish and a Chelsea fan because the two things seem to be at odds with one another. The only thing I will say as a minor point of defence is that both brothers do condemn the words and actions of their own fans too, as they should.
The fact is that the word 'yid' is deemed offensive. I think none of those who chant it would use the word outside the football context.
There are Jewish people who find even the badge of honour tag offensive when we use it, they're not all Chelsea fans and we need to be aware of that as a group of supporters.
The difficulty I have is that the word isn't ours to own, even if it's used for the best reasons. Just start looking at the logical extension of this with other racial, sexist or homophobic slurs and consider how comfortable you would be with chanting them, even as a badge of honour.