Look...let me shed some light on this.
1. No one "really" cares who got in trouble with the law on either team as long as the team is winning
2. On a Tuesday, when you see a guy wearing a yellow tie and you hate yellow, I am sure you don't get in his face and scream at him. So...if its a Saturday and a guy is wearing a shirt of the opposing team you don't need to get in his face.
3. It is not how YOU think you are behaving...it is how the people around you feel you are behaving. It is like letting out a loud fart at dinner and saying "Oh, don't worry, mine don't smell" it really is not the point.