In the context of the entire conversation, which I heard at some point when this story first broke but haven't heard since, I think Imus was joking that the team looked very manly. At one point he said they looked like the Toronto Raptors.
That being said, I think someone could argue that he was talking about the
entire team. I personally don't think that he was referring to everyone, but I could see someone thinking that.
Here's a transcript.
There are definitely strong racial aspects to it, but I think he's being sexist more than anything else.