i think you are forgetting the resposibility that goes along with the freedom of speech you are able to excercise. We all have the right to say whatever we please, what we all tend to forget is that those things we say usually carry consequences that we have to endue.
Imus, and anyone else has the right to say what they please. What he forget to do (and what gets alot of people in trouble) is weight the consequences of his words BEFORE he opened his mouth.
He said what he pleased, now he's got to pay the consequences. It's that way for anyone. You can go into a crowded club full of black people and yell the N word. You have the right to do that. BUT, you also need to realize that you will pay the consequences for the freedom to say that.
Nobody is suffering from a violation of their freedoom of speech. It's just that in this day and age, nobody thinks they have to be responsible for anything they say or do. When someone is held accountable for their actions and/or speech, they get all pissed off and whine that their freedoms are being taken away.