One of the main reasons why this got blown way out of proportion is that there are interest groups and people in the public eye that purport to speak for an entire demographic. Out of all the material that was included in this special, and generally throughout his career, this joke shouldn't have even registered as a blip on the public's radar. It is a comedian's job to test the limits. If he was a public official, it would be one thing. But, riffing on stage is not the same thing.
The public attention has probably done more to boost his ratings than cancel him, which is the funniest part of this whole debacle.