That's a component of TinyMCE. I wouldn't agree with removing it as I consider special characters to be a necessary part of word processing.

I wasn't aware that it was a standard part of the editor though, I already use a plugin to customize the visual editor to make it more similar to standard word processing tools.