If anything is going to be removed, there must be a way for the user to re-add it. A customizable tool bar would be ideal. I deal with a lot of historic content on our site from old newspaper and magazine articles. Full-justify is something that gets used on a regular basis.

As far as bouncing back and forth between visual and text views, I do it all of the time. The 'visual' view is lacking in many aspects. For example, I use the Columns plugin by Konstantin Kovshenin. Because it's invoked using a short code, the visual editor doesn't know what to do with it.

The visual editor also has a nasty habit of putting media references where it want to, not where I want the item to appear. I am constantly switching to text view to clean up.