@Chip Bennett -

It is deliberate design intent. WordPress assumes, and intends, that the user will choose either the Visual editor, or the HTML editor, but not both during the same editing session.

I really don't think this IS or SHOULD BE the way WordPress core developers view editing. In my experience normal users switch back and forth regularly to write code that can't be generated by WYSIWYG and to check/validate what the WYSIWYG is doing behind the scenes. This is totally natural and something that is implied to be okay by the UI which makes switching back and forth so easy.

Also remember that posts can be edited by multiple users, so even if one user has Visual Editor turned off in their settings, another user can come and without even noticing, simply by opening the post editor, completely mess up formatting that the non-visual user had created. This is a huge problem, and the only solution, unless you are the only person who can edit your posts and you NEVER use the visual editor, is to avoid any formatting that will get broken by the visual editor. I do this by switching back and forth if I'm ever in the HTML editor and not sure if a given bit of code will work when switched over.

Avoiding all non-WYSIWYG-compatible HTML is not ideal, but assuming that users will only use one or the other is just not realistic, and I hope that isn't what "WordPress assumes" because if so it's fantasizing.

If you want to be able to switch between the two, it is simple enough to filter the TinyMCE init configuration to prevent the wonky behavior caused by the switch.

"Simple enough" is pretty relative. Most people need solutions that work out of the box, and in this case having links in captions is not something, at least IMHO, that should involve avoiding the Visual Editor entirely or having to install/write plugins that hack it's internal behavior.