Hey Jeffro thanks for writing this up! What a mess this situation is. I just spent a few hours investigating the underlying code to figure out WTF is going on and what we might be able to do to fix it. I updated the trac ticket with a LOT of info and ideas in case anyone is interested:

http://core.trac.wordpress.org/ticket/18311#comment:9

It's a huge mess with lots of contributing factors blocking HTML from being present in caption text. Fundamentally it comes down to the fact that the caption text is stored in an attribute of the shortcode, rather than between and . The result is that it can only have the kinds of things that normally make sense in an HTML attribute (attributes are things in html tags like href or src), which notably does not usually include other HTML tags! Ideally we'd rework the shortcode entirely, which would suit me fine because it has never really seemed like the right way to me, but is probably never going to happen.

Instead if we want links in captions we'll probably need to hack them in with some crazy mess of code as well as writing a lot of supporting code for other parts of the admin UI that deal with images and captions. Ugh.