I think WordPress.tv is quite good already. The only things I can think of to improve are getting rid of the social media junk on each video, and upgrading the quality of the video streams.

I've noticed that I can sometimes stream 1080p 60fps YouTube videos, yet struggle to stream a low bitrate video on WordPress.tv. Video streaming is tough. I think it would make more sense to store the original videos on WordPress.tv for archiving purposes, and for allowing downloading of videos, but to upload the videos to something like Vimeo, Viddler, YouTube, or whatever service is deemed most appropriate. I think those dedicated services offer a much more reliable and pleasant streaming service than what can be done in house. Although if you can figure out how to do it well on a self-hosted setup, that'd be awesome too :)

And please KEEP the easy to use download buttons. Those are much more user friendly than the hackish ways I need to do that for YouTube.