I think one of the biggest drawbacks of developing for them (and therefore using them) in the first place is that most themes don't develop for them.

The chicken and the egg...

As long as that is the case and therefore theme switching is potentially destroying one's site, I think nothing much will happen with Post Formats, I know I don't develop for them and actually remove them from sites I develop for clients.