That's a great discussion actually and there are several aspects to be covered there.

First off, what would a review be like? Similarly to the plugin reviews, it's likely that reviews could be boosted positively by colleagues or negatively even by bots and other trolls. It's hard to put up criteria for voting that is objective enough. IIRC the Foundation actually requests that after a WordCamp speakers should be evaluated by the local WordCamp community in a poll but some WordCamps don't follow that, not sure if it's a requirement or "good to have" but I remember voting for some events, including sending that form for WordCamp Sofia 2012.

Another thing I like to ask for is a demo of the slides + a quick video for new speakers. It's not so complicated to record a 3-5min video from a webcam (most notebooks have those) to verify the idea and speaker skills .

Additionally I'd like to cover the community side. In order for the community to grow, we need to welcome and embrace changes, including new speakers, authors, developers etc. If you review the major WordCamps you will notice a significant group of speakers speaking at several consecutive events. Now, I'm not saying that there should be a limit of events a speaker should be entitled to speak at, but I have spoken with several people interested in applying for speaking responding with: "Well probably 2/3 of the speakers are regular people speaking everywhere, or related to the organizing committee so I don't stand a chance and people won't recognize me". I'm not sure what's the best way to encourage new speakers to join the fun but adding other layers of complications are another step of discouraging new people to try due to the fear of failure that everyone is afraid of.