That would be an interesting poll (the criteria based on which someone chooses a plugin). Mine are:
1. Ratings per Rating number (one or two 5-star ratings mean nothing, especially if they come with no actual review).
2. Downloads (which, as we already mentioned, could be more accurate)
3. Author's responsiveness. This could be measured by the update frequency and how many recent support tickets remain unanswered.

In my opinion, another problem with the plugin directory is that it's difficult to distinguish quality plugins from the bad ones. Here's some of my thoughts about features that could improve that:
1. Restrict the number of tags a plugin can have, in order to avoid tag-spamming.
2. Introduce hierarchical categories - e.g. Performance, SEO, Appearance etc. where a plugin can only be under a single category.
3. Better featured plugins. Right now, featured plugins section shows the same plugins. It would be more useful if it promoted different plugins every day, based on their quality. The criteria could combine things such as the number of positive reviews during the past week, handpick based on some special occasion (e.g. a good security plugin in case of a virus outbreak) or even the author's reputation (I would be willing to try a new plugin by Otto or Nacin even if it had zero reviews and downloads at the moment).