Cancel

Jeff Chandler

Plugins on WordPress.org Now Show More Accurate Ratings Data

Ratings Featured Image
photo credit: Rating – (license)

Those who host plugins in the WordPress plugin directory may have noticed a change to their plugin’s ratings. That’s because the ratings system has been reset and rebuilt by Samuel “Otto” Wood. The ratings now correspond exactly with reviews. According to Wood, the change has been two and a half years in the making:

Back when we launched the review system 2.5 years ago, we tied ratings to reviews. However, up until that point, we had existing ratings in the system. At the time, some argued that the ratings should be wiped out and everybody start fresh. I argued for the opposite, that we should leave the existing ratings in place until such time as we had enough reviews in the system to build up a good body of ratings.

A few weeks ago, Wood was checking out the ratings for the new WordPress theme directory when he noticed there were over 150K reviews. “Out of those 150K reviews, less than 10K are for themes” Wood said. This was enough to initiate the change and remove ratings not attached to a review. Since the system was rebuilt, some plugins have experienced 1 and 5 star rating swings, while plugins like WP eCommerce have seen its average rating rise from 2.9 to 3.4 stars.

In addition to displaying more accurate data, review spam has been neutralized. Review spam has been a serious problem for the past several years. According to Otto, this is no longer the case:

Ask any of the moderation team how many duplicate accounts they’ve seen to vote up their plugin and down those of competitors. Even though the reviews were removed, until yesterday, the ratings made by those accounts remained. This is no longer the case, and the ratings made by those spam accounts has now been effectively neutralized.

In summary, ratings and averages now reflect the most accurate data from the point in time reviews were introduced. Spam and fraudulent data hasn’t been lost or deleted but rather ignored from influencing the data set. Ratings that were added before mandatory reviews are also ignored.

By waiting two and a half years, Wood has avoided resetting the entire system which would have caused everyone to start from zero. This way, authors are able to maintain their ratings and reviews without starting from scratch. If you host a plugin on the WordPress plugin directory, let us know if your ratings changed, especially the average.

Category: ,
Tags: , ,
38

38 responses to “Plugins on WordPress.org Now Show More Accurate Ratings Data”

  1. SO HAPPY ABOUT THIS!

    Thanks Otto and anyone else who had a part in making this happen.

    The new system is fairer.

    Back in the day, it was possible to leave a rating without a review, but this was changed recently.

    People are more likely to leave a 1 star rating when they don’t have to leave a comment/review with it explaining why.

    So once the requirement to include a review with the rating was added in, plugins got less 1 star reviews. But since the old 1 star ratings were never deleted, old plugins were at a disadvantage and had lower average ratings than newer plugins, even if they were of the same quality.

    This change made our average rating for WP All Import jump from 4.1 to 4.4: https://wordpress.org/support/view/plugin-reviews/wp-all-import

    It also reduced our number of 1 star ratings from 27 to 15. And our number of 5 star ratings went from 90 to 87 – which proves people are way more likely to leave a 1 star rating without a review.

  3. Good work, Otto!

    I think this ‘reset’ has been good, but I’d love to see the rating shown always be a rolling “last two years” score.

    Some plugins have >5 year histories, and ratings older than ~2 years are basically meaningless. It also makes it hard for plugins like wp-ecommerce to recover from historically low ratings – a rating 5 years ago has exactly the same weighting as one made yesterday.

    Or, for example, some big company buys out a plugin, and takes it in a new direction – they’ve also bought the ratings history, which is nice for them (presuming they bought a good one), but over time is increasingly misleading for would-be new users.

    • Add a link to inside your plugin saying “If you like the plugin, please leave a review on WordPress.org.”

      You have 10,000+ downloads – I’m sure you can get a few five star reviews posted if you just ask people who genuinely like your plugin!

    • Yes but not for this. You missed a notification (you can use RSS or get emails) so these things can happen. But what you do next can change your review. Ask them about the problem, see if you can solve it, and then if you do, ask the, to change their review from 1 to 4.

      Abusive reviews, unfair reviews, attacks? Yes those are deleted. This one is what we would deem fair. They have a problem, they opened a support request, no one got to them in a couple weeks, they can’t use the plugin.

      To get alerts, go to https://wordpress.org/support/plugin/media-credit and at the bottom there’s a link to the RSS feed or to get email notifications of new support posts.

    • Awesome… Comment was deleted (Jeff – I hit submit and it vanished, no warning no moderation notice, just gone).

      Yes there are ways to report abusive or spam posts however this is not something you would want to flag for ‘review’ or anything because it’s 100% valid.

      Go to https://wordpress.org/support/plugin/media-credit

      At the bottom there’s this: https://cloudup.com/cm2qO243MYO

      You can get email alerts that way. I’ll have to make a post on make/plugins about how you do this, since a lot of people seem to have missed it. But you can get notifications, and if someone submits a support ticket and you missed it, then yes, they should be annoyed.

      How you correct the issue and get a higher rating is all on you, but I would suggest instead of saying ‘we don’t get notifications’ you apologize for missing theirs, link back to it and offer to help. If you resolve the issue, you can then ask them to edit their star rating :) Boom. 4 stars. (IMO I’d personally hold off on 5, since you DID miss the initial support post)

      • Thanks Mika, I have since found that hidden setting, but I didn’t know about it back then. It sure could benefit from some additional publicity for new plugin contributors.

        Back in early 2014, I had just started as a contributor and de facto maintainer of Media Credit after a long period of dormancy for the plugin. I had no idea that there was a way to get hold of all new support requests and so I “patrolled” the forum every other day, initially. Somehow, I still missed that guy’s post. Two and a half weeks later, instead of bump of his original topic to get my/our attention, he posts a new topic, asking whether the plugin was still supported and simultaneously gave the one-star review. I have to admit, that pissed me off somewhat, because we never got the chance to fix the initial oversight. Furthermore, while the bug he reported was fixed about a month later with the next release, he never even bothered to acknowledge this. (I’m not talking about rescinding the bad review here, but about an answer to the initial support post. A “yes, that fixes it” or “no, I’ve still got the following issue” would have been enough.)

        Sorry for the rant, but lack of communication/feedback like this bothers me in an an open source project. (Everything’s different when we are talking about paying customers, of course.)

  5. It’s a nice addition, but I agree with David Anderson that only the recent ratings (e.g. last 1 or 2 years) should be promoted. For example, if a plugin worked great at the beginning but at some point introduces spammy behavior or has a major security flaw, the overall rating might not correspond to it’s recent state. On the other hand, a plugin that fixed it’s bugs would be unfair to have a negative rating because of flaws it used to have a few years ago. For historical reasons the overall rating could also be visible, but in a less prominent way.

    Another thing that I find misleading has to do with the plugin’s downloads. I believe that plugin updates shouldn’t add up to the downloads count. Instead, a download should count only the initial plugin installation at a specific domain. That’s because at the moment a developer could influence this number by releasing frequent meaningless updates, making the plugin appear more popular than it really is.

    • Another thing that I find misleading has to do with the plugin’s downloads. I believe that plugin updates shouldn’t add up to the downloads count. Instead, a download should count only the initial plugin installation at a specific domain.

      I’m not even sure all downloads should count. I sometimes download a plugin to see how it works, check out the code, etc. That doesn’t mean I end up using it. It would be far more interesting and useful to see an “in use” count rather than a downloads count…

      • @Julie @Niackery
        I agree. Sometimes you download something for testing purposes (e.g. on localhost) or you download it and then you deactive or even remove it. As you suggest, the ideal functionality would be something like “this plugin is active in xxx websites right now”, but I don’t know if it’s possible.

      • I download far more plugins than I actually end up using and I’m not a web developer at all. I don’t look at downloads but I did notice a stat about how often the plugin is actually being used. That is something real versus test runs.

    • The download count is a crappy number, we know. Better statistical data regarding plugin installations is coming soon, once we’re confident in the accuracy of the numbers over time.

  6. Too bad there is no solution to change bad rating from people who don’t come back after having an answer to their question or did not make any compatibility test with their theme or other plugins.

  7. Appreciate the upgrade! It’s increased our average rate from 4.4 to 4.5 which is huge! (https://wordpress.org/plugins/rating-widget/)

    I think it’s a smart move, one of my concerns with the previous ratings was that I couldn’t know who are the people behind the votes.

    Having said that, I still think there’s a big issue with the reviews system, many people base their review on expectations, even though it’s not what the plugin does. E.g “That’s the worst plugin ever, it doesn’t support X”. Even if the description page explicitly mention “We do NOT support X”.

    You can’t buy a scooter, and then be pissed off that it’s not a Lamborghini.

    I think that simple bullets with good vs bad reviews examples right before the review form can drastically change that.

    Hope that someone is hearing me there :-)

    • I think the real problem is descriptions and screenshots given for the plugin. Often descriptions leave me guessing as to what the plugin will actually be able to do and how I will be able to use it, or not.

      I tend to skip reading more than the header description and then look for screenshots to see if I can understand the plugin and whether or not it looks too complicated and messy to consider trying at all.

      Often the screenshot says a lot more than the written description. But, if the screenshots interest me I will then go back and read more.

      Some descriptions seem to claim the plugin does more and works better than it actually does. I’ve seen some descriptions actually written for a premium version but posted to the WP plugin directory version. This is confusing and of course it sets higher exceptions than the plugin actually delivers.

  8. The two things that I look at when choosing a plugin are: the rating and the number of outstanding support requests.

    The rating gives a good idea of how people have found the plugin and the support queries answered shows how enthusiastic the author is about providing support.

    I’ll head over and check out the new system

    • I agree that how developer deal with support question tells a lot …
      For me support part is where I look, reviews mostly say nothing.
      In support I see if developer is active there, what kind of problems user have with plugin …

  9. I look at 3 things when choosing a plugin

    1) Rating

    Higher stars tend to mean it is a good coded plugin

    2) If it’s compatible with my WordPress version

    If it hasn’t been updated in a while then the chances of not working with my WP ver. are greater chances

    3) The amount it has been downloaded

    The more people download the plugin, the greater chances it is a good plugin, people won’t download crappy plugins, right?

    I am sure if enough people complain about a crappy plugin, the plugin gets removed, right?

  10. That would be an interesting poll (the criteria based on which someone chooses a plugin). Mine are:
    1. Ratings per Rating number (one or two 5-star ratings mean nothing, especially if they come with no actual review).
    2. Downloads (which, as we already mentioned, could be more accurate)
    3. Author’s responsiveness. This could be measured by the update frequency and how many recent support tickets remain unanswered.

    In my opinion, another problem with the plugin directory is that it’s difficult to distinguish quality plugins from the bad ones. Here’s some of my thoughts about features that could improve that:
    1. Restrict the number of tags a plugin can have, in order to avoid tag-spamming.
    2. Introduce hierarchical categories – e.g. Performance, SEO, Appearance etc. where a plugin can only be under a single category.
    3. Better featured plugins. Right now, featured plugins section shows the same plugins. It would be more useful if it promoted different plugins every day, based on their quality. The criteria could combine things such as the number of positive reviews during the past week, handpick based on some special occasion (e.g. a good security plugin in case of a virus outbreak) or even the author’s reputation (I would be willing to try a new plugin by Otto or Nacin even if it had zero reviews and downloads at the moment).

  11. It’s great to see my Members plugin ratings jump up a bit. I believe it had 19 one-star reviews from anonymous users. Now, it only has 2 one-star reviews. It’s awesome to see who’s behind those bad reviews too. One is from someone who rated the plugin down because it didn’t have the feature he wanted. The second was from a user who made a fixable mistake and blamed the plugin (granted, it’s a feature I could improve). Anyway, now I have that 4.8 rating with new change.

    Everyone loves those 5-star reviews, but I think it’s the 4-star reviews that are the most helpful to developers. Those are usually from people who genuinely love the plugin but have ideas about making it better.

  12. Although its really great every star-rating now has a review matching it, I think its a bit unfair that the older reviews are also closed but apparently count towards the over-all star-rating for the plugin.

    I would prefer to see the reviews remain open so they can be addressed as needed … or to see the star-rating reflect the “current” open reviews only, much like the rolling (current) support topics count.

    • Review threads, like all other threads in the forum, auto-close themselves after 1 year. Honestly, a year seems like plenty of time to respond.

      Now, don’t get me wrong. Reviews, including their ratings, can still be edited by their original authors regardless of the closed state. So, if you address the problem, then there’s still a chance the person can come back and adjust their review. It’s possible.

      • I did that recently. I wrote a 1 star review, then waited a few years to make sure they didn’t make any more botch ups (they broke an upgrade in spectacular fashion) and then bumped it up to a 4 star review. I’d like to have left a comment pointing this out though, but didn’t have the opportunity since the comments were closed by then.

      • I agree a year should be plenty of time to respond … just thinking more consistency as compared to other “metrics” being displayed on the plugin’s description page as being more “fair”.

  13. Are Ratings to be used for (forceful) Help? I’ve seen a few 1 star reviews on nice plugins only because the user didn’t know (or bother) how to use it. :/ Something needs to be done about that.

    Would a voting system on a given review help reviewers give a more helpful rating?
    Such as, other users can vote on a review. “Found this review Helpful?” Y/N.

    • I like the idea of voting reviews up or down, but the system would have to be sophisticated enough to weed out older upvoted reviews. I can just imagine a situation where a plugin introduces an update that breaks things, gets a bunch of bad reviews which are upvoted, then fixes the problem, but the bad reviews remain prominent… :(

      • Hmm, I see your point… In such a situation the Plugin Author would need to work quickly to fix such an oversight. Then the Fixed Updated Plugin reviews (hopefully positive because the Author fixed it quickly) help ‘naturalize’ that effect?

  15. Just noticed the change, I’m now in a happy place as pretty sure had other WordPress SEO Plugin authors/users downgrading my little plugin (Stallion WordPress SEO Plugin) and was rated 1 star on something like 40 ratings with no reviews (lots of 1 stars with no review) which made no sense for a plugin with less than 20K downloads!

    Now it’s rated 4.8 from 5 reviews which makes far more sense for a plugin not many people use.

    David

  16. Plugin ratings and reviews are important. Themes go through a review system before they are listed but plugins don’t have the same standards. So it’s more “user beware”. Not every plugin has a review/ rating but it does give you something to go on when you try to decide which plugin to try. Even the best plugins are a risk to upload and activate.

You may also like

Newsletter

Subscribe Via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

%d bloggers like this: