What Good Is Plugin Compatibility Data If Users Are Not Participating?

PluginCompatibility It’s a fear many WordPress users had in 2009 and it’s one that continues to be near the top in terms of reasons why people won’t upgrade their sites. It’s the fear that their plugins won’t be compatible with the new version of WordPress. Back in October of 2009, the WordPress core team tried to address these fears by adding a plugin compatibility section to the WordPress.org Plugin repository.

Developed by Michael Adams, the plugin compatibility section worked on the simple premise of selecting which version of WordPress was being used with which version of the plugin and clicking on either a broken or works button. The only people who could vote were those who had a WordPress.org user account. After voting, a bar was displayed showing two percentages. If there was no data to be shown, a yellow bar was displayed saying no data. A complaint heard from plugin developers at the time was that the percentages had no clear meaning.


For example, taking the Google XML Sitemaps plugin shown above, what does 88% really mean? That it *will* work on my setup? Or only 88% of the time? Or I have an 88% chance of it working? The % doesn’t mean any of those things, of course. The only certainty that I would take from the percentages is that a proportion of users who voted (12% in this case) *decided* that it didn’t work with their version of WordPress. But suppose the incompatibility is to do with a conflict with another plugin, rather than with WP itself? How thoroughly did the voter confirm the source of the problem before voting? – Comment By Ade On The Post – Are Your Plugins Compatible?

Making The Broken Button More Meaningful

One of the biggest complaints from plugin developers was that when a user would select broken, it wouldn’t trigger a box to provide feedback as to why they thought it was broken. Plugin developers found themselves with users reporting their plugin to be broken with no idea as to why they were arriving at that conclusion. Although I can’t find the specific time and date when it happened, at some point after that conversation, the behavior of the broken button changed so that if it was selected, a support topic creation screen would show up. It didn’t force users to submit feedback but if the plugin was considered broken to the user, the next logical step would be to create a support thread.

Plugin Broken Feedback

Four Years Later, Still A Lack Of Data

Out of curiosity, I decided to take a look at all of the popular plugins listed on the front page of the plugin repository thinking they would have the most data based on their popularity. It turns out, that was not the case. Most of the plugins had 50 reports or less. It’s important to note that with each plugin version bump, the compatibility data resets. I even reviewed the previous versions to view their data and for the most part, it was less than the most recent version. It’s my conclusion that few users are reporting working plugins let alone, broken ones.

What’s Happening With The Data And Where Do We Go From Here?

I got in touch with Otto as he works for Audrey.co and is part of the WordPress.org team to figure out what is going on with the data and if there are any plans to implement ways to get more user’s to participate in the reporting process. According to Otto, “the compatibility data is shown on plugin upgrades. If there is data there to display, then you’ll see a message like this: Compatibility with WordPress 3.6: XX% (Y “works” votes out of Z total). With some numbers to fill in those spaces, of course. I believe that it displays a different message if the plugin has been explicitly marked as tested-up-to the current version in the readme file, or “unknown” if there is no data to display. With the 2 year cutoff, this data has become less useful, since a lot of plugin authors are updating their plugins and keeping that information relevant.”

“Additionally, the original idea was that you’d mark plugin version X as compatible with WordPress version Y, but since we would prefer that all people run the latest WordPress version and keep things like plugins up to date too, then the whole thing starts to not make a lot of sense anymore. Obviously, the ideal is for “version” to just disappear, what with auto-upgrade and the like. Chrome has a version number too, but do you know what version your browser is? Or care?”

According to the author, this plugin was 100% compatible with WordPress 3.6.1

I then asked Otto if no one is providing enough data to make the feature useful, then what’s the point in having it?

Well, yes, if nobody uses that to provide the data, then the information isn’t useful. But in the same sense, this is like ratings. We need to find a way to make it useful, or ditch it, or rethink the whole thing. For ratings, we integrated them with reviews. So that in order to leave a rating, you have to leave a review. For compatibility, we have not done quite the same thing yet. However, if you select that a version is incompatible, then it forwards you to the support forum form where you can leave a support forum post about it. Since doing that a couple of years back, this has resulted in a slight increase in support threads for plugins, so in that sense I consider it useful. A simple “this doesn’t work” is not as handy as “this doesn’t work and here’s why” to a plugin author.

Has there been any talk regarding that feature perhaps to make it easier to report? Granted, once someone installs a plugin and it works, that’s the
last time they visit the plugin page unless they need support. Has there been any thoughts to possibly send a nag or notification to the user after a set amount of time to ask them if the plugin works or is broken? Although I could see that as being a huge annoyance.

I am not aware of any discussions along these lines at present. I cannot imagine that we’d want to add any form of “nag” like that to core. That’s a strong disincentive to this sort of thing.

On the whole, while these metrics are nice to look at, I’m unconvinced of their usefulness all by themselves. The compatibility feature is nice and shows some interesting data, but not particularly useful data. There is no incentive for people to click the “it works” button, and clicking the “it’s broken” button doesn’t provide useful feedback. I’d kinda prefer to somehow surface reviews and support threads at the time where you’re updating or reading about a plugin. I don’t know how doable this is at present though.

The case for “make it easier to leave feedback” is a problematic one, because it opens a backdoor that we really don’t want to have. See, currently you have to be logged in to WordPress.org in order to do these various things. This is so that we can prevent spamming of the stats, essentially. Some people have still tried, of course. But you don’t have to be logged in to WordPress.org, or even have a forums account, to see the data on your own install. So displaying something outside our site that they can click essentially means that we can’t have the login requirement anymore, since it could be anybody, and then you open that door for abuse.

Some consideration has been given to using OAuth for this, so that you can login and leave that feedback that way, securely, but this is fraught with problems, and anyway it’s a heck of a thing to do just for this sort of feedback.

At the end of the day, with all things considered I wanted Otto’s thoughts on that specific feature of the plugin repository.

In the long run, a redesign of the plugins/themes directories are in order, and in that process, maybe we’ll change things up, integrate things better, come up with more methods for incentivizing that sort of useful feedback. But that is ultimately a human psychology problem. Not my area of expertise. I’m a coder. I write code.

Last but not least, I asked Otto if he was able to see a timeline or history of compatibility data that’s been reported? Something like trends from 2009 when it was introduced to 2013? I was curious as to whether the amount of data received by users has stayed roughly the same for the past few years.

The data is not time stamped, so no, there’s no way to compare it over a time range.

My Thoughts

I’ve been reading posts about the feature when it was introduced in 2009 and thought the comments back then were interesting. At that time, it seemed like the biggest request was to have the ability to say whether a plugin works or is broken from the Installed Plugins screen. This way, it would be quick to mark all of the plugins that work without having to browse to the individual plugin pages to provide that feedback. We still don’t have anything like that in core. I wonder if a simple change like that would get more people to provide compatibility data? If I could mark a plugin as works from the installed plugins page, I’d do so.

Having people provide feedback after they say it’s broken was a huge step towards making the whole process useful for plugin authors. I agree with Otto in that a drop down box selecting a WordPress version is dumb at this point as all versions of WordPress “should be” or “assumed to be” the latest version and it’s just whether or not the plugin works or is broken. The WP version number becomes less of a factor when auto updates for core come around.

At this point, it looks like any future enhancements of the plugin compatibility feature will be rolled into a to-do list for the plugin repository redesign. It’s not something that is of utmost priority so that would make sense. However, I’d like to hear from both plugin authors, and users to find out what would make the plugin compatibility area more useful and if you had the chance, how would you get users to supply the information needed to determine whether certain versions work, or are broken?


14 responses to “What Good Is Plugin Compatibility Data If Users Are Not Participating?”

  1. In one of my more popular plugin’s settings page, I have a box called “Like this plugin?” as the title. Beneath it is a list with three links:

    * Give the plugin a good rating.
    * Donate a few dollars.
    * Get me something from my wish list.

    This was sort of an experiment I started a couple of years back to see what would happen with this type of thing. Guess what? This plugin has the most ratings and given me the most donations.

    I think you just have to make it really obvious/easy to users. Many users are never going to come back to a plugin’s page after initially installing the plugin. Basically, they have no reason to do so. But, if you give them a quick and easy way to do these things, that all changes.

    Obviously, this is just from personal experience with a single plugin, but I think it’s been an interesting trial-run thus far. I really need to write a blog post on this at some point.

  2. @Justin Tadlock – Interesting. I’d still like to be able to report whether a plugin is Broken or Works from the installed plugins screen. To me, that seems to be the most logical place to put that information. That way, I don’t have to browse back to the individual plugin page. Also, I understand most plugins have a settings page but to have one for the sake of getting me to give the plugin a good rating is not something I’d be happy with.

  3. I never look at the plugin compatibility section when downloading or updating plugins.

    When I get a plugin for the first time I always go by regular updates and popularity. Normally a high number of those two will mean the plugin still works well.

  4. @Martin – lol that’s another good point. In fact, I have rarely looked at that section as well although when I download/install the plugin for the first time, I do use the compatibility section as just another option in determining whether it would be a good fit or not. But if I install the plugin and it works fine, I don’t pay attention to the compatibility information after that

  5. I think it might help if someone deactivates a plugin that a box shows up asking why with ‘testing’ being one of them but if someone selects that the plugin doesn’t work it could then take the person to the plugin page so they have the opportunity to explain why they think it doesn’t work

  6. What about increasing the data density by eliminating the unnecessary minor WordPress version selections? There is almost never any reason that a Plugin would stop working from one minor version to the next, so why split up compatibility data between, e.g. 3.6 and 3.6.1?

    This would not only provide better data, but would also encourage more participation.

    Likewise: the “Tested Up To” tag should align with major versions. (In the rare case that a minor version causes a Plugin incompatibility, the “Tested Up To” (and “Requires”) tag could list the minor version.)

    I also think opt-in integration with the WP-Admin (even if via oAuth) is critical to widespread adoption. So much of the Theme/Plugin/core experience is built into the Admin, that I would imagine that most people never actually go to wordpress.org/themes or wordpress.org/plugins anymore. If true, I doubt that most people would do so just to hunt down all of their active Plugins, to rate or provide compatibility data.

    With oAuth connectivity to wordpress.org, any active Plugin could be reported as “compatible” (with Plugin version and WP version, since they’re already included in the update API), and it would probably be fairly trivial to hook into the Plugin deactivation, and add a “Report compatibility issue” checkbox that would log the Plugin as incompatible, with Plugin and WP version. And with oAuth, I would think that someone could write a support forum topic during the same process.

    (Ratings could use the same approach, as an added benefit.)

  7. This drifts a bit OT … but I have noticed across ‘recent times’, that my extensive plugin installation is very stable. I rarely have or see reports of any plugin-related issues. (It’s also true that I’m fairly savvy about what *not* to try to do, because it is obvious (to me) that it will invite trouble.)

    I am not as active now, adding & removing plugins as previously, but I remember well how dicey it was, to run a large collection of plugins, a few years back. That_is_history.

    And right there, you could be looking at an important factor in how people (don’t) participate in compatibility reporting. Nutin’ to report. No exasperation, to motivate them. Also, the ‘rating game’ really only rewards nice cheerleading; negatives are negative, and cool peeps don’t do that.

    Point #2. I am not interested in and don’t mess with certain important categories of plugins. These categories tend to be the ‘elite’, the ‘glamorous’, and the darlings of developers & small-business folk orbiting around WP these days.

    I’m not into a lot of the bikini-clad female across the hood of the showroom model action. And you know what? There is actually “evidence”, which bubbles up in public talk rather regularly, that that is where a lot of the issues arise.

    Not in the large category of un-classy and orphan plugins in the repository (liberally spicing my collection) which we see trotted out routinely as a fav whipping-boy. I’m a virtual ‘animal shelter’ for this stuff, yet problems have been vanishingly rare, for quite awhile now.

    It may be, that it’s the cool plugins & fancier developer-wares, that cause most problems … and the usership of this category of plugins will be especially & notably loath to say so.

  8. I still visit the repository when viewing plugins… can’t seem to view/sort by the same degree when browsing only via the ‘add new’ interface on the wp-admin (maybe I am doing something wrong?)

    Like others, I also go by ‘latest update’ and ‘frequency of update’ to determine whether or not a plugin is suitable for my purpose. I know that Tobias (of TablePress fame) does encourage people to give a rating/review in his correspondence, like Justin Tadlock said, and it seems to work quite well.

    Some niche plugins have not been updated in awhile yet they still work. I like the idea of ‘major update’ compatibility (as suggested by Chip) – it would certainly be nice to have a ‘It works!’ button after installing (or It doesn’t work! with a quick popup for users to explain why) for convenience.

  9. I have to agree whole-heartedly if there was an easy way to mark the plugin as working or broken right in the installed plugins screen I would do it without a second thought.

    The problem that I think most of us have, as you mentioned above is if it’s working as expected we tend to never go back and rate the plugin (or at least mark is a working/broken).

    A simple link in the Installed Plugins list right after or around “Visit Plugin Site” saying something like “Does it work? Yes | No” that would automatically integrate with the Plugin repo stats could work. I know this probably is a lot easier said then done, but I think this would be a good fix to help support the plugin developers (and even theme developers if there was similar integration) and make it easier for plugin users to know if it works.

    All that said though, I never consider the working/broken stats when finding a plugin just because half of the ones I look at don’t have enough data, and those that do have a 50/50 split. But, I am experienced with WordPress – what about the new users or non-tech savvy users who won’t try a plugin because they are afraid something will break that they can’t fix?

    I mean, looking this morning Jetpack and even bbPress don’t have enough data to say if it’s compatible with 3.61 and they are hugely popular and widely used. No one has come back to rate it since the update. At this moment this data is extraneous because of the lack of participation (users fault) and accuracy (lack of data). If we could get the community involved in an easier, simply integrated way then the value of that data would be what we are looking for now.

  10. I agree with Chip Bennett, most people are installing plugins from their WordPress admin area, so they are less likely to go to wp.org and report if a plugin is broken.

    How about the core includes an option for plugin authors to decide whether or not they would like users to give feedback upon uninstallation of the plugin. Plugins that participate in the program will display a feedback screen when a user deactivates their plugin.

    This screen can show rating stars, broken or worked form. If a user clicks on a star a popup window opens asking them to leave their review.

  11. I think, best way to address is, adding a new column for number of downloads for current version of plugin (obviously including updates). If that number is good enough, it will give some comfort to any user, if this version of plugin is in use or not. It gonna give a better idea of popularity of current state of plugin as well.

  12. As a plugin author, the most important thing is the incompatibility issue. I am more interested to why a user believes a plugin doesn’t work and a user who says a plugin is not compatible should be asked to raise a support query. I believe this should also be the case from 1 ratings since they are of no value to a plugin author.

    Also, having version numbers isn’t really needed. As a plugin author, I want the user to be using the latest version of the plugin and am not likely to support an old version. As a WordPress user, you should always be running the latest version of WordPress.

  13. I believe Justin’s comment opener is the closest thing to getting the best possible response: you have to invite response to reasonably be in a position to expect some. So, I’d think about a logical, visible spot to include a little sentence reminding the new plugin user to provide some qualitative response (i.e. posted, textual feedback) and (especially) imploring the user to also responsibly give feedback on compatibility, with a link to the spot where to select the WP and plugin versions plus the “works” selector.

    All this does is essentially drive up the number of compatibility votes. And I believe in the power of statistics; the greater the sample, the bigger the representative value. Put with a sideways smile, it’s much harder for a large number of trolls/ignorants/malcontents to be deliberately wrong, amid a much larger body of users generally painting the big picture of user experience.


Subscribe Via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

%d bloggers like this: