It’s a fear many WordPress users had in 2009 and it’s one that continues to be near the top in terms of reasons why people won’t upgrade their sites. It’s the fear that their plugins won’t be compatible with the new version of WordPress. Back in October of 2009, the WordPress core team tried to address these fears by adding a plugin compatibility section to the WordPress.org Plugin repository.
Developed by Michael Adams, the plugin compatibility section worked on the simple premise of selecting which version of WordPress was being used with which version of the plugin and clicking on either a broken or works button. The only people who could vote were those who had a WordPress.org user account. After voting, a bar was displayed showing two percentages. If there was no data to be shown, a yellow bar was displayed saying no data. A complaint heard from plugin developers at the time was that the percentages had no clear meaning.
For example, taking the Google XML Sitemaps plugin shown above, what does 88% really mean? That it *will* work on my setup? Or only 88% of the time? Or I have an 88% chance of it working? The % doesn’t mean any of those things, of course. The only certainty that I would take from the percentages is that a proportion of users who voted (12% in this case) *decided* that it didn’t work with their version of WordPress. But suppose the incompatibility is to do with a conflict with another plugin, rather than with WP itself? How thoroughly did the voter confirm the source of the problem before voting? – Comment By Ade On The Post – Are Your Plugins Compatible?
Making The Broken Button More Meaningful
One of the biggest complaints from plugin developers was that when a user would select broken, it wouldn’t trigger a box to provide feedback as to why they thought it was broken. Plugin developers found themselves with users reporting their plugin to be broken with no idea as to why they were arriving at that conclusion. Although I can’t find the specific time and date when it happened, at some point after that conversation, the behavior of the broken button changed so that if it was selected, a support topic creation screen would show up. It didn’t force users to submit feedback but if the plugin was considered broken to the user, the next logical step would be to create a support thread.
Four Years Later, Still A Lack Of Data
Out of curiosity, I decided to take a look at all of the popular plugins listed on the front page of the plugin repository thinking they would have the most data based on their popularity. It turns out, that was not the case. Most of the plugins had 50 reports or less. It’s important to note that with each plugin version bump, the compatibility data resets. I even reviewed the previous versions to view their data and for the most part, it was less than the most recent version. It’s my conclusion that few users are reporting working plugins let alone, broken ones.
What’s Happening With The Data And Where Do We Go From Here?
I got in touch with Otto as he works for Audrey.co and is part of the WordPress.org team to figure out what is going on with the data and if there are any plans to implement ways to get more user’s to participate in the reporting process. According to Otto, “the compatibility data is shown on plugin upgrades. If there is data there to display, then you’ll see a message like this: Compatibility with WordPress 3.6: XX% (Y “works” votes out of Z total). With some numbers to fill in those spaces, of course. I believe that it displays a different message if the plugin has been explicitly marked as tested-up-to the current version in the readme file, or “unknown” if there is no data to display. With the 2 year cutoff, this data has become less useful, since a lot of plugin authors are updating their plugins and keeping that information relevant.”
“Additionally, the original idea was that you’d mark plugin version X as compatible with WordPress version Y, but since we would prefer that all people run the latest WordPress version and keep things like plugins up to date too, then the whole thing starts to not make a lot of sense anymore. Obviously, the ideal is for “version” to just disappear, what with auto-upgrade and the like. Chrome has a version number too, but do you know what version your browser is? Or care?”
I then asked Otto if no one is providing enough data to make the feature useful, then what’s the point in having it?
Well, yes, if nobody uses that to provide the data, then the information isn’t useful. But in the same sense, this is like ratings. We need to find a way to make it useful, or ditch it, or rethink the whole thing. For ratings, we integrated them with reviews. So that in order to leave a rating, you have to leave a review. For compatibility, we have not done quite the same thing yet. However, if you select that a version is incompatible, then it forwards you to the support forum form where you can leave a support forum post about it. Since doing that a couple of years back, this has resulted in a slight increase in support threads for plugins, so in that sense I consider it useful. A simple “this doesn’t work” is not as handy as “this doesn’t work and here’s why” to a plugin author.
Has there been any talk regarding that feature perhaps to make it easier to report? Granted, once someone installs a plugin and it works, that’s the
last time they visit the plugin page unless they need support. Has there been any thoughts to possibly send a nag or notification to the user after a set amount of time to ask them if the plugin works or is broken? Although I could see that as being a huge annoyance.
I am not aware of any discussions along these lines at present. I cannot imagine that we’d want to add any form of “nag” like that to core. That’s a strong disincentive to this sort of thing.
On the whole, while these metrics are nice to look at, I’m unconvinced of their usefulness all by themselves. The compatibility feature is nice and shows some interesting data, but not particularly useful data. There is no incentive for people to click the “it works” button, and clicking the “it’s broken” button doesn’t provide useful feedback. I’d kinda prefer to somehow surface reviews and support threads at the time where you’re updating or reading about a plugin. I don’t know how doable this is at present though.
The case for “make it easier to leave feedback” is a problematic one, because it opens a backdoor that we really don’t want to have. See, currently you have to be logged in to WordPress.org in order to do these various things. This is so that we can prevent spamming of the stats, essentially. Some people have still tried, of course. But you don’t have to be logged in to WordPress.org, or even have a forums account, to see the data on your own install. So displaying something outside our site that they can click essentially means that we can’t have the login requirement anymore, since it could be anybody, and then you open that door for abuse.
Some consideration has been given to using OAuth for this, so that you can login and leave that feedback that way, securely, but this is fraught with problems, and anyway it’s a heck of a thing to do just for this sort of feedback.
At the end of the day, with all things considered I wanted Otto’s thoughts on that specific feature of the plugin repository.
In the long run, a redesign of the plugins/themes directories are in order, and in that process, maybe we’ll change things up, integrate things better, come up with more methods for incentivizing that sort of useful feedback. But that is ultimately a human psychology problem. Not my area of expertise. I’m a coder. I write code.
Last but not least, I asked Otto if he was able to see a timeline or history of compatibility data that’s been reported? Something like trends from 2009 when it was introduced to 2013? I was curious as to whether the amount of data received by users has stayed roughly the same for the past few years.
The data is not time stamped, so no, there’s no way to compare it over a time range.
My Thoughts
I’ve been reading posts about the feature when it was introduced in 2009 and thought the comments back then were interesting. At that time, it seemed like the biggest request was to have the ability to say whether a plugin works or is broken from the Installed Plugins screen. This way, it would be quick to mark all of the plugins that work without having to browse to the individual plugin pages to provide that feedback. We still don’t have anything like that in core. I wonder if a simple change like that would get more people to provide compatibility data? If I could mark a plugin as works from the installed plugins page, I’d do so.
Having people provide feedback after they say it’s broken was a huge step towards making the whole process useful for plugin authors. I agree with Otto in that a drop down box selecting a WordPress version is dumb at this point as all versions of WordPress “should be” or “assumed to be” the latest version and it’s just whether or not the plugin works or is broken. The WP version number becomes less of a factor when auto updates for core come around.
At this point, it looks like any future enhancements of the plugin compatibility feature will be rolled into a to-do list for the plugin repository redesign. It’s not something that is of utmost priority so that would make sense. However, I’d like to hear from both plugin authors, and users to find out what would make the plugin compatibility area more useful and if you had the chance, how would you get users to supply the information needed to determine whether certain versions work, or are broken?
In one of my more popular plugin’s settings page, I have a box called “Like this plugin?” as the title. Beneath it is a list with three links:
* Give the plugin a good rating.
* Donate a few dollars.
* Get me something from my wish list.
This was sort of an experiment I started a couple of years back to see what would happen with this type of thing. Guess what? This plugin has the most ratings and given me the most donations.
I think you just have to make it really obvious/easy to users. Many users are never going to come back to a plugin’s page after initially installing the plugin. Basically, they have no reason to do so. But, if you give them a quick and easy way to do these things, that all changes.
Obviously, this is just from personal experience with a single plugin, but I think it’s been an interesting trial-run thus far. I really need to write a blog post on this at some point.