I’m in favor of WordPress collecting more anonymized usage data that could help make informed decisions on changes or improvements to core, such as tracking changes to the WordPress user interface, which buttons or settings are used most often, etc.
A good example of when this data could have come in handy is the recent removal of the justify and underline buttons from the editor in WordPress 4.7. During the discussion on whether they should be removed or not, a number of people questioned if there was any user data available that would indicate how much they’re used and help gauge the impact of removing them.
The only data available to help make an informed decision was provided by Mel Choyce. Choyce shared statistics from WordPress.com and its variety of editor interfaces that indicated Bold, Italic, and Links are used the most while Lists and Blockquotes are the second most used buttons.
The Center and Left alignment buttons are used often, but the data doesn’t determine if people are using them to align text or images. Information on which headings are used most was not available. The team did not have any usage data specific to the WordPress core editor.
In the ticket, Andrew Ozz, who maintains the TinyMCE component, chimed in and agreed that good user data is needed.
In an effort to obtain usage data before removing the buttons, Ozz created a small plugin to perform testing with five existing and first-time users. Interestingly, he discovered that both types of users clicked on the kitchen sink button to display the second row of buttons and didn’t click the button to hide them again.
Ozz also shared other results from his limited testing.
I know these test results are extremely limited and cannot be used when making a decision, but they are an indication of what ‘real’ testing may reveal. In this case it shows that moving buttons to the bottom row will have no effect on the usage of these buttons as they will still be visible at all times.
This super limited testing also indicated another (much bigger) problem: somebody mentioned this some time ago (think it was @mor10), around 20% of the WordPress users don’t even know there is a second editor toolbar, and some feel ‘pretty stupid’ after discovering it. I think this is bad UX and something that can be fixed easily by having the second toolbar open by default, and fixing it is more important and will improve the UX for these 20% of users a lot.
Imagine how useful it would be for core developers or others if there was usage data like this on a grander scale that could fuel rapid improvements and help discover and eliminate pain points.
Matt Mullenweg, co-creator of the WordPress project, has closed the ticket with the Telemetry Proposal as it’s not within the three project focus areas for 2017.
“There is no part of current or potential WP development that is being held back by the lack of this existing, as there are easy and current ways to answer questions with data to the extent it would inform our decisions,” Mullenweg said.
Morten Rand-Hendriksen responded to the closure saying that the quantitative user testing falls squarely within the Customizer focus area.
“I would argue since the release of the Customizer some years back, it has gone through a multi-year large-scale quantitative user test with incremental tweaks and improvements,” Rand-Hendriksen said.
“This is in line with standard agile development. At this juncture, the Customizer can be considered mature, and moving a mature solution forward requires hard data on usage, use cases, and user needs. This goes beyond standard user testing to large-scale data collection, which is what this ticket aims at addressing.”
Perspective From a WordPress Release Lead
There are WordPress core developers who have shown interest in a similar system. At the start of the WordPress 4.7 development cycle, Drew Jaynes, who led the WordPress 4.2 release cycle, expressed interest in creating an opt-in data collection system.
The idea received positive feedback that included people offering to help. I asked Jaynes what his thoughts are on such a system and how it could benefit core development.
“There’s some discussion about what form that collection should take initially, but I think there’s consensus that it should be opt-in, and take one of two forms (or a hybrid of the two): active (surveys in the admin) or passive (anonymized usage) data collection,” Jaynes said.
“Either way, I think having this data available would benefit the entire community, regardless of the obvious practicable application within core development.
“All of that data can and should be used to inform decision-making in WordPress going forward. The core team really needs to hit the reset button on the concept of the 80/20 rule, including what and whom it represents.
“We should be building modern WordPress for the modern WordPress user, and resting on Matt’s instincts coupled with the core team’s experience is no longer enough to maintain positive forward momentum.”
Jaynes cites the editor as an example of where having the data would be helpful and that without it, pursuing an idealized ‘modern editor’ in WordPress is premature. The data could also help provide insight into improving the new user experience.
“A common complaint is that the WordPress admin can be really overwhelming to new users,” Jaynes said. “Having real data about how frequently the various core screens are used could really inform decisions about maybe paring it down, or hiding some things over time that are used less and less.”
While collecting data could help with making informed decisions, he doesn’t think it should stop the core team from experimentation.
“I think having real, citable data could really reduce the amount of backlash we’ve seen with a few releases in the last couple of years,” Jaynes said. “Areas where core team decisions left some group of users feeling jilted.”
“It’s worth mentioning that there’s absolutely value in allowing the core team to experiment, as long as we’re careful not to latch onto something that got merged as the only way we’ll ever need to solve that problem; that’s where we get into trouble.”
Who Are The 80/20 Users of WordPress?
The most striking statement in Rand-Hendriksen’s proposal is that WordPress development is occurring without having any idea who the 80% or 20% of users are.
“During the development of WordPress 4.7, I was involved in several conversations centered around assumed use of features,” Rand-Hendriksen said.
“The general argument was that based on the 80/20 rule, certain features should be added while others should be removed. I kept bringing up the well-known fact we don’t have a clue what features 80%, or even 20%, of WordPress users actually use so any claim of validity in the 80/20 rule is guesswork at best.”
Collecting usage data is standard practice. Microsoft Windows, Mozilla Firefox, Chrome, iOS, and a number of other software projects have opt-in data collection systems that are used to improve the product. They also provide insight into how customers are using their products.
WordPress development on the other hand relies on the support forums, data collected from WordPress.com, limited user testing, verbal feedback at WordCamps, and other small data points. Collecting usage data from WordPress could show trends and provide evidence for changes related to the decisions not options philosophy of WordPress development.
Collecting usage data isn’t going to solve all of WordPress’ woes but having it available to make more informed decisions is better than not having any data at all. Although an opt-in data collection system in WordPress won’t be a core focus any time soon, it’s encouraging to see the idea has merit and is something some core developers are interested in seeing become a reality.
I’d gladly opt-in and share my usage data with WordPress.org as long as it was anonymized and displayed publicly in aggregate. Would you?
[poll id=”60″]
Hello,
I completely support the collecting usage data idea, this is something that should have been added a long time ago in my opinion, and it is something that would be an improvement and it would help to give a voice to users like myself who are often on the bad end of things when features/themes/options/widgets/et cetera that I like and rely on get removed and/or messed up et cetera.
-John Jr