Solving the Mystery of How People Actually Use WordPress

I’m in favor of WordPress collecting more anonymized usage data that could help make informed decisions on changes or improvements to core, such as tracking changes to the WordPress user interface, which buttons or settings are used most often, etc.

A good example of when this data could have come in handy is the recent removal of the justify and underline buttons from the editor in WordPress 4.7. During the discussion on whether they should be removed or not, a number of people questioned if there was any user data available that would indicate how much they’re used and help gauge the impact of removing them.

The only data available to help make an informed decision was provided by Mel Choyce. Choyce shared statistics from and its variety of editor interfaces that indicated Bold, Italic, and Links are used the most while Lists and Blockquotes are the second most used buttons.

The Center and Left alignment buttons are used often, but the data doesn’t determine if people are using them to align text or images. Information on which headings are used most was not available. The team did not have any usage data specific to the WordPress core editor.

In the ticket, Andrew Ozz, who maintains the TinyMCE component, chimed in and agreed that good user data is needed.

In an effort to obtain usage data before removing the buttons, Ozz created a small plugin to perform testing with five existing and first-time users. Interestingly, he discovered that both types of users clicked on the kitchen sink button to display the second row of buttons and didn’t click the button to hide them again.

Ozz also shared other results from his limited testing.

I know these test results are extremely limited and cannot be used when making a decision, but they are an indication of what ‘real’ testing may reveal. In this case it shows that moving buttons to the bottom row will have no effect on the usage of these buttons as they will still be visible at all times.

This super limited testing also indicated another (much bigger) problem: somebody mentioned this some time ago (think it was @mor10), around 20% of the WordPress users don’t even know there is a second editor toolbar, and some feel ‘pretty stupid’ after discovering it. I think this is bad UX and something that can be fixed easily by having the second toolbar open by default, and fixing it is more important and will improve the UX for these 20% of users a lot.

Imagine how useful it would be for core developers or others if there was usage data like this on a grander scale that could fuel rapid improvements and help discover and eliminate pain points.

Matt Mullenweg, co-creator of the WordPress project, has closed the ticket with the Telemetry Proposal as it’s not within the three project focus areas for 2017.

“There is no part of current or potential WP development that is being held back by the lack of this existing, as there are easy and current ways to answer questions with data to the extent it would inform our decisions,” Mullenweg said.

Morten Rand-Hendriksen responded to the closure saying that the quantitative user testing falls squarely within the Customizer focus area.

“I would argue since the release of the Customizer some years back, it has gone through a multi-year large-scale quantitative user test with incremental tweaks and improvements,” Rand-Hendriksen said.

“This is in line with standard agile development. At this juncture, the Customizer can be considered mature, and moving a mature solution forward requires hard data on usage, use cases, and user needs. This goes beyond standard user testing to large-scale data collection, which is what this ticket aims at addressing.”

Perspective From a WordPress Release Lead

There are WordPress core developers who have shown interest in a similar system. At the start of the WordPress 4.7 development cycle, Drew Jaynes, who led the WordPress 4.2 release cycle, expressed interest in creating an opt-in data collection system.

The idea received positive feedback that included people offering to help. I asked Jaynes what his thoughts are on such a system and how it could benefit core development.

“There’s some discussion about what form that collection should take initially, but I think there’s consensus that it should be opt-in, and take one of two forms (or a hybrid of the two): active (surveys in the admin) or passive (anonymized usage) data collection,” Jaynes said.

“Either way, I think having this data available would benefit the entire community, regardless of the obvious practicable application within core development.

“All of that data can and should be used to inform decision-making in WordPress going forward. The core team really needs to hit the reset button on the concept of the 80/20 rule, including what and whom it represents.

“We should be building modern WordPress for the modern WordPress user, and resting on Matt’s instincts coupled with the core team’s experience is no longer enough to maintain positive forward momentum.”

Jaynes cites the editor as an example of where having the data would be helpful and that without it, pursuing an idealized ‘modern editor’ in WordPress is premature. The data could also help provide insight into improving the new user experience.

“A common complaint is that the WordPress admin can be really overwhelming to new users,” Jaynes said. “Having real data about how frequently the various core screens are used could really inform decisions about maybe paring it down, or hiding some things over time that are used less and less.”

While collecting data could help with making informed decisions, he doesn’t think it should stop the core team from experimentation.

“I think having real, citable data could really reduce the amount of backlash we’ve seen with a few releases in the last couple of years,” Jaynes said. “Areas where core team decisions left some group of users feeling jilted.”

“It’s worth mentioning that there’s absolutely value in allowing the core team to experiment, as long as we’re careful not to latch onto something that got merged as the only way we’ll ever need to solve that problem; that’s where we get into trouble.”

Who Are The 80/20 Users of WordPress?

The most striking statement in Rand-Hendriksen’s proposal is that WordPress development is occurring without having any idea who the 80% or 20% of users are.

“During the development of WordPress 4.7, I was involved in several conversations centered around assumed use of features,” Rand-Hendriksen said.

“The general argument was that based on the 80/20 rule, certain features should be added while others should be removed. I kept bringing up the well-known fact we don’t have a clue what features 80%, or even 20%, of WordPress users actually use so any claim of validity in the 80/20 rule is guesswork at best.”

Collecting usage data is standard practice. Microsoft Windows, Mozilla Firefox, Chrome, iOS, and a number of other software projects have opt-in data collection systems that are used to improve the product. They also provide insight into how customers are using their products.

WordPress development on the other hand relies on the support forums, data collected from, limited user testing, verbal feedback at WordCamps, and other small data points. Collecting usage data from WordPress could show trends and provide evidence for changes related to the decisions not options philosophy of WordPress development.

Collecting usage data isn’t going to solve all of WordPress’ woes but having it available to make more informed decisions is better than not having any data at all. Although an opt-in data collection system in WordPress won’t be a core focus any time soon, it’s encouraging to see the idea has merit and is something some core developers are interested in seeing become a reality.

I’d gladly opt-in and share my usage data with as long as it was anonymized and displayed publicly in aggregate. Would you?

[poll id=”60″]


41 responses to “Solving the Mystery of How People Actually Use WordPress”

  1. Hello,

    I completely support the collecting usage data idea, this is something that should have been added a long time ago in my opinion, and it is something that would be an improvement and it would help to give a voice to users like myself who are often on the bad end of things when features/themes/options/widgets/et cetera that I like and rely on get removed and/or messed up et cetera.

    -John Jr

  2. Because I value privacy, I’ve always opted out of data collection. However, I also care about the future of WordPress, so recently I’ve changed my mind, for core at least. I think it’s very important for core developers to begin making decisions based on data.

  3. Not exactly disagreeing, but leaders decide for the herd, not follow where the herd wants to go. Imagine steve jobs asking at the point of time how to design the ipod? The answer he would have got would have lead him to put physical buttons on it (because everybody at that time from inertia where mimicking the walkmans of the world).

    You don’t need user feedback to know that “kitchen sink” is bad UX as it is right there in the text book for UX 1.0.1. Quoting “never hide functionality from the user that is only revealed when he clicks or hovers magic areas”. You don’t need feedback to know that left and right alignment are wrong and their usage is the result of users not understanding how themes work.

    You don’t need feedback to know that users do not want the rest-api, xml-rpc, oembed, pingbacks and many other features. If wordpress actually was partially following the 80/20 rule it would not have been as bloated as it is now, but there is actually no interest by core developers to follow such a rule, as it will be too limiting for them and will prevent them from adding there toy projects to core.

    • You don’t need feedback to know that users do not want the rest-api, xml-rpc, oembed, pingbacks and many other features.

      Um, I personally know people who want exactly those features. Those are all useful features for a lot of people.

      This reinforces the requirement for actual data, instead of just a few people saying things they don’t like.

      • @guarav,

        sure some people want it, and they are useful features, but the amount is far from even being 20%.

        Lets say something really useful like xml-rpc. Probably a minority of people use it use the wordpress apps on their phones to publish content, this leave jetpack as the biggest user of the feature, and jetpack is installed on less than 5M sites while there are about 70M wordpress sites. These stats come from the bad statistics man so they are probably wrong in some way or another, but assuming even just 40M actual different installs, a feature will need to be useful for 32M of them.

        Probably no feature, except for the very basic ones, can reach the 80% mark, but even if you draw the line lower you will find that there are a lot of “rarely used” code in core, and you need to draw the line if you want to do the “decisions, not options”.

    • To be clear: This is not about user feedback, it’s about collecting hard user data. We are not asking people what they think something should look like or whether one thing is better than another. Instead we are gathering passive data on how they actually use the current solutions. This cuts through the noise of “in my opinion user interfaces with rounded corners are easier to use” to find things like “nobody ever uses this feature we spent 6 months developing.” User opinionating about features is done in the qualitative user testing stage. This is what we do now. What we lack is quantitative user testing to show how people interact with the application.

      • @Morten, it is a feedback, just not titled with such a “checkbox”.

        As I said, I am not against it, but this is going to end up as a self selecting group since you will need to opt in. Any developer that will work for a client will not do that as it is not his site and he has no reason to waste time talking about it with the client, so here goes monitoring of all SMB and big shops sites, and you might be left with just blogs. And than most people that will agree to opt-in have probably a more technical background and understan the importance of such monitoring.
        So you might end with monitoring technologically oriented blogs, which are obviously not a statistically representative of the wordpress install base.

        Any data is better than no data, at least you will be able to spot areas of huge success and failures, but this will nor provide a good answer to the question “what do users need?” as users usually do not know what will be good for them, and you will never know if a feature is not used because it is not useful or because it has no good exposure in UI/documentation.

        You can not replace knowledge based decisions with decisions based on popularity.

  4. I agree. Because WP needs to be better. I assume there is a lot of feedback already happening. Not sure if this feedback is actioned, but there are some fundamental issues with WP that have been there as long as I know. Such as a proper image gallery without resorting to a plugin.

    A much better text editor. I assume coding issues or something like that is what restricts developments to the text editor, but whatever it is, it needs an overhaul.

    WP has advanced to the point where it should be thought of as a website builder, not a blog builder. It’s much more than that. This would change the mindset of future developments I believe.

  5. This might be useful to the core team. The feeling I get when checking out the Trac from time to time is that the core team is mostly made up of hardcore bloggers and that they are unable to grasp the diversity of sites people are building with WordPress.

  6. It starts by anonymous user data, continues by collecting some basic content such as the site’s name, URL, post titles. Then it goes on to just noting actual content within the sites and categorizing websites based on their content. Next we move to match emails of WP admins that are involved in multiple sites, target them with the new hosting campaign, or selling shares of automattic directly to facebook, you know…

  7. I think there’s no debate that data driven decisions are better than wild guesses.

    I can understand why this might not be a priority due to the current road map, but I’d like to know what are the current tools available for data driven decisions.

    In the Trac ticket Matt mentions that “there are easy and current ways to answer questions with data to the extent it would inform our decisions”.

    If data is available it would be good to open it up to the community :-)

    Btw, open data doesn’t impact development resources as it would just be a dump and people could just take a stab at it, I know I would have lots of fun with it :-)

    • open data doesn’t impact development resources as it would just be a dump and people could just take a stab at it

      This isn’t really true. Anonymizing, cleaning, and collecting data is a huge amount of effort at this scale which is why it would be a distraction from the main focuses.

      Dumping data indiscriminately is a good way to accidentally release private information.

      like to know what are the current tools available for data driven decisions has a lot Calypso instrumented with event tracking (called “Tracks”). See recordTracksEvent on github. If there are things that are not instrumented, they are just a PR away from being added.

      It’s not perfect because it is not wp-admin, but for the main focuses it is a pretty good proxy and has data for millions of users which is not something any new telemetry system would be likely to have for years (if ever given the differences of opt-in vs not).

      Analyzing the data is also not zero effort, and I tend to find that having a hypothesis about what you are trying to learn from the data is a good prerequisite to actually going to look at the data.

      Data tracking can also only take us so far and (as some others have noted) we can’t let it completely drive all decisions. It’s a feedback mechanism, but it’s not the only one. More user interviews for instance would be good to see.

      • Hi Greg,

        I have to say that I find it ironic that we are here debating about Open Data in an Open Source project :-)

        Agree with you, sure there’s some work involved in order to prepare/clean/anonymize the data, but it’s a one-time investment so that other external resources (ie non-devs) can help out in analyzing the data. I think we are really looking at different profiles in terms of skill-set. (Btw, I did not say we should dump data indiscriminately. I simply meant that data dumps once established can be automated, unlike development of new features in core.)

        Concerning the second part of your answer, thanks for the info about Calypso. So I guess that some people use data after all :-)

        Totally agree that analyzing data is not a zero effort, but again we are looking at different skill sets. I wouldn’t dare playing with core, but I have no problems with data. I guess the same is true for many other people in the community.

        Agree with you about user feedback and using other methods, and I don’t see any contradiction between those methods and Open Data, they are not mutually exclusive (assuming the right anonymity is in place etc etc).

        • Hi Luca,

          There’s sort of two threads here:

          1) A one time dump of data. I think this is pretty hard to do without some hypothesis of what the data is to be analyzed for. There is lots of data, but it is hard to know how to clean it without knowing what it will be used for. Right now most of the discussion is just generically about “open data”. If there are requests for specific data, I’m happy to engage in those discussions with the data I have access to. I would bet that all Automatticians feel the same way.

          2) Building an automated system. This is a big project and won’t be in place in time to get any useful data for the current focuses. As a comparison, Automattic had a dedicated team of 4-6 developers who spent 1-2 years on our tracking system. It continues to need multiple people to maintain it and uses a lot of servers.

          Even for the simpler system that I floated in October, it would be a lot of work, and take quite a while (years?) to have a big impact. Assuming we can’t make decisions without such a system seems like a distraction from building a better editing and customization experience.

        • Hi Greg,

          I understand your points 1 and 2 and I am aware of the workload for a full scale project, we are not at the solution phase yet, still debating whether there’s a need for data.

          When it comes to the telemetry proposal from Morten, I think he’s simply trying to say: let’s get some data cause today we have none or very little regarding installations.

          As I mentioned, I can understand this might not be a priority considering the current roadmap, but as the famous philosopher Francis Bacon quote goes “Knowledge is power”.

          Data is the raw state of information which can then be turned to knowledge.

          I did like my philosophy classes back at high school… ;-)

          • User data is the voice of WordPress. I agree “knowledge is power,” it is the one resource most simple to gather. Just ask the user and they will tell you in their own voice. Then armed with this knowledge you have the “power” or information to make the changes or move in a certain direction.

            So much discussion about something so simple. At least this is how it appears to me, a user of WordPress.

  8. All telemetry will tell you is which portions of the system or features within the system are tailing off in usage. That’s not a bad thing but don’t overthink this as some magic answer that will set the direction for WP. It’s more or less simple housekeeping.

    Also it brings the danger of killing off a great idea that was poorly implemented, since the usage data will tell you it’s not resonating with the user base.

    • In the original proposal, that is a requirement. In fact it goes further: The telemetry feature is disabled by default and not installed on the site. If the admin chooses to take part, a plugin is installed that can be removed at any time. This ensures full transparency and active, informed participation.

  9. If we don’t move forward with anonymous data collection in core, couldn’t we still create an anonymous data collection plugin? This could be announced via channels such as WP Tavern and Post Status, and any individual could choose to download and use it–or not.

    While this approach wouldn’t collect all data from everyone, it would still collect some data from some people, which is more than nothing.


  10. User data is vital to the forward movement of any product. As a WordPress user since 2006, I would willingly participate in sharing my experience using WordPress if it would help WordPress core. I would not have to do it anonymously, it would be part of giving back to WordPress.

    I’m not a developer, but recently started helping people with their WordPress website setup. I will go the extra mile to show them the admin functions and do what I call “good housekeeping,” and basic settings.

    I’ve attempted to write video tutorials and go back and forth on this, but the main problem users have is just not knowing how the basic functions work. They get it real quick once it is explained, because WordPress is built with the user in mind.

    • I totally agree with you. The core problem is that many users need to understand where within WordPress they affect the look of their site. I’ve had frustrated users that had a theme which front-page was completely composed of widgets and they had no idea where to start editing.
      Same goes for attachment driven galleries – rearranging was impossible without a plugin. Even the simple difference between a post an a page can be a mistery to some.

      However I think a proper usbility-test can not just be skipped if you have anonymous user data. There is still now way to tell, what the users intention was when he was using wordpress.

  11. No, Nein, Ne, and so forth.

    I don’t like third-parties collecting information about me. I don’t use Akismet or JetPack for this exact reason.

    I want to know EXACTLY what someone is going to collect about my website. How often will it be collected. Who will have access to that information. How do I know extra information will not be collected without asking me?. I want justification for EVERYTHING that will be collected.

    I do the same on my Android, why are you asking for my GPS location for a tetris, pacman or non-multi-player games/apps.

    How can I be sure the IPs of commenters and authors on my sites will not be collected? Their e-mail addresses? etc..

    I don’t like third-party collecting any information from my sites, even though they pinky-swear that they won’t collect personal information.

  12. privacy first. Which is why I love parts of this idea.

    seeing this “disabled by default and not installed on the site. If the admin chooses to take part, a plugin is installed that can be removed at any time”
    come from someone who is involved in wp made me almost shed a tear. That’s exactly what I’ve been saying about every new ‘feature’ since about 2.5

    I’d also add in to this optional, off by default idea, that it’s also optional what gets shared, and clearly shown what is shared.. and that consideration about other users of said install not giving permission could be affected, (mutlti-author / user blogs) and if visitors to said web site could be affected at all – even comment spammers have privacy rights in some parts of the world I guess – so make it clear, and maybe make parts optional – like only stuff done in admin area, by admin..

    The reason I would share this data – to show that just about every hour spent developing features since 2.5 has been for the very few – a complete waste that is not only ignored by the masses, but does nothing but slow it down and add security problems.

    I’d love the stats to show how many of us are actively using rest api, gravatar, embeds, and trackbacks, xmlrpc..

    I try to keep up with wp – and I run a bunch of ‘turn these things off plugins’ – and didn’t know until last week, that if you make a private page, password protected, with 100 links to your enemies’ web sites, and publish that password protected just for your evil twin to read.. it pings and tries to embed all your enemies and tells them where you are on the web.. ugh!

    so a plugin to disable embeds.. try it again.. nope – still sucking down server resources and crashes browser.. looks like it’s still pinging..
    more research.. oh! a plugin to silent publish..

    silent publish plugin! woo hoo! wait, looks like that breaks the password protection feature for some reason.. unpublish. trash. empty trash.

    a plugin needed to publish silently.. even though it’s marked private, password protected, and settings in discussion to block search spiders is checked and the rpc pingomatic is removed..

    maybe the user stats sent back would convince the team to release wp basic? remove all the stuff that should be plugins – I mean it’s nice to have the hooks you’d need to run rest api in core.. but most of us will never use it.

    maybe user stats would get gravatar taken out? emoji crap removed? These things could be plugins, not core :))

    now I install more plugins to block wp functionality with a new install than I add plugins to add features..

    disable g fonts, disable xmlrpc, disable trackbacks, disable embeds, disable gravatar (which doesn’t work as well as I’d like) – still need..

    anyway, I would share the data on several sites to add to the knowledge that the justify and underline buttons are used more than 95% of the features added since 2.5 ;)

    oh, and the stupid video that loads when you update to new versions – can we please.. I mean usage stats should show – I’ve updated 20 sites and not watched the vid once.. multiply that by the thousands of other updates that never watch it.. and can we please stop with that!

    I mean, add a graphic with a play button to show there is a cool video to watch – but pre loading; wasteful! It crashed my browser three times the past week.. it’s partially too many tabs in firefox and browser plugins – but geeze! something else wasting resources, and sending data to third parties that I’d prefer it didn’t – such a waste..

    Especially since it appears that that data is not being used to make a good decision about whether to keep this awesome feature that took so much time and creativity – I mean everyone should be force fed this 4.7 video!

    I saw @Gaurav says he knows users who are using rest, embed – etc. I would posit that users who use that kind of thing have no problem installing a plugin to add those functions – I mean if you are using rest you probably know how to add a plugin.

    Of course it does not go the other way –
    – most users that are forced to give up privacy and resources to ggl fonts, gravatar, and the places they link to in private posts, are generally not aware that these things are happening, and are likely not aware that they can add a plugin to remove them.

    Any chance this data collection would get akismet and hello dolly removed as well? perhaps show 20% of admins visit the editor / footer.php and realize there is no way to get to the sub folders of themes where the powered by stuff is hidden these days.

    There is no good way to know what the users don’t know.. but maybe less core features and more plugins..

    the buddypress crew seems to do well with adding hooks and stuff for plugins, although I’d like to see more features that users scream for there, at least they are not adding tons of third party stuff into the core of it every few months.

    hooks for plugins to do things is nice.. plugins forced in that are never used – that let your site get hacked next month – not the best thing.

    removing the ability of the old theme options to work.. I wonder if the anonymous stats would show how much time was spend going from screen to screen in the back end – looking for how in the heck we added those cusotm css things back in the day.. to change. them.. and could not find them.. then back and forth time and time again.. to not find what we were looking for.

    We know it was there – we can see the custom code on the site.. no note saying “hey this was taken away because you upgraded wordpress” – no clue at all.

    those kind of stats I’d gladly share.

    I’d prefer no embeds and pings to sites – but if you all think it’s really that awesome – then it would be easy to have wp pop up when it detects that you posted something with a link in – pop up and say should we ping that site and check for cool embed code? Then ‘your post would look good and pull the data like this – showing it – and slaying your privacy and that of your visitors any time the page is viewed.. but it’s ready.. and you can’t align it left.. that’s not the way themes work.. and if they change their page next year, you could have some stuff on your page that instead says ‘hacked by WJEKRE’ – but that’s because you are embedding from a site that has no ability to update plugins automatically..

    Or maybe the site is unusual and setup to auto update plugins, but they have one that’s no longer in our repo – even though 10,000 people are still using it, cuz they liked how things were in those ancient times of 2010.

    yep, embeds, love them!

    my usage stats may show that – and that would not be cool, because I don’t check for that when I write a post – I’d be horrified if I saw an embed when I published. What’s worse is all those people that don’t know how it works (the privacy / lack of control) – although you might see people hunting for ways to style the embeds.. sees embedded content, looks for styling options, clicks kitchen sink.. hovers…

  13. Once telemetry is in place and functioning for a few years, would it upset the development community to find that the create post feature is used 90%+ of the time, and that most just publish without drafting even once?

    The real question is what you’d do with that data. How would you be able to make decisions with the results?


Subscribe Via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

%d bloggers like this: