If you’ve ever wondered how the WordPress Theme Directory identifies and ranks the themes that display under the popular tab, you might be surprised to learn that it has nothing to do with ratings. Popularity is a somewhat subjective quality to measure in the first place, but WordPress.org has an algorithm in place to give users an idea of which themes are trending.
The number of sites using a theme is the first metric that comes to mind for popularity, but it cannot be limited to that or else the results might be skewed towards older themes that are not in fact popular anymore. Samuel “Otto” Wood, in response to a Twitter thread about how themes appear to be gaming the system, created a video to explain how the popularity algorithm works.
“Popular is currently active installs divided by age of the theme and a few other factors,” Wood said. “Ratings don’t currently factor into it. We’re trying to work out an algorithm to add ratings to it. For the most part it’s a popularity thing.”
I asked him for more clarification on the other factors and he said it varies depending on what data they have and how that data looks over time. “We adjust it to have it change fairly regularly, but still reflecting what is popular at the time,” Wood said.
Dion Hulse, who also helps maintain WordPress.org infrastructure, said, “It’s not published by design, to hopefully prevent authors gaming it too much. Other than what Otto has said, they only other public part of it is that a theme must be at least two weeks old to be included in the popular rankings.”
At the end of last year, theme download counts were replaced with the number of active installs to try to prevent theme authors from gaming the system in order to appear on the Popular tab. Gaming the system is much more difficult now but there are other ways to do it, due to the imperfect measurements for active installs.
How WordPress.org Measures Active Installs for Themes
The topic of active installs was re-ignited yesterday by Matt Medeiros, a small business owner and WordPress.org theme author, who suspected the author of a recently popular theme of gaming the system.
Gaming .org theme downloads/active is running rampant. This theme “already has 10k active installs.” DL count roughly 3k @Otto42 pic.twitter.com/aaYwWcsvu8
— Matt Medeiros🎙 (@mattmedeiros) October 25, 2016
The Vertex theme, new to the directory this month, already has more than 10,000 active installs, despite having a much lower download count and no ratings. Medeiros made a video explaining what he perceived as suspicious activity propelling it to the fourth most popular spot on WordPress.org. This prompted Samuel Wood to film a response, which breaks down some of the flaws in the system that measures active installs.
“The active install count is a count of sites that have reported to us that they are checking for an update for that theme and that it is active,” Wood said. He explained that the 10,000+ in this instance and in many others is a rough estimate because WordPress.org does not have an exact measurement of how many sites have installed a theme:
What we are actually counting is yes, the number of active installs of a theme named Vertex. That may include themes not in our directory, such as this one from Elegant themes. If the theme isn’t in our directory we still get reports about it. The only way for us to not do that is to have the theme itself that theme check them for updates instead of checking us for updates, which their theme should indeed do. I would go so far as to say that any commercial theme should indeed be checking their commercial site, their systems not ours, but if they don’t have any special code to handle that case, then yes it will report back to our API server looking for updates. And if it has the name Vertex then yes, it’s going to be counted as being the Vertex theme.
In this instance, the creators of the Vertex theme were made aware that the name was already in use outside of WordPress.org. They offered to change the name of the theme to iVertex during the review process, but the Theme Review Team decided that it wasn’t necessary.
“There’s no gaming going on behind that,” Wood said. “If somebody does game the system, I will find it. I guarantee you. However, this is not such a case. Our current algorithm looks at active installs and if you happen to pick a name that is very popular [outside of WordPress.org], you can have a large number of active installs without it actually being this theme that is being counted.”
This problem could be solved with a unique ID for themes. Wood said there is a six-year-old core ticket that he would like to implement, but it’s not currently a priority.
“Unfortunately, there is no real push to do that,” he said. “At some point in the future, I would like all themes and plugins in the WordPress directory to get a unique ID which they can put in their headers and when they report back, having that unique ID will uniquely identify that theme or plugin. It would eliminate so many problems for me. It would make updates so much easier. I would be able to do a whole lot of useful things with it and it would let me fix this problem right here.”
How the Popular Themes Algorithm Affects Theme Businesses
Getting unique IDs in place is no easy fix, as theme reporting would need to be overhauled and multiple systems that interact with each other would need to be changed. It would be worth it in the long run for more accurate reporting on WordPress.org, which is the first place many users browse when looking for a new theme. If the active install counts are not a proper representation of installations via WordPress.org, then it doesn’t make sense to pin the popular algorithm to that number.
I asked Matt Medeiros why he chose to highlight the situation with Vertex, and he explained why many small business owners have a strong interest in the popular themes page.
“It’s getting harder and harder to survive as a theme shop these days,” Medeiros said. “Indie authors are overshadowed by huge theme shops and mega marketplaces. For me, .org is the only source of distribution for my themes, and I have a desire to see the repo become the defacto place to find quality themes. Sadly, not all themes are submitted to ‘do it right’ and many crop up to make a quick cash grab from unsuspecting end-users. Top spots are worth A LOT of money (see: Zerif) and coupled with an aging search engine for themes, gaming to the top is worth it, even in the short term. Unsuspecting users, who find searching too challenging, settle with what’s in front of them.”
Medeiros referenced Zerif Lite, which was recently removed from WordPress.org, due to violations of guidelines that began to be enforced after the theme was approved. The theme was pulling in tens of thousands of dollars for ThemeIsle, thanks to its frequent spot among the most popular themes on WordPress.org. Ionut Neagu, the company’s CEO, estimates that Zerif Lite’s suspension from the directory will diminish the company’s $70K/month revenue by 50%.
“There’s a lingering distaste in the review process for freemium upsells,” Medeiros said. “Not everyone agrees that folks should have an upsell product here, and that’s unfortunate. As a small business owner, that revenue helps me reinvest into the WordPress community, WordCamps, hire and train people in developing with WordPress. At times the review process, mixed with the emotions of others, feels like a rising tax against the small software business owner, like myself.”
When drawing attention to the suspicious active install numbers for the Vertex theme and the flaws in the algorithm, Medeiros drew resistance from Wood, who does not see the directory as a place where businesses should expect a return for their investments.
“The theme directory is not an advertisement to peddle your wares,” Wood said. “It is exactly that sort of thinking [that] makes me want to ban all themes with a ‘paid’ version from the directory entirely. The theme directory is not an advertising means. That is not its purpose. If that is your intent, then you are doing it wrong.”
Medeiros said he isn’t interested in reviewing all themes for possible fraud, nor does he expect Otto to do the same. Vertex is just one example among many where the flaws in the algorithm are on public display.
“I chose to highlight this issue because I think the repo can be a better choice for users than a for-profit marketplace,” Medeiros said. “Either way, I can only hope unearthing this conversation helps affect change for the better.”
I think it’s long past time some type of Steering Committee was structured around .org. The statement made by Mr. Wood is frustrating, because I’m not sure why these are his decisions to make. In practice, I understand they fall to the person doing the work and the legacy of our Benevolent Dictator, but given the size of the community and the impact of these decisions, surely it’s time for a new model.
The fact that so many businesses have been able to monetize WordPress, including Automattic, play a huge rule in it’s success and popularity, and it’s past time to stop the freemium/money bad mentality.