Call for Feedback on Theme Review Action, a New Automated WordPress Theme Testing Project

Automation. It is one of those dreams in the minds of many reviewers from the Themes Team. If there was a tool to take care of 90% of the issues, the team could focus on the 10% not easily found by automated scripts.

Enter the Theme Review Action project. Steve Dufresne, a WordPress Meta team contributor, put out a call for testing and feedback of the new project on Monday.

“If we could combine some of the existing code analysis tools, automate away some of the manual testing and open them up to more development workflows, could we improve theme quality, alleviate pressure on manual testing, and speed up the theme review process?” asked Dufresne.

The project currently runs several test suites, including the current Theme Check plugin. Theme authors can run the texts by running the NPX command in their theme folder, adding it as an action on GitHub, or cloning and running it locally. Running via NPX is not currently supported on Windows.

Right now, theme authors are needed. Regardless of whether you are building themes for the directory, clients, third-party marketplaces, or a theme shop, this is an opportunity to give back to WordPress. It is also an opportunity to improve the tools that you could benefit from as a theme developer in the long term. Automated theme tests help the entire theme ecosystem.

“Theme authors need to be open to this and understand that it is not all about requirements,” said Carolina Nymark, a Themes Team representative. “It is about improving theme quality.”

The project was, in part, informed by a Themes Team proposal in early 2020. Denis Žoljom identified three problems the team was fighting against:

• People don’t like to read requirements or handbooks.
• Some of the issues that are popping up are repetitive and could be caught automatically.
• Reviewing themes in Trac is really cumbersome.

The proposal’s focus was on moving reviews to GitHub, focusing on the third point. However, the Theme Review Action project could be the start of handling one or more issues.

The obvious solution is that the project can be automated. However, because the Theme Review Action project can be set as a GitHub Action, it leaves room for the team’s GitHub review proposal.

“Two things I mentioned to Steve — and these are my opinions –, is that we need checks that run on theme upload and on live themes, and we need a long-term solution,” said Nymark. “There have been attempts to automate testing before that have not been followed through, and without a plan for how the tool will be used, I am worried about spending time on it.”

The team had hoped that the Theme Sniffer project would lead to more automation at one point. It is hard to get hopes up after previous goals never came to fruition.

“I too have a similar concern where the project might not get enough adoption to make it to .ORG checking, and that is one of the reasons (other than just being super busy) that I haven’t been able to prioritize looking at [Theme Review Action],” said Themes Team rep William Patton.

While the team and some theme authors still use the Theme Sniffer, the UI leaves a lot to be desired. Nymark pointed out that it was hard for theme authors to distinguish between the baseline requirements and recommendations.

“To display messages from automated tools that are not strictly requirements is very difficult to get right,” she said. “For example, if a tool started reporting CSS linting errors for the WordPress CSS coding standards, many people would find that too opinionated and limiting.”

Theme authors, the group that reaps the most financial and reputational benefits from the theme directory, have often been reluctant to chip in. Few companies spare an employee to perform reviews or work on tools that developers and the team need. Calls for testing, feedback, and discussion often go unanswered, leaving a select few to do the lion’s share of the work. For this project to be successful and not feel like something foisted upon them down the road, theme developers need to be in the mix.

In the first episode of the WP Briefing podcast, WordPress Executive Director Josepha Haden Chomphosy talked about focusing on automation as one of this year’s goals. If there is one team that could use such tools, it would be the Themes Team.