45 Comments

  1. Mika E (Ipstenu)

    In defense of the theme team, the theme and plugin review systems are totally different. We’re much more cavalier about what kind of plugins we accept, since plugins are (by their nature) meant to do anything. At plugins, we’re not stringent about ‘best practices’ either and we don’t have ‘plugin requirements’ in the same way. We try to enforce safety and security as best we can, but people are encouraged to reinvent certain wheels.

    Themes need to be way way way more consistent and deliver the same quality of experience. You really can’t compare the two systems.

    Report

    • Ajay

      Mika, how big is the plugin review theme vs. the Theme review team?

      I agree with Tadlock’s point on following guidelines. I wouldn’t be surprised that a lot of themes submitted break basic guidelines which ends up choking up the queue for genuine good themes which don’t

      Report

      • Jeffr0

        Otto, Mika, and Pippin although anyone on the .org team can get involved if need be but those are the big three. Let’s see how far off I am.

        Report

      • Mika E (Ipstenu)

        Pippin, Otto, and I do the brunt of the reviews and people wrangling. But there are five active team members.

        I have to stress, it’s a very different world than themes. I’ve looked into being a theme editor, and I find it daunting. Of course, I bet an experienced Theme Reviewer would look at the chaos of plugins and think we’re insane ;)

        Report

    • Ed

      @Mika – Absolutely zero reason for you to feel that you have to defend anything. Personally I think it is insane how much you guys (THE WP Team) take on personally. Obviously your workload is divided up into teams within THE WP Team, but still as WP popularity only continues to increase and more and more folks use WordPress then the strain and workload for you guys will only continue to increase. Users are asking for more personal hands-on attention more and more. Something has to give at some point. ;) I posted a general though in another WP Tavern post to maybe farm out things/tasks to a 3rd party source. Maybe the day is coming (or has arrived) where supply and demand for everything WordPress is just too much to handle for you guys in-house and it is time to farm out non-critical/non-essential stuff to a 3rd party to take some of the workload/strain off of you guys so that you can focus on strictly essential/critical WP stuff.

      Report

      • Ben

        The problem is the pool of community-driven individuals who do these reviews of plugins and themes (for free mind you) is much smaller then the pool of total users and developers. For example, Mika works for a web hosting company, with the company being nice enough to pay for some of Mika’s time to help out at WordPress.org. She does not “work” for WordPress.org. Neither does anyone in the theme review team – maybe one works for Automattic? So that means until we can “develop” more community-minded folks to help out, we’ll always have this issue. Actually I don’t think its a completely solvable problem due to human nature (self-interest versus community interest and just general priorities of individuals that can change over time). I do think we can reduce the burden of entry though and that will help in encouraging more engagement.

        Report

  2. Omaar Osmaan

    Been around for a year in the Slack Channel (#themereview), engaged in arguments about automation, including other possible improvements for the Theme Directory-

    In the last couple of months I’ve realized that, the whole process of the Reviewing Themes on WordPress repository needs rethinking. It’s broken, sadly.

    Report

  3. Jami

    With so many WordPress community members having some dev knowledge, I’m really surprised not more are volunteering to pick up a theme to review. If just 50 community members each reviewed just one theme a month, things would drastically improve.

    The actual theme reviews aren’t all that difficult and takes minimal time (maybe 1 hour a week?) but perhaps the hardest part is just getting started. The trac system is confusing and intimidating, and it’s not always clear where to ask for help.

    There’s the theme reviews handbook and Slack channel like mentioned above but I can understand how it would be difficult to simply get started.

    Report

    • Ashley Evans

      I think you really highlighted the problem Jami. Getting started feels incredibly daunting.

      Theme review is something I’d like to get involved in but I’m feeling like I don’t really know what I’m doing in the process. Even though the instructions are pretty clearly laid out, the Trac system feels daunting. Especially because I didn’t feel like I really ‘clicked’ with it during my own theme review.

      Report

  4. Henry Wright

    A well written plugin or a theme that follows guidance closely can go crazy at a future date after it’s been accepted. I’m wondering if there are measures are in place to set themes or plugins back on the right path (if they go “bad”)?

    Report

    • Mika E (Ipstenu)

      Plugins yes. A couple of us get emails for every single plugin check in (thousands a day, makes the trac firehose look tiny), and we have SVN block the most egregious issues. But we also have a lot of email filters to pick out naughty people.

      We do rely heavily on people reporting issues to plugins(AT)wordpress.org (if you do, please remember to link to the specific plugin :D )

      Report

    • Justin Tadlock

      We actually just suspended a theme recently for something like this. This was reported to us. We’re currently giving the theme author a chance to bring the theme back into compliance.

      One of the methods we plan on utilizing is a “report” button to report such abuse in the future.

      Report

  5. Ahmad Awais

    I have been planning to release a few free WordPress themes at WordPress.org but I confess this time consuming review process scares me off. Now if I think about it, I guess that’s the reason why I end up releasing a plugin.

    If I may be bold enough to ask, why is the theme review team so strict about such standards, don’t you think that’s actually limits the creativity? That said, I respect the set-standards and understand how it helps. The point I really want to make is that what would happen if theme review team will adopt the standard plugin review like Mika stated “We try to enforce safety and security as best we can, but people are encouraged to reinvent certain wheels.”

    Just thinking out loud.

    Report

    • Ryan Hellyer

      This was discussed in depth before the theme review team was created. Some of us thought that the review process was far too stringent restrictive and would limit creativity. The decision however was made to create a restrictive environment and that that the loss in creativity was worth it for the benefits of ensuring the theme repository contained a higher quality of code.

      So basically, you are not alone in your thoughts, but there seems to be more people who feel the opposite way.

      Report

    • Justin Tadlock

      I’m actually all in favor of reinventing the wheel. But, in such discussions about how guidelines limit creativity, we need to be talking about specific things. Specifics matter.

      Designing with certain limitations doesn’t always mean less creativity though. Good designers/developers are going to build awesome stuff regardless or because of limitations.

      There’s also another side to this: plugin compatibility. Many things we get pretty strict about because if we’re not, themes simply break plugins. As a plugin author myself, I rarely see a conflict with a repository-hosted theme. I see conflicts almost daily with non-repository themes.

      Present your case to me on Slack. I’ll either tell you how you could perhaps build it within the context of the guidelines or help make your case for a guideline exception to the other admins.

      Report

  6. Howdy_McGee

    I’d like to contribute to the Theme Review team but I’m worried that I may not have enough time to go through the review process. I’ve been developing custom themes for a few years now – can anyone comment on about how much time it takes to completely review a theme? To give a theme a good look through how much time should I look to set aside or is it really variable on the extense of the given theme?

    Report

    • Jami

      It ultimately depends on the theme but generally speaking, it doesn’t take me longer than an 60 – 90 minutes to do an initial review. There may be some back and forth as the theme developer is making updates before you approve it but it’s truly not a big time commitment.

      There are some people that take on multiple themes at a time which requires a greater commitment but if you’re just sticking to one theme, you’ll find that it’s quite manageable and enjoyable.

      Report

    • Blair2004

      That’s exactly what i told on my topic. Review process seems to be long…
      @Jami, we did’nt have the same workflow.

      https://www.facebook.com/groups/advancedwp/permalink/1045428428852704/

      Report

  7. Dave Chu

    Jeff,
    Thanks, that’s very helpful!

    Although I’ve been building themes for quite awhile now, I had not seen the full theme requirement list for wordpress.org. It sounds like good basic practices, and not as many as I’d anticipated. I’m pleased that I’m doing most of them already. :) I had thought that the Themeforest WP requirements were more arcane, but as I take a look at them now, they don’t look too bad, nor is it hugely different from the .org one. Maybe I’m getting smarter. ;)

    I haven’t been using the Theme Unit Test material, but I’ve been helped a lot by using wptest.io, another excellent set of test data.

    Even though I enjoy doing things my own way, I can see the reasons for enforcing standards. The good results of that can be seen by the general quality being a lot higher now! We who have been around for awhile remember the kinds of clunkers you could run into a few years ago.

    Still, that backlog sounds daunting. So if I release my stuff free, I think I’ll just do it on my own site, yet use the standards to help me build it cleanly. That way, I get the benefit of the good advice without the waiting line. :)

    Thanks, Dave

    Report

  8. itmightbe

    Design isn’t taken into consideration at the moment, that’s the big issue. It would bring the reviewing time frame at around 1 month. Just take a look at the newly released themes, it is really frustrating and depressing to see such themes pass through…

    At this stage you need to wait at least 2-3 months to get a theme accepted & live. If for some reasons you are unlucky and get a fake reviewer that period gets at around 4-5 months. Insane…

    Also, already existing themes being submitted by users, with just a few modifications should get deleted on the spot. There is no innovation/learning in that. They just want a freaking link and flood the trac with useless stuff.

    Report

  9. Trevor

    I just joined the review team and did my first review over the weekend, the tickets are out of control and they need help. For that review it was opened 15 months ago! And the theme developer had used an options framework, so all that work and time only to be told they had to switch over to the customizer and do more work.

    Themes are so important to the end user, as it’s their main interaction with their site and their site’s ultimate identity. I hope the review team can sort out a better way to keep on track, but in the meantime, if you can, lend them some help.

    Report

    • Blair2004

      All those old ticket may be deprecated now… and this is so dishearting (i’m a young WP developper)…

      How to make people get more involved on theme review ?

      Report

  10. wLc Designs

    It took roughly that long for my theme to get approved last year. At first I was miffed until I took a look Trac tickets. The amount of theme submissions they receive weekly is insane. I don’t envy their position.

    Report

  11. Blair2004

    I agree with the fact that theme reviewer a volunteers, but 6 months for a theme to be approuved, it’s long. I’m not so far with 7 weeks already for Span Theme (https://themes.trac.wordpress.org/ticket/27606).

    About Ashley Evans case, this is a perfect example that the system is not broken necessarily, but it’s no more efficient.

    Report

  12. Andre

    I know the review team is working on improving things because I often go into the Slack and watch and read their meetings. It’s good to know they are aware of what is going on. I also know that they work extremely hard to try and maintain and make the whole process better.

    Still, the review process needs additional fine tuning. I brought this up to their attention before but themes are approved only on the basis of code and not design. I would love to see the theme directory focus on design quality as well, something to seriously compete with marketplaces and to offer users with quality themes. The end-user doesn’t care about code (although important), they want themes that look great and offer features.

    I will say though, compared to just a couple years ago, quality theme design has definitely improved when I look at the directory.

    My idea was to have a 3-tier review process:
    1st is to do an initial visual design review.
    2nd is to do a code review.
    3rd is for the admin’s to give final approval.

    The 1st tier would greatly reduce the backlog of themes and offer up faster review times for the ones that get through, perhaps even increase reviewer/author interaction with recommendations. I believe ThemeForest does an initial visual design review and their review times are down to like a week.

    I also believe that to ensure you have quality reviews, you need quality people. Reviewers should somehow be interviewed by the Admins, as though it’s a job they are applying for. To ensure the reviewer will take the role seriously and not walk away from the middle of a theme review, to fully grasp what it means to be a reviewer, to understand that it does require your time, and to let the author know you are there for them.

    I agree that design can be subjective, but reviews should be 50% design and 50% code quality. People with a flare for design and enough experience would be ideal for this review process.

    I’ll be honest, I’ve considered ending new submissions (I currently have 3 themes in the queue), but we will see how the next month goes. I’m also thinking about adding the downloads to my site and provide support from my site as well…I’ve had many people waiting almost 3 months for my themes to go live at .org.

    I know I’ve been asked to be a reviewer, but I’m still out on that decision for the time being.

    Report

    • Justin Tadlock

      Unless we begin rejecting/failing themes, adding an additional design review layer will actually increase the amount of time that themes take to get through. However, it’d push well-designed themes through the queues faster, of course.

      TRT is in the business of working with theme authors to bring their themes up to certain standards. The comparison with ThemeForest doesn’t apply because they don’t do “reviews” in the same sense. They have more of a pass/fail system, whereas TRT covers each specific issue with theme authors.

      The official directory is a different beast. We try to be as all-inclusive as possible. We want first-time theme hackers as well as seasoned developers. That’s why I, among several others on the team, are almost entirely against any sort of design review. It creates a larger barrier to entry for first-timers. There are some objective design issues that we’re willing to go along with, but the biggest battle we must win first deals with the underlying code. A poor design never broke anyone’s entire WP install. However, some poor PHP code has.

      TRT’s position is that we want to get rid of the initial review process altogether. That’s the eventual goal. We want submissions to be automatically approved and set live. Then, we take the time to work with theme authors to fix any potential issues later. That’s where I believe having design feedback would be most helpful.

      The idea is to become less like gatekeepers and more like educators who pop in to help with things that improve the experience for the end users of these themes.

      Report

    • Ryan Hellyer

      Instead of focussing on design reviews per se, I think the best designs should be brought to the front of the repository, and traffic driven to them. This doesn’t need to involve long manual reviews. A simple star system from approved and unapproved dot org users would work fine IMO. That way it can be automated, and not create a backlog like the regular theme review process.

      This gives motivation to theme designers to create something amazing looking, but does not block crappy looking junk from being submitted. Also, one persons junk may be another persons Picasso, so I don’t see any point in blocking something purely based on it being an eye sore.

      Report

  13. Tony Hayes

    Perhaps one way of encouraging new reviewers to get on board and speeding up the process is to offer a priority queue for anyone who completes a review of another theme?

    I think that contributors could be rewarded like this and many theme authors would jump at the chance to get their theme reviewed sooner simply by reviewing someone else’s, especially since the wait times have gotten long.

    It would also encourage them to familiarize themselves with the review process in more detail themselves and so more quickly identify issues in their own submitted themes, reducing the workload on the other reviewers.

    Another similar option, which may help resolve the backlog, is to (also?) offer a paid priority submission (that goes to the reviewer on completion) … I realize this may conflict a little with the “free” philosophy of wordpress, but as a temporary solution it could be tested at least until the backlog is reduced to a manageable size.

    Report

  14. Tony Hayes

    On a different note, for myself, I probably won’t bother to submit my new theme (even though it’s awesome) because:
    1. I am a primarily a plugin coder and so my approach to theme authoring is improving on standards, as such my code is hardly likely to ever pass the theme check standards. :-/
    2. Coding the customizer (mostly callbacks) for every theme option is way too much of a pain, and I have specifically used theme options framework to avoid it in the first place only to find out when looking to release my theme it has now been made a “requirement”? :-(

    I personally think the customizer requirement should be dropped, as there can be many options in a theme that do not even fit the need for a being a customizer option as they do not affect the visual outcome at all. Anyone know how strict this requirement is or can I just limit to the actual relevant display options and hope to get approved? If so, I might bother to do it and fix any issues for #1 too.

    Report

  15. Mahesh Waghmare

    Yes, It’s too long process of Theme Review.

    Nobody know ( Theme Author / Theme Reviewers ) the current process of Reviewed Theme.

    E.g.
    – How much reviewed by Theme Reviewer?
    – Which things are Reviewed?
    – What the percentage of Theme Reviewed? (10%, 30%, 50%, 96% etc.)

    If ( Theme Reviewer ) create any `CheckList` or `ToDo` list of known issues with `Canned Response` such as:

    ————————-

    # header.php

    1. Required: Have a valid DOCTYPE declaration and include language_attributes.
    E.g.
    <!DOCTYPE html>
    <html <?php language_attributes(); ?> >

    ..
    ..
    ..

    # footer.php

    ..
    ..
    ..

    Etc.

    ————————-

    Why This? Is this Useful / Useless?

    – I think Its useful for New Reviewer. Lots of things are common in each theme.

    If there is any predefined `ToDo list` / `CheckList` of known issues with `Canned Response` (It will be Google Sheet Or any other) then:

    – It’s useful for Theme Author to fix the listed known issues.
    – It’s useful for Theme Reviewer to updated the status of reviewed items. (YES/NO)
    – It’s also useful for New Reviewer to know which things are reviewed. He/She will cross verify the reviewed items before approved it.

    ————————-

    MOST IMPOTENT: Reviewer could not approve the theme from reviewing just `ToDo list` or `CheckList`.
    – Reviewer need to check each file line by line.
    – Use developer plugins for review.

    Reviewing themes is a great way to learn theme development and what not to do.

    ————————-

    I think (definitely) `Old Theme Reviewers` follow some common `steps` for any new theme review. Don’t know is this write or wrong. Just share the things which I thought.

    ————————-

    Report

  16. Thomas

    When I first started developing my theme WP-Forge, I decided that I was going to follow the guidelines set forth by the TRT. I also used the wptest.io like Dave Chu mentioned, which is, in my opinion, better than the standard theme test data.

    However, when I finished developing the theme I did not submit it to the TRT. I was afraid to. I was afraid that she wasn’t good enough and that I didn’t do things properly. I feared that I would have to go back and do it all over again. So instead, I offered it for download on my site as well as Github.

    But as time went on, people were asking me if i was going to submit it to the TRT. Eventually I decided that I would and I did. It took about three months to get my first initial review and to my surprise, the changes that were needed were minor and mostly dealt with the escaping of functions (I believe thats what its called, correct me if I’m wrong). I was actually thanked in an email for doing things “the right way” by a member of the TRT, which made me feel great.

    Now I have a theme that is coded to standards and used by a lot of people and I get told all the time its the best theme they have ever used.

    If developers would just try and do things right the first time and maybe ask questions on the TRT Slack channel before they submit, they may find that the changes they may have to make are miniscule. This could reduce the time it takes reviewers to cover a theme.

    But I do like the automated idea. This could speed things up considerably and allow the TRT to work with the developers. Educated developers would mean better built themes.

    Report

  17. Ed

    Otto has already created this plugin: https://wordpress.org/plugins/theme-check/ which checks Themes to make sure they are up to theme review standards. So how about making this plugin mandatory and then taking it one step further and have Otto’s plugin do any/all additional automation that is currently being done by human Theme auditors and once Otto’s plugin is run on a Theme and the Theme “passes” then automate that “pass” by notifying a Theme auditor that the Theme has “passed”. Of course a side by side data/code change comparison would also have to be done at final Theme audit to ensure that nothing was changed after the Theme “passed”. Seems like this would be a smart way to go and build on.

    Report

    • Justin Tadlock

      Theme Check is already mandatory. If a theme fails on any required or warning stuff from that, it doesn’t even get uploaded to the system.

      The more automation the better. Contributing to that plugin would be a great way to help both the team and theme authors out.

      There’s only so much such a code can do though. For example, it can’t tell us that a theme is licensed correctly. It can’t distinguish between genuine and spammy links that are hardcoded. It doesn’t know if custom post metadata is related to the visual design vs. holding content. That’s why we still have human reviews (though we’d like to get away from having to do initial reviews before making themes live).

      Report

      • Ed

        Good to know that Theme Check is already mandatory. Yep, humans have to do the final audit, there is no getting around that, but anything that can be automated helps speed up the whole process. My choice of words above was a little too “magical”. Scanners are a very convenient tool, but of course they only work if there is a matching pattern. Known vs Unknown. ;)

        Report

  18. Ed

    This idea just popped into my head and it is either brilliant or just plain dumb. Personally I am leaning more towards the brilliant range. ;) It is fairly common practice in a lot of forums to award “status points”, “status titles”. By adding something like that to all WP user accounts you would be creating incentive to answer other users forum questions. This would probably reduce the WP Team’s workload in regards to answering random forum questions. Personally I feel that I would take on more random WP forum questions if my user account displayed something like this for example:

    Username
    Member
    WP SuperStar (yeah LOL, but it gets the point across)

    Report

    • Tony Hayes

      forum points don’t accurately reflect someone’s coding abilities though, just how active they might be, unlike the point system on stackoverflow etc which is far more of a combination of ability and activity. but nothing like this is going to indicate how well someone has created a theme itself.

      for themes submissions, a simple user meta flag (or count) indicating the user has a previous theme(s) that passed review could work though, if it would allow them to (somehow?) put new themes online sooner, at least in the trust that they will not do anything dodgy to put their other themes/reputation at risk… and it means they at least familiar with the review process themselves having gone through it before, those themes are more likely to have less problems from the outset.

      and/or, since the entire review process is openly published anyways, there could be an option to just check a box saying “I have self-reviewed my theme and to my knowledge it passes all the required review tests.”
      This in itself could give enough pause to authors to reduce the number of themes submitted that are just plain going to fail as they are not ready yet.

      Report

  19. Amanda Rush

    I’m glad to see the comments in this thread from the theme and plugin teams. It’s somewhat comforting to know that the Accessibility Team isn’t the only team that’s completely overwhelmed with work, and the next time someone in the wider accessibility community bitches that there aren’t enough accessible themes, or that accessibility-readiness should be required, or that we could enforce plugin accessibility if we really wanted to, I’m passing along this post.

    Report

  20. Steve Burge

    We wrote a post on the GoDaddy blog about our experience: https://garage.godaddy.com/webpro/wordpress/how-to-get-your-theme-published-on-wordpress-org/

    Many themes are published in a few weeks or months, but if you’re not proactive with your review, it can take longer. Our approval process took nine months — and some themes have been in the process for more than a year.

    If we learned anything, it’s that we needed to be (gently) pro-active. The review team was overwhelmed and the theme authors who advocated for themselves probably got published faster.

    The Slack channel was a big help … that was by far the most useful source of advice that we found.

    Report

  21. Steve Truman

    Just a follow up on this post and I am not complaining – just stating our experience. The last thing I want is to upset the review team and get pushed back in the theme review que.

    12 months on for when this post was published – if the theme review system was considered broken because theme authors where waiting 2 months for the review process – what would you call it now where the wait is 5-6 months for the first review?

    We submitted our Theme Responsi 5 months ago and are still waiting for a reviewer to be appointed (I know they are not appointed – but rather pick a theme to review). At the time we submitted it was number 860 in the review que. Over the last 5 months it has now progressed to number 96 in the que. There are currently 3 themes being reviewed of the 95 theme that are ahead of ours in the que – so that means there are still 92 to be reviewed before ours.

    When we submitted the theme the WordPress version was 4.5.1 – it is now 4.6.1 with 4.7 due out in November – my guess is that it will be released before the theme is reviewed and approved for release.

    We where told on the slack theme review channel to expect a 4-6 month wait for the first review so it is not as if anything has changed in the last 5 months.

    Has anything been done to ‘fix’ the long wait for a theme review that you raise in this post? Again in your post you said some themes where waiting 2 months or more for a full review (oh I wish). Now 12 months on it is 5 to 6 months for a reviewer to even look at the theme for the first time.

    That would suggest that the situation is getting far worse instead of improving.

    Report

Comments are closed.

%d bloggers like this: