26 Comments


  1. Hmm… I just tried to run Oenology through the service, but someone had already done so. It only got a 99. :/

    It seems that the site has escalated the severity of an INFO check (use of include()/require()) to the level of a warning. The end result is that a legitimate (and proper) use of include()/require() – to include functional files in functions.php – causes the rating to be downgraded.

    So, there’s some room for improvement there. But, in all fairness, there’s room for improvement in that check in the Theme Check Plugin as well. That INFO output should only be given if include() or require() are used in template files. The use of include() and require() should be ignored when used in functional files. So, that’s not entirely the fault of the themecheck.org site (though I’m unsure why an INFO check is being escalated in severity).

    Reply

    1. Thanks for your input, Chip! The site’s creator is definitely open to feedback and I’ll bring this to his attention.

      Reply
  2. gbaudhuin

    Great article ! Chip, tolerance level on includes/requires is a tough question because it’s hard to make everyone happy. Differenciating between functional and template files seems to be an interesting solution. I just have to find a secure way to differenciate those two kinds of files.

    Reply

  3. _s gets only 77. Not supporting avatars is a “Critical” problem. Seriously?

    Reply

    1. I’m sorry but I have to lol about that. While gravatars are a “standard” in wordoress themes, NOT using them certainly doesn’t mean that a theme isn’t secure or written with quality code. You should definitely submit feedback to the themecheck.org site author.

      Reply

      1. There are some false positives that will be corrected in the next few days (backticks in sql, false variable functions in printf). But some checks report true suspicious code : use of curl_exec, use of iframe, etc. One may use them in its own theme, but since the target of themecheck.org is people who download or distribute themes, theses warnings seem legit.

        Reply

  4. I tested my theme Spine and it gave a critical error because it wrongly flagged jQuery in a PHP file as a “variable function”

    Reply
  5. Ted Clayton

    The general idea with this ThemeCheck service is cool & useful. I like it, and despite some reservations, will continue investigating.

    Certain of the implementation-details that have been selected, look like they are or invite ‘issues’. Certain of these are readily addressed/fixed, while others; maybe not-so-easy.

    Calling what is done here “validation”, and then asserting that “security and code-quality” objectives have been achieved, is overstating what it is & does. This terminology & claim poses the service as something more-authoritative than it is. “Validation” of code is a term that already has an established meaning; it’s an advanced & challenging (‘impressive’) field of computer science. It’s virtually impossible than any ‘real’ validation is being done.

    So what is the ThemeCheck website doing, then, if not “validation”? Well, this service is of course based on a top-ranked WordPress plugin called Theme-Check, written by Simon Prosser and Samuel Wood (Otto). On their Theme-Check FAQ page, they describe in more detail what it does:

    … The theme check is designed to be a non-perfect way to test for compliance with the Theme Review guidelines. Not all themes must adhere to these guidelines. The purpose of the checking tool is to ensure that themes uploaded to the central WordPress.org theme repository meet the latest standards of WordPress themes…

    This theme checker is not perfect, and never will be. It is only a tool to help theme authors, or anybody else who wants to make their theme more capable. [emph. added]

    That’s a very humble description, compared to ThemeCheck’s terminology & claims. Toning down their highfalutin language shouldn’t be too hard. (I would check with Mr. Prosser & Mr. Wood, and ask to quote them & their FAQ-language. Those are heavy-hitters at WP, and their ‘implied endorsement’ is meaningful & valuable. )

    Next: the numerical scoring system. This is where the overall working-design of the ThemeCheck website might be harder to tweak.

    When we say things about products in the market, or more pointedly, when we publish statements about things & people, we fall under moral & legal obligations. We can’t just up & trash-talk the products of other entities, without responsibilities & consequences.

    If we are going say negative things about products, we have to have our ducks in order. It has to be real & verifiable, if we are going to eg rate the Ford Super Duty at 0, and the Dodge Cummins Ram at 100. Ford could take a dim view of that, and they do have recourse.

    To the extent that this scoring system is arbitrary & subjective – that numeric valuations have been ‘pulled outa thin air’ and assigned to points of Guidelines deviation & compliance – the ThemeCheck website is on shaky ground. They’re ‘makin it up’ – and can be challenged, successfully.

    Somebody is likely to ‘take exception’ to how their theme is be posed, at ThemeCheck. They have the right to ask that an accounting be forthcoming. People who publish claims & characterization about products are, after all, held to be morally & legally accountable.

    It would be a big surprise, if this scoring system is a consistently accurate or meaningful reflection of the usability & merits of themes. Best to tackle that weakness, sooner rather later.

    Other than than … nice idea, and best of luck with it!

    Reply

    1. I only wish this statement where true, “Somebody is likely to ‘take exception’”

      Sadly if you ever tried to get someone in another country to retract a statement; or even get a web host or other service provider to take down a site for overt copyright infringement, you may have experienced how little this “someone might do something if you do” phrase has to do with reality–about as much teeth as my sons 1 year old Bearded Dragon lizard…

      Reply
      1. Ted Clayton

        True: the ethical & legal often get short shrift on the net.

        But, the net can also do an impressive ‘judge, jury & executioner’.

        Especially when trying to get established with a strong self-identifying (competitive) net community – like WordPress.

        Once members with a meaningful voice decide they don’t like an activity or service – they don’t talk to lawyers or write essays on ethics – they just come around broadside and start pounding away.

        I don’t think that is necessary or called-for, certainly not at this stage. The Theme-Check site is a good idea, basically, and it could be a upgrade for WordPress generally, and for themes in particular. Themes need more attention; more acclaim … plugins are kinda hoggin’ the glory these days.

        I figure a shot or two across the bow is plenty-good for now, just to clarify proprieties.

        Reply
    2. gbaudhuin

      I’ve been thinking a lot about your point : “Calling what is done here “validation”, and then asserting that “security and code-quality” objectives have been achieved, is overstating what it is & does. This terminology & claim poses the service as something more-authoritative than it is.”

      I mostly agree with your point of view. Some claims definitly need to be adapted. This service is meant to be trustable, and as such, it should not over-promise what it does. However, the main goal of themecheck.org is to offer an easy and quick service. So, the claims also need to be easily understandable… We’re working on it.

      As it’s been said here, the service is new and we need time to adjust it, in a better way for everyone.

      People’s remarks on this page are of great help.

      Reply
      1. Ted Clayton

        In most uses of ‘actual’ validation, it is paired with a verification phase. It is a dual process, usually seen as “verification & validation”.

        Let’s consider eg the verification & validation of a pudding recipe (which is a program for converting ingredients into a define outcome or product, under specified manipulations).

        Eggs: check. Milk: check. Flour: check. We have now verified the correct ingredients.

        Oven preheated 350 degrees F: check. Ingredients combined in the given order: check. We have now verified the proper preconditions for program execution.

        To perform the validation phase, cook the verified ingredients: execute the program. Did the program-execution result in pudding? ‘The proof is in the pudding’. Do not use children to judge (or test) the pudding: use your in-laws, preferable your mother-in-law. If it’s pudding, the recipe is validated.

        To validate code, it must be executed. You can’t know if it produces the intended pudding, unless you run it.

        Verification is static. In an airplane version of pudding, we verify that the plane is ‘ready’ or ‘prepared’ to fly using a preflight checklist (while parked static on the tarmac). The case of an aircraft clarifies that the proof (or validation) is dynamic: it is in the flying. The preflight is the verification, in an airworthiness V&V.

        The WordPress Themes Guidelines appear to be a preflight checklist. The ThemeCheck plugin appears to be an automated tool, for conducting the static preflight or (pre-execution) verification component of a two-phase theme verification & validation.

        And then, verification & validation itself is but one phase in various multiphase processes that aim to establish eg code quality, and security, among other states & metrics.

        Reply
  6. Burak

    Considers Turkish characters as non-printable characters. FAIL.

    Reply

  7. I wouldn’t be too hard on the service. It’s new. Give the guy time to work out the issues.

    I neither endorse the service nor disparage it, because honestly I’m not certain whether or not it’s a good thing. I’ve looked at the code he has on github a bit. There’s potential there.

    On the whole, if it helps people make better themes, then that’s good. But the site seems more geared towards providing a numeric service for users to use, and I’m not sure that that is necessarily the best approach to take. A user shouldn’t be choosing based on code quality, but on design quality. The code quality should be a given, for all themes. We’re not there yet, but still.

    Back when we first started doing theme checking via automated means, the first tools pross wrote were like this: upload the code, have a process scan it. I rejected that at the time in favor of a plugin because it lets people check their own code, on their own test sites, and not having to rely on a central service. I still think that is more useful for developers, to be able to give them the tools directly instead of having the tools yourself and requiring them to jump through hoops for information. So what I’ll probably do is see where he goes with it in the code, and then steal all the best ideas for incorporation back into the Theme Check plugin. Makes the most sense. :)

    Reply

  8. Very cool idea, I like the concept and theory quite a bit.

    However, after testing a couple themes and looking at the test results of other themes it seems that some updates to the code need to be addressed.

    My themes first run through resulted in a very bad score, I host its code on github and therefore it had the .git folder included. While it’s good to warn that a hidden file is included with the theme, I am thinking that it would be better if it checked for malicious code within those files.

    Also, some of the things that knock down the score aren’t related to whether the code is good or the security of the theme, rather these warnings are related to theme author preferences.

    I can’t wait to see the updates and improvements. This is a very cool idea.

    Reply

  9. This can be a useful service if he’s providing checks that aren’t present in the plugin, but I think it needs some changes before it can be useful as a user-facing service. If the target audience is theme purchasers, the vast majority won’t be able to judge the merit of any particular error/warning, so they’ll need a lot more hand-holding on what really matters.

    Theme Friendly’s format has more potential as a user-facing service. It provides separate scores for errors and warnings, provides a manual review score, and lists features separately so that missing features aren’t interpret as “problems”. It also has a section where issues can be explained. This requires more work, though, so it’s not as simple as an automated scanning.

    Reply

  10. Really useful service this. Kudos to Guillaume for taking the useful Theme Check plugin and making it more widely available. I did notice a few themes that I tried, it said it they didn’t have a theme name in the css file, even when they did. No doubt these little issues will get sorted out over time though.

    Would also be good if it allowed for different versions of a theme. Currently if you try to validate a theme and another version has already been validated, it wont save the results.

    Thanks for sharing this great service Sarah and well done Guillaume!

    Reply

  11. This is a great tool. I’ve been using ThemeCheck plugin, but what I see from comments above, this service does more. It’s interesting to see how many scores my themes can get.

    Thanks for sharing!

    Reply

  12. I’ve used the Theme Check plugin with some nice results.

    Do you know of a similar plugin (or site) to validate the code of a plugin?

    Reply

Leave a Reply