7 Comments


  1. Hi Jeffro,

    I’m with you – although larger companies often have a culture of extensively testing each update.

    They have dev, test and production servers and promote changes through these, following through an extensive test plan. Oh and they can’t just do this, they have to take the change to the Change Advisory Board for approval (which may take a couple of weeks), create the test plan, the roll back plan, let all the stakeholders know of the work and the potential outage, co-ordinate the update so there is zero chance of the site going down when something major is happening on the website, etc.

    And yeah, they need to do all this for a 1 click, 10 second update. So they love a stable product that doesn’t change much.

    Thankfully a lot of these companies are realising that they need to change, but that’s the traditional enterprise IT mindset.

    Hmm, that’s all going into my WordPress and Government talk at WordCamp Gold Coast…


  2. At the risk of following on other conversations… In my opinion the reason that you have seemed to misinterpret this is due to the level of CMS’ness you use WordPress for.
    At a mid-to-enterprise level, having 5 individual releases release’s in 16 months (not including security / bug fix upgrades) is simply way too much – especially given how much changes in each one.

    Chris, like many of us, is ecstatic that security vulnerabilities are fixed and released ASAP. It’s the additional releases as well as security ones that really start to cause hassle.
    In the 7 months & 20 days since 3.1 was released, we’ve had 8 releases of WordPress. 8!!! That’s a release once every 29 days so far this year.
    That’s not sustainable past a certain point, and that point is different for many clients/businesses (purely anecdotal evidence: 2 of my big clients, and 2 medium clients this year have asked to be migrated away from WordPress to avoid just this).

    Chris also headed the last section “Security & Support Concerns”.
    It’s not just security. Each time there is a WordPress update to the backend UI (/wave flyout menus – 2 changes to workflow of Admin UI in 2 releases – mental) all training documents and manuals need to re-edited to match the new UI and workflow. Non-tech savvy people then need to go through some for of training (even if it’s just an e-mail update) to inform them about all the changes. That’s not a small amount of work, and something that people forget when talking about upgrading – it’s not just clicking a button and magically everyone knows where everything is and how everything works.

    Obviously we all love WordPress, but realistically it’s very “CMSlite” – especially compared to the alternatives (Drupal, Joomla, Umbraco, Typo3 etc). The ability to knock out a micro-site or community site in English quickly is wonderful in WordPress. As soon as you want stability or extensibility of data, it’s ability to match the competitors on this front drops quickly.

    If I may give another example close to home:
    e.g. We’ve had Custom Post Types in WordPress since 2.8 – June 11, 2009 – and a UI for them since WordPress 3.0 – June 17, 2010 – but in your last 2 plug-in review you’ve raved about how they use Custom Post Types because its so new/unexpected/rarely-used. And you’re not wrong !! Right now, the data isn’t that extensible – or if it was the Custom Post Formats that were part of 3.1 would have been an Akismet-esque included plug-in as originally discussed.

    CMS’s aimed at non-bloggers/small-websites simply don’t work well acting the way WordPress does. And that’s ok, its clearly not the market it’s aiming for, but lets not get defensive of WordPress every-time someone points out it’s faults :)


  3. @Stephen Cronin -

    larger companies often have a culture of extensively testing each update.

    Wait, are you saying that we shouldn’t test software updates to mission critical systems now? Why wouldn’t we test it? Is this what happens after a year of 1-click-update, that we start to forego testing?

    They have dev, test and production servers and promote changes through these, following through an extensive test plan. Oh and they can’t just do this, they have to take the change to the Change Advisory Board for approval (which may take a couple of weeks), create the test plan, the roll back plan, let all the stakeholders know of the work and the potential outage, co-ordinate the update so there is zero chance of the site going down when something major is happening on the website, etc.

    One of my clients has a legal requirement to have the share price on the front-page correct to under 15 minutes previous. Their website can not go down and cannot be akamai-type cached. Thorough testing of any update to any of their systems is not done because of their old-fashioned IT strategy, it’s because without it, they are royally fornicated to the tune of millions of dollars.

    And yeah, they need to do all this for a 1 click, 10 second update. So they love a stable product that doesn’t change much.

    Um, no mention of training/support/documentation/translation the thousands of staff members on the UI and Workflow changes that WordPress pushes out every release. Obviously, that’s just 1 click and 10 seconds to update too, right?

    Thankfully a lot of these companies are realising that they need to change, but that’s the traditional enterprise IT mindset.

    There’s a very funny dicotimy going on with WordPress. On one hand we get upset when people say that WordPress doesn’t tick the box for every solution; yet on the other hand we play the David and Goliath card when talking about Enterprise level IT systems. Pick one :)

    Even clients who have embraced the business transformation required to move to agile development and deployment still need to actually test software before releasing it live.

    EVERYONE should test each update for each of their sites.
    Testing before you update the core of your Content Management System should be done by, surprisingly, you.

    Why should Enterprise level clients be any different?


  4. @Kevinjohn Gallagher -

    Wait, are you saying that we shouldn’t test software updates to mission critical systems now?

    If you wait until a “release” before you begin testing, then of course you’re going to have problems. Realistically, you should be under a continuous cycle of testing and development at the same time.

    For example, why is your “dev” site not running trunk? You’d get at least a month or two head start on any testing and new development for upcoming changes, and by the time release rolled around, you’d be already tested and up and running. All it would take would be a push to the test system, a final check, and you’re ready to roll it out within a day of release.

    It’s not that testing is bad, it’s that you’re starting to test it too late. IMO, of course.


  5. It’s not that testing is bad, it’s that you’re starting to test it too late.

    Oh I agree, but the OP, and many here who lament “big business” are aghast that there is use of, god forbid, an actual test plan (test plan 9 from outer space, presumably).

    I realise that sometimes people think that I’m a bit brash, or loud (almost 20 years in Scotland – I can’t help it, this is just how I talk haha) or opinionated. I probably am all of those things. But in the last 3 years, watching WordPress’s explosion, it pains/worries me that so many take stock from their singular viewpoint.

    What started out as Guidelines, suddenly became Standards, before morphing into Evangelism, and now close to Rhetoric (e.g. Thou shalt upgrade the very second a new WordPress version is release without question or business case); and those who still view and act upon these things as Guidelines, as after all it’s open source software, are viewed upon with distain and distrust.

    1-click-upgrade is wonderful, truly bloody awesome, but it’s also had a negative effect, where a greater number of people put their faith in the testing thats already been done by the community, and just click-it-and-see. There is always a balance between making it easy for someone, and making them lazy.

    Which has added another problem. The original thought process was that there were people out there who didn’t know how to upgrade, and now they can. Well now we have people who know how to upgrade, but are fed up of doing so. People choosing not to upgrade simply because “meh, there’ll be another one in a week or so anyway”. Security wise (just to bring it back to Jeffro’s post), we’re making people very numb to updates, something that is not good at all.

    If you wait until a “release” before you begin testing, then of course you’re going to have problems. Realistically, you should be under a continuous cycle of testing and development at the same time.

    why is your “dev” site not running trunk? You’d get at least a month or two head start on any testing and new development for upcoming changes, and by the time release rolled around, you’d be already tested and up and running.

    At the risk of pouring salt into old wounds, I have a simple answer to that: Capital_P_Dangit()
    (plus the clusterf**k that was 2 years of bbPress and BackPress of-course).

    Faith in the Alpha > Beta > RC > release system no longer presides in my company, or my clients. When the core team (hi Matt) are able to add in features after the RC to rewrite file paths on a marketing whim, then what’s the point of Beta’s for people like us? When did “feature freeze” mean “feature freeze, unless we want to add some”

    I mean, the current Beta, not alpha or nightly but Beta for 3.3′s admin section cant be navigated in IE7. Not a small feature, but the Admin doesn’t work, at all, in IE7. How in the blazes of all that is Joss Whedon did that hit Beta? If only there was a test plan (9 from outer space). Just to clarify, every single person on the core team and who works on the releases for Automattic and the dev team, every single one of them didn’t test it on IE7, or even IE8 and hit compatible mode.

    To quote Wikipedia here:

    Beta… It generally begins when the software is feature complete. The focus of beta testing is reducing impacts to users, often incorporating usability testing. The process of delivering a beta version to the users is called beta release and this is typically the first time that the software is available outside of the organization that developed it.

    And yet we’ve got a Beta version in 3.3 that NO features work ing IE7, and no-one tested it. Not in-depth testing btw, but no-one even loaded it once or they would have seen the menu not working. Bluntly, with Beta’s like this, and releases with the likes of Capital_P_Dangit(), running trunk on our dev site sometimes feels like farting against thunder – something must smell terrible for us to even take notice!

    Put frankly, and with all due respect to someone who I’ve admired for a long time Otto, the state of WordPress beta’s over the last year (and 4 releases) have been so far from what I’d call a beta, that we only use it to give us a heads up of anything truly huge thats changed.

    I’ve been saying it for years now, and people really ridicule me for it, but maybe we need a slightly better ratio of Project Managers to Happyness Gardeners, cos right now for the WP core it’s 6 developers, 0 Project Managers and 1 community manager employed by WP.org (who tested the Beta with 12 people on macs & 8 on PCs, but everyone used FF or Chrome). Frankly, if thats a good test plan, then can the last person here switch off the lights please?


  6. @Kevinjohn Gallagher -

    My comment was just a quick one to point out that enterprise level companies have different point of view. Perhaps, I better put some context around my comment.

    Although I have worked with large commercial companies, over the last 4 years I’ve been working in Government, so that’s the main perspective I bring. My experience in govt has been that the ICT department (which I spent several years with) is similar to most enterprise level ICT departments, although a) they are even more conservative and b) ahem.. perhaps they aren’t as well organised.

    I’ve seen some things which are just silly. When I was running the web team, we had a case where we could not get small, non-critical CSS tweaks in the template onto the site without having to go through dev > test > prod. This despite the fact we’d tested it on our own local server. At least we didn’t have to go through CAB (Change Advisory Board) for this level of change, but it still took a couple of days for the server team to push the change through to the live server. And when I say live server, it wasn’t a public facing website, it was an internal facing one, with non-critical information, which know one knew about yet.

    I defy anyone to argue that’s sensible!

    However, I’m not arguing that testing before upgrading isn’t sensible. I’ve been quite vocal about that in the past, for example in my The Solution To The Lack Of WordPress Beta Testing post from last year, where I:

    argue against the unthinking update immediately advice
    state that there are not enough end users involved in beta testing
    propose a staged rollout (such as Google uses for Google Analytics)

    Admittedly I could have been far more confrontational in that post, but I don’t think that’s constructive.

    Our department recently chose a new CMS and WordPress wan’t even on the radar. I’m going to go into the reasons why WordPress doesn’t have more traction in Government (at least in Australia) in my upcoming WordPress and Government talk. One of the reasons is this very testing issue. The recent discussions in the community about moving to a Chrome like ‘auto upgrading in the background’ feature is just not going to cut it with the enterprise.

    Our websites have quite a bit of content that we are required by legislation to have. We don’t lose millions of dollars if the site goes down, but there’s always the potential for the government to come under criticism from the media and the opposition party. And if the Premier or a Minister is announcing something, the site had better be up!

    So I am a fan of testing. However, the traditional enterprise level ICT approach (at least that I’ve seen) has been to put risk mitigation and process above delivering benefits to the business and to customers. That needs to be balanced and that’s what I meant when I said:

    Thankfully a lot of these companies are realising that they need to change, but that’s the traditional enterprise IT mindset.

    I’m not actually saying they shouldn’t bother testing. However, I do believe that they can look at ways of streamlining the process. A major version release may require a lot of testing. Security releases within a major release may require less testing, depending on what was changed (which you can look at in trac). If WordPress is being used for the main mission critical website, test more! If it’s being used for a secondary less important website, maybe you don’t need to test as much.

    My original comment was obviously too brief to get my position across, so hopefully this has made my position clearer.

    Oh and by the way, we still use IE6 here. IE7, we dream of IE7! :)

    (actually people can request to upgrade to IE7 and I get a real browser because I’m in web).

Comments are closed.