Cancel

Jeff Chandler

Behind the Scenes of WordPress 4.2.3 With Gary Pendergast

When WordPress 4.2.3 was released last week, not only did it patch a critical security vulnerability, but also adversely impacted a number of sites. Changes to the Shortcode API which were necessary as part of the patch caused some plugins that rely on the API to break. These changes were not immediately communicated to plugin developers. Nearly eight hours after its release, a post published on the Make WordPress Core blog explained the changes.

The release process of WordPress 4.2.3 left plugin authors and users scratching their heads. On one hand, point releases are not supposed to break anything. On the other, affected plugin authors were left in the dark for nearly eight hours wondering why a point release broke their plugins.

Gary Pendergast who works for Automattic, is a WordPress core contributor, and a member of the WordPress core security team, reached out to me for an interview. In the following conversation, we discuss what happened behind the scenes before 4.2.3 was released.

He clears up some confusion on when the changes to the Shortcode API were implemented. He also admits the team made some mistakes and has already implemented changes to improve the release process. One of those changes includes publishing a post on the Make WordPress Core blog as soon as the update is pushed out to sites.

I appreciate and thank Pendergast for reaching out to me to have this conversation. I look forward to similar collaborations with members of the core team in the future. A transcription of this interview is not available but if you have it transcribed and would like to make it available to the public, please contact me.

[powerpress]

Category:
Tags: , ,
38

38 responses to “Behind the Scenes of WordPress 4.2.3 With Gary Pendergast”

  3. The issue still remains that automatic updates should NOT be set to “on” by default.

    You can say “I’m sorry, we won’t do it that way again” all you want, but the issue is still that this shouldn’t have happened and it will again. Maybe not in this area, but in another.

    But thanks for reaching out @Gary Pendergast.

    Create forum topic from comment
    Reply

  6. I don’t understand the reasoning behind shutting off Auto Updates even in light of this issue. It seems far better to have a site that is slightly off kilter for a day or two because some plugin confilcts are being sorted out than to have a site that is hacked. I was under the impression that security trumps everything else. Did I miss a class?

    Create forum topic from comment
    Reply

    • @Bob Schechter,

      Yes, you missed the class on risk. Everything is risky, and so you have to weigh different risks in each case to decide what is the best course of action. The core team has no way of weighing the risks for each site. That is what each site admin should be doing.

      In my case, for example, there was zero chance of any of my sites being affected by the latest security issues which this release was designed to patch. Since updating software always involves some risk, that meant that my sites were at infinitely more risk from the updates than from the security issues.

      In such circumstances I certainly want to be updating manually.

      Create forum topic from comment
      Reply

    • Hello Bob,

      Automatic updates can screw up a website, imagine if TechCrunch, BBC or other big sites that use WordPress crashed?

      Obviously every update can’t be tested with every singe combination of themes & plugins out there.

      I have a copy of all my own websites that are testing sites, each “copy” site has the exact theme/plugins as their “master” site. I test out onto the “copy” site. If nothing goes wrong, I go update the “master” site.

      I am follow twitter accounts of WordPress, WPTavern and about 45-ish other members of WP Community. Also more e-newsletters that I can remember right now. So I am covered when it comes to updating.

      Imagine if let’s say Amazon had an update to whatever software their site uses, and it crashes. It can lead to thousands, maybe million or two or more losses.

      I personally have turned automatic updates off on all my personal and clients websites.

      Since most of sites I manaage use use JetPack, I know when there is an update on my site, that I have to go update the other sites.

      If it’s an URGENT SECURITY update, then I will know WPTavern will cover it.. I have notifications ON on my Twitter alert for WPTavern twitter account.

      Create forum topic from comment
      Reply

      • I’m sure neurotic power users will always take exception to not controlling every aspect of their sites, but your imaginings I think overstate the risk. You note that there is no way to test every scenario, so to that end regardless of your vigilance, there will be gaps. I just don’t see the downside to assuming WordPress has done their best to ward off the catastrophic events you describe. And Microsoft has made it’s share of update blunders. I believe the better idea is having backups and being able to step back if necessary.

        Create forum topic from comment
        Reply

        • “I believe the better idea is having backups and being able to step back if necessary.”

          And that is exactly the point. Don’t “automatically” update without allowing people the choice of whether or not to back up their sites first. Thus, automatic updates should have the option to be turned on or off in core. Not requiring a plugin or hack of the wp-config.php file.

          The resistance to allowing that one option is amazingly illogical.

          Create forum topic from comment
          Reply

          • Yes, as I was thinking through it, I realized that without the ability to step back, it does become a bit more risky. Nevertheless, as Miroslav points out, I am lazy, tough I prefer to think of it as having far better things to do and worry about. In the realm of risk/reward, for me at least, this is a no-brainer.

            Create forum topic from comment
            Reply

          • Core needs to include a hook that triggers backup plugins to do a backup before applying any patches for those who want to continue allowing automatic updates.

            As I’ve stated before, the fallout from this is that more and more and more admins are deciding to opt out of automatic updates. I’ve already disabled them on over 100 of the sites I manage. Most of the sites I run cannot afford to be down for 8 hours or a whole day (or more). And it’s become painfully clear that Automattic/WordPress Core can not be trusted (by me).

            Create forum topic from comment
            Reply

        • I never once said I don’t backup my sites and clients sites.

          I am sure TechCrunch backs up daily. Not all sites have that much content for a daily backup.

          I mostly (on my own sites) a weekly backup to DropBox.

          Nothing Neurotic about wanting to look at what the update contains (any new feature(s)) and doing the update yourself.

          Some people believe doing these things automatically equals lazyness.

          I have NEVER taken more than 2-3 hours to do an update. I read on twitter all the update information, then update on my test site, then voila update the actual site. Depending on the number of updates…it takes me less than 1 hour. Most of the time 20 minutes.

          Create forum topic from comment
          Reply

          • If you are doing a single site, great. If you are doing dozens or (in my case) hundreds, it’s a total pain in the rear – especially if the “fix” isn’t relevant for your installations. So, let’s say 200 sites, 10 minutes each, that’s 2000 minutes… or about a full week’s work. at 20 minutes, it’s a couple of weeks, and at the speed of WordPress updates, it’s almost turned into a fulltime job – and generally one that doesn’t pay the bills, because it’s maintenance work that generates little real new income.

            Create forum topic from comment
            Reply

  7. I’m having trouble understanding the instant decision of some to disable auto updates because of this release. The background update feature is extraordinarily successful for both core and plugins.

    It has undoubtedly prevented a significant number of site security exploits. I understand frustration over the small percentage of sites that took hit, but let’s not throw the baby out with the bathwater. Leaving sites vulnerable for the sake of convenience is a bad risk gamble, imo.

    There is a better way to resolve the kinks in system folks.

    Create forum topic from comment
    Reply

    • I envision a future where WordPress will automatically perform checks for compatibility issues when an update like this occurs, and when it finds one, will automatically notify the core team, the site owner, and the plugin/theme author. Then it will automatically assess the risks involved, and make the decision whether or not to roll-back to the last version or not.

      That would take a lot of work on the part of the core and plugin devs. But I think it would be totally worth it, and probably necessary to improve user confidence in the update process.

      Create forum topic from comment
      Reply

    • @John Teague, I am glad that this update did not break any of yours site. But I run sites that are very high traffic and rely on shortcodes for most of the content on the site. Those sites broke, they stopped displaying content. And has been pointed out elsewhere, there was next to ZERO chance that particular exploit could be used on our sites. So, the update was NOT critical enough to break them.

      I am THRILLED that the core team will be more forthcoming on posting the details regarding security updates as soon as they are released. It would have saved me 3 to 4 hours of figuring out the problem on my own and coding my own solution. But I think the decision itself to fix a mostly non-issue and deliberately break sites was wrong. And erodes my confidence in Core and the process.

      Create forum topic from comment
      Reply

      • Believe me when I say that in hosting and managing 200+ client sites we had more than our share that needed to be repaired. I didn’t mean to infer that there were not avoidable mistakes in the way this release was communicated and managed. I won’t debate the potential harm that was prevented. I’m just concerned that, with the scores of successful automatic updates behind us that produced virtually no problems, it’s important to look at the totality and keep things in proper perspective. I understand your frustration.

        Create forum topic from comment
        Reply

        • Proper perspective. What is that exactly?

          Each update has come with issues. None of the updates have had the impact this one had, but every one of the updates had sites which were taken down with white screen and various other issues which popped up.

          The fact that there has been a tolerance for those issues (because they weren’t as wide spread) doesn’t make those issues non existent.

          Yes, let’s do put things in their proper perspective.

          If you manage 200+ sites and you have certain SOPs then there is a likelihood some of the issues others have experienced won’t happen in your arena. However, because you don’t experience them does not mean they are not happening elsewhere.

          I manage a substantial amount of sites myself. Many of them are my own sites, many are not. I can handle issues arising on some sites because I use them to determine what issues I might have on other sites (test sites). So, letting automatic updates happen on some of those is what I want. That gives me something to gauge whether or not an update is going to possibly be an issue. If none of those sites (set up different ways FWIW) have issues then I don’t mind doing a quick backup to the sites I manage and doing the update.

          However, when there is an issue, I am very thankful I don’t have automatic updates on for the other sites. I have time to check into the issue(s), resolve them and then after there is a solution, I know what to expect and proceed appropriately.

          Now, I will reiterate, not one of the automatic updates has been flawless. NOT ONE. Each of them had issues. It’s the acceptable amount of sites the core developers and others making the “smart decisions” have figured into their “success” definition which makes the difference.

          There was boasting going on when WP 3.8.1 was automatically updated. They claimed “almost” 100% success rate. However, there was also an acknowledgement made that even though there is a successful update that just means that the update completed successfully.

          That does not mean the site functioned correctly after the update process was complete. Check WordPress support forums. You will find examples of every update failing somewhere.

          That is going to happen because of the thousands of different plugins and themes available. That also is because even WordPress Core Developers do not know all of the code enough to guarantee there isn’t going to be issues. The documentation is not complete (as was clearly acknowledged in the previous, recent post on this site about this latest release). There cannot be 100% success rate with 100% functionality after the update is complete unless there is a standard far above what is currently enforced throughout WordPress.

          Therefore, there should be the option in core to turn automatic updates “on” with a warning that doing so can result in damage to a site. Those who choose to turn it on, assume the risk. And that “risk” is due to not following the long established advice of professionals throughout the software world and WordPress, which can be found in their Codex here:

          https://codex.wordpress.org/WordPress_Backups

          Given the fact that WordPress does not announce these updates ahead of time, it is vitally important that the option to back up a site is available before the site is updated by WordPress. And it should be an option in core.

          That is perspective. And it is proper. ;-)

          Create forum topic from comment
          Reply

          • I really think you need to go to a WordCamp and sit in on a few sessions in the “Blogger” track and get to know the people that use WordPress in ways that are different from your own.

            Having an easy option to turn off automatic updates would be a huge mistake. If you have the gumption and means to turn off automatic updates, then you’re likely someone who is capable and willing to manage and install updates in a timely manner yourself and take responsibility for what may happen to your site.

            Making the option a checkbox allows people to make that decision without understanding the implications of what they’re doing.

            Create forum topic from comment
            Reply

  8. Big picture here for a second. Security should be a priority above all. So even a control freak like me has to accept the automated updates. That said though, (and off the subject a bit), even if they make the core 100% secure, which is impossible historically speaking, the whole WordPress system is set up to be exploited by other means.

    You can secure the core all you want, but still you can have huge security holes in your site thanks to Themes and plugins. If you look at themecheck.org, there are countless themes with security warnings, and nobody has a similar tool for plugins.

    Until the WP core is changed so that only “safe” themes and plugins can be activated, all the talk of securing the core is almost meaningless. You are patching one of the three possible “danger” areas. You have to secure all three of them, and even that is not 100% effective, since that leaves the actual server, which in most cases we are at the mercy of the hosting companies, but at least you will know for sure that your WordPress installation is secure.

    Why is it that nobody is talking about third party Theme and Plugin security? Is it because the developers are busy way more important things, like distraction free writing, and placing menus in the customizer?

    Create forum topic from comment
    Reply

  9. Great interview with lots of good information shared. Also great because it helped to humanize a guy from the shadowy core-development team, and (at least for me) helped to dispel some of my impressions that they are all a bit too arrogant. Pendergast sounded like a nice guy, not trying to cover up anything, admitted to some fault, and changed my opinion about a few things. Thanks to both of you.

    Create forum topic from comment
    Reply

  10. As always, I’ll just keep saying that users need options and control, obviously if they also take responsibility for the consequences of their choices.
    Automatic minor updates on by default? All right, less savvy users will likely welcome that. But have a setting in core to disable this behavior for those who wish it. And, alternately, in the same place, also one to also have automatic major updates turned on for those courageous enough for that.
    Site gets hacked as a result of having updates off and not applying one that patched the vulnerability used in the hack, it’s the user’s fault. Site has any sort of issue, including even a brief downtime at a wrong moment while the update itself takes place even if everything otherwise goes well, not to mention something more serious as it happened now, due to updates being automatic, it’s WordPress’ fault. Simple.

    It’s a piece of software specifically aimed at being used by many millions of people, it powers a large chunk of the entire web, the fact that all those users have wildly differing needs and preferences and skill levels must be at the front of the developers’ minds and the design policies must revolve around that, not relying on plugin developers to always fill in gaps, and at times even trying to discourage them from doing so when it comes to certain features.
    And security may or may not trump all. That’s for the user and, depending on the severity of the risk, the hosting provider to decide.

    Create forum topic from comment
    Reply

  11. “Making the option a checkbox allows people to make that decision without understanding the implications of what they’re doing.”

    And that implication is exactly why users should be warned about turning automatic updates “ON”.

    Automatic updates being forced without backups (which, if you actually read what I wrote you would know) is a dangerous practice. And WordPress itself has told people that for years. Now, it’s dangerous to back up your site before updating?

    Hardly.

    I don’t care to go to a WordCamp for an indoctrination into your way of thinking. I have spent 38 years working with computers, I understand the risks of updating without backups. What, you are trying to get me to join your religion? LOL

    Nice try though.

    BTW, how many ways do I use WordPress? And how many ways do my clients use WordPress?

    If you presume to know, you are way over your limits.

    There is NEVER a good point to be made for not backing up a website before updating it, unless it is a new install with nothing on it or a stagnate website which hasn’t been used between updates. And even then, if something does go wrong, you will have to do a new install if you haven’t done a backup. Unless the person involved is familiar with db maintenance and getting past a failed update.

    BEST practice, back up a website before updating. PERIOD!

    Create forum topic from comment
    Reply

  14. In the end, a rushed patch to fix a security issue in the end hurts many user`s sites. Combined with automatic updates, this conspired to create a lot of collateral damage, perhaps more than the initial security hole may have in the same time period.

    Making a significant coding and feature change in order to fix a security hole is always a challenge. It does point out the issues of automatic updates being very binary (yes or no, not much subtle adjustment), I understand the staff reasoning for what they did, but at the same time, I feel for the individuals, companies, and groups who woke up to horribly broken sites that could not be fixed as quickly as they were hurt. More so I guess because the nature of the security update is mostly to do with logged in users, and many corporate and private blogs do not allow user logins or submissions. They are likely users of shortcodes, and likely got most of the harm and little benefit.

    Perhaps it`s time to start considering dependencies before applying an update.

    I don`t use automatic updates for these reasons. It also keeps me from ending up on the x.0 versions that generally have a x.1 update within days.

    Create forum topic from comment
    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

You may also like

Newsletter

Subscribe Via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Discover more from WP Tavern

Subscribe now to keep reading and get access to the full archive.

Continue reading