WordPress 6.4.1 Fixes a Critical cURL/Requests Bug

WordPress contributors have worked quickly over the past 24 hours to prepare a 6.4.1 maintenance release after a critical bug emerged from a change in the Requests library, causing problems with updates on servers running older versions of cURL.

Hosting companies began reporting widespread impact of the bug. Tom Sommer, from one of Denmark’s largest hosting companies, filed a GitHub issue outlining how the cURL timeouts were affecting sites:

  • #657 breaks downloads towards https://api.wordpress.org/ and many other sites when using Curl 7.29.0 (and perhaps other versions)
  • Error: RuntimeException: Failed to get url 'https://api.wordpress.org/core/version-check/1.7/?locale=en_US': cURL error 28: Operation timed out after 10000 milliseconds with 807 out of -1 bytes received.
  • It also causes issues with the REST API in Site Health with the error: REST API response: (http_request_failed) cURL error 28: Operation timed out after 10005 milliseconds with XXX out of XXX bytes received”
  • It also prevents WordPress plugin and core updates, basically anything that relies on the internal Curl handler in WordPress.

The issue became a top priority as it wasn’t clear how it would be possible for users to receive an update.

“Even if you fix this now the issue prevents any future auto-upgrade to a 6.4.1, since it breaks Curl requests, so the only way for people to update would be manually,” Sommer said. “The longer you wait, the bigger the problem will become.”

Nexcess reported tens of thousands of sites being affected by the bug. The issue was beyond what most users would be able to manually patch on their own, relegating hosts to figure out how to update their customers.

“All my websites locked after updating to WordPress 6.4,” Javier Martín González reported. “The ones without updates are working normally.”

The bug was also reported to be causing causing potential Stripe API, WP-Admin, and performance issues.

Liquid Web/Nexcess product manager Tiffany Bridge summarized how this problem emerged:

It looks like:

  • Someone reported a bug having to do with an interaction between his Intrusion Protection System and WordPress
  • They then submitted their own patch to WordPress
  • The project lead for that area asked the submitter to write tests, which he did not do
  • Then they merged the PR anyway, despite the lack of tests
  • Meanwhile hosts are all going to have to revert that change ourselves on our own fleets so that our customers can still have little things like core and plugin updates if we are running an affected cURL version. (7.29 confirmed, there may be others)

WordPress core contributors will have to get to the bottom of how this bug was allowed through, via a postmortem or other discussion to prevent this from happening on such a large scale in the future.

WordPress 6.4.1 updates the Requests library from version 2.0.8 to 2.0.9. as a hotfix release to mitigate the issue. It reverts the problematic change. Version 6.4.1 also includes fixes for three other separate issues. Automatic updates shipped out this evening for anyone with sites that support automatic background updates.

11

11 responses to “WordPress 6.4.1 Fixes a Critical cURL/Requests Bug”

  1. 6.4 completely locked me out of the admin dashboard, it didn’t even through a PHP error, just timed out. I ended up downloading 6.3.2 and overwriting wp-admin/, wp-includes/ and the root .php files with it. It did a database migration and then all was working again.

  2. It is interesting that Tom and Nexcess seem to be saying that thousands of their users have been affected because it’s all core’s fault…

    But I haven’t seen anything but cURL 7.29 affected – and that version is 10 years old and riddled with security issues. It is also notable that anyone who still hasn’t moved from centOS 7 (EOL june 2023) is likely still running this version, because that project is dead.

    So at this moment, it does look like anyone affected has a site on hosting that is likely running outdate EOL OS and ancient versions of cURL.

    That is not to say WP Core didn’t handle this badly, but the Nexcess lady seems to be deflecting and not taking any responsibility for their stack.

    With regards to her reported sequence of events, I would also disagree.

    I don’t think missing tests are the real issue, the real issue as far as core is, the ticket should have been closed immediately as a won’t fix. This is an upstream bug and should have been reported upstream and not been handled in core… IMO

    I mean should you write tests to test every single versions of cURL going back 10 years? Why on earth is anyone still running 10 year old cURL?

    Its a mess, but not just with an update to core and how bugs are handled, it exposes a lot of hosts.

    • Hi. “Nexcess Lady” here.

      It’s really important to understand how enterprise-level hosting works. Hosts charged with delivering enterprise-level environments frequently run older versions of Linux intentionally because they are stable and predictable. But that means that they often have older versions of packages like cURL on them.

      However, the one of the benefits of Enterprise Linux distros is, among other things, active support for and backported patches to those packages. Just because a package version is no longer being patched by its own maintainers doesn’t mean it isn’t being patched at all. Just some context for folks who aren’t aware of how Enterprise Linux works. It’s a lot easier (and better for customers) to mitigate things like older packages needing patches than it is to mitigate the instability inherent to the latest-and-greatest.

      While my colleagues at other hosts aren’t calling attention to themselves in this situation, a great many of them are in exactly the same position, and intentionally so.

  3. Hi Tiffany,

    A couple of things first

    It’s really important to understand how enterprise-level hosting works.
    Just some context for folks who aren’t aware of how Enterprise Linux works.

    This sounds like we are going to descend into an appeal to authority, certainly has sniff of it…

    Many of us are also well acquainted with the technical aspects behind this level of hosting, and some of us even provide environments that regularly receive Top-Tier in the Review Signal WP Hosting Benchmarks.

    But since you chose to focus on this, let us expand that a little, to give a wider context.

    Enterprise level hosting plans are also the biggest ticket hosting plans that a hosting company sells. Anyone who works within this industry also understands that it is also the highest margin plan. Let’s keep that in mind as we unpack your reply.

    Prior to this we weren’t really focusing on a single sector of (Enterprise), but we can because that just leads to some more interesting questions.

    It’s a lot easier (and better for customers) to mitigate things like older packages needing patches than it is to mitigate the instability inherent to the latest-and-greatest.

    This is a fantastic combination of a straw man and a false dichotomy, outstanding effort!

    No one spoke of the latest and greatest.

    CentOS 7 and RHEL 7 are in their dying throws.

    But since you speak of Enterprise Linux RHEL 8 was released over 4 years ago, with 8.1 (4 years) and 8.2 (3.5) released shortly after. Since then there have been multiple multiple versions…

    And RHEL 9 was released a year and a half ago with 9.1 a year ago, and two more releases since then.

    However, the one of the benefits of Enterprise Linux distros is, among other things, active support for and backported patches to those packages. Just because a package version is no longer being patched by its own maintainers doesn’t mean it isn’t being patched at all

    From the bug ticket, it was not just cURL 7.29 that was affected, all versions prior to 7.45 were (any version that did not default to http2).

    cURL 7.45 is still ancient, but what is of note is that 7.29 is the package that CentOS and RHEL7 come with, and yes these are long term support environments, but they are so close to complete EOL, and even though you say they are still supported with patches we can see that they are not really/fully supported.

    Now with CentOS 7, and the people hosting on this non-enterprise Linux distro. In may ways I do understand their plight… They were dumped on by RedHat from a great height, they have nowhere to go… for those unaware the project was killed, and the follow up version centOS 8 actually EOLed in December 2021 almost 2 years ago. They have a tough choice, where to go, what to do, especially those using CloudLinux, given its currently in Beta.

    However, it is 3 years since CentOS accounced they would discontinue CentOS… three years is a long time to get yourself together and get a plan in action, and leaving it to the last 6-8 months is a poor management.

    For those smaller outfits, its especially tough… but for massive hosting companies with entire teams, massive hosting companies that sell… checks notes… Enterprise Level high margin plans… it could be considered a dereliction of duty. CentOS supposedly is receiving updates, but we can see from the comment below that the cURL package for CentOS has had 0 updates/patches in the past 2 years, despite multiple vulnerabilities since then.

    But you spoke of Enterprise Linux, so that is not your point.

    RHEL7 (Enterprise Linux) and CentOS (Community Linux) have a shared codebase, but maybe things are different for RHEL 7 – the common enterprise level hosting environment that may be using this version of cURL 7.29?

    Here are some of the CVE’s that came up in a quick search for RHEL 7 cURL 7.29 that they have deemed out of support scope or won’t fix in maintenance phase 2:

    https://www.cve.org/CVERecord?id=CVE-2023-27535
    https://access.redhat.com/security/cve/cve-2023-27535

    https://www.cve.org/CVERecord?id=CVE-2022-35252
    https://access.redhat.com/security/cve/cve-2022-35252

    https://www.cve.org/CVERecord?id=CVE-2022-43552
    https://access.redhat.com/security/cve/cve-2022-43552

    I stopped there, but I am sure there are more for many more packages.

    And here are their words re this phase of support:

    “During the … Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7, Red Hat defined Critical and Importantix impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) will be released as they become available.”

    So they define what is Critial and Important, and only provide backports for those fixes.

    Now people can argue about severity, but even medium severity flaws can by their own admission be escalated to critical, and at scale the chances of this happening increase with scale.

    Furthermore this clearly shows that they do not want to patch things unless they absolutely have to… they want you to upgrade to RHEL 8/9… because… profit (of course).

    So when you speak of these things being maintained and things being backported, either that is very selective or are you suggesting to have people on your team doing this? Because as we know this is very difficult, and many hosting panels are unable to support servers with packages from outside the distro repositories.

    But if companies have the resources to fill the gap where RHEL is failing them, then surely they could have migrated to a more modern version of RHEL, there are several available that are stable and not “the latest and greatest”?

    So the question has to be asked, with regards to these enterprise level environments where there was a newer supported OS, why have these companies not migrated sites from RHEL 7 servers to RHEL 8.x or 9.x servers?

    Everyone should remember, these are the most profitable hosting plans with the highest margins, those that the providers swell in interviews about providing resources for dynamic workloads, yet within a ten year cycle you can’t migrate them from a nearing end of life poorly maintained Enterprise Linux Distro to an available and more modern and currently maintained distro within an appropriate timeframe?

    Maybe there is an excuse for your big box low margin hosting, I mean what can people expect for your $5 a month plan?

    But your $273 a month plan, surely that pays for a simple migration?

    Why not? It’s only a management problem, done properly there should be little downtime to users.

    Perhaps the answer is simple?

    Profit, hitting targets, milking customers, and keeping investors happy.

    • Anonymous poster who pretends to understand enterprise hosting, but completely misses the mark.

      So, since you seem like you know everything there is to know, and think migrations at scale are “simple”. How do you handle the migration of 100K+ sites, to a completely new operating system, with zero downtime and no IP changes? (There is an answer to this, but its not by any means “simple” or “fast”).

      So who exactly do you work for? You’re being way too condescending towards Tiffany, who isnt afraid to hide her name/affiliation to just be another “concerned party of interest”. I’m guessing you either A) Work for a tiny hosting shop and has never seen any type of scale, or B) Are just a troll with a point to prove. So which is it?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Newsletter

Subscribe Via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

%d