WebP by Default on Hold for 6.1 After New Objections From WordPress Lead Developers

Last week Performance team contributors were working on refining their follow-up patches for the multi-mime/WebP feature, after the main work for it was merged into core for 6.1 at the end of July. This includes smaller related items like the shim for non-supporting browsers and adding PDF support, which are being handled in separate tickets.

The proposal to generate WebP images by default for new JPEG image uploads has been controversial from the beginning. While the Google-sponsored contributors driving the project have made some revisions after an initial round of significant critical feedback, other contributors have continued to voice concerns that they said were not being considered. Several contributors reported issues with the feature and suggested that it should start by being opt-in, an idea that was summarily dismissed before the main work was committed.

Last week, WordPress lead developer Andrew Ozz weighed in on the ticket with new objections:

Like @MatthiasReinholz@eatingrules, and others I think this approach is perhaps lacking. Why would there be twice as many image files taking up a lot of extra space when half of them will never ever be used anywhere?

IMHO a better approach would be to just make all image sub-sizes WEBP. If JPEGs are indeed needed, these can be generated on-the-fly as needed. There is no point of clogging the web servers storage with all these useless files.

On the other hand, if the WEBP file sizes are actually larger than the JPEGs, that would probably mean that better tools are needed, and this patch is premature.

Six weeks ago, in response to one complaint that the “resources for conversion would be tremendous,” Google-sponsored core committer Adam Silverstein confirmed that resources for generating the images on upload would “increase dramatically.”

“However resources to serve an image will be lowered,” Silverstein said. “Since image uploading is very rare compared to image serving, the extra effort to compress and store images should be worth it.”

This is another problem Ozz cited in his objection to this approach.

“Actually that dramatic increase of resources usage when uploading an image is a very bad side effect here,” Ozz said. “It means a lot of uploads will fail, and leave the users stranded. It also will increase support requests for both WordPress and the hosting companies dramatically. Don’t think this is acceptable. Because of this, even if image multi-mime support is needed in WordPress, the current approach doesn’t seem like a good solution.”

Approximately 24 hours later, Google-sponsored contributor Felix Arntz commented to advise that the WebP fallback mechanism to JPEG for older browsers was ready for commit and that he planned to commit it in a few days.

“Please do not commit any more code here unless it is to address the dramatic increase of resources needed to create image sub-sizes after upload,” Ozz said. “As I said above such increase is unacceptable.

“Is there any data about how much more resources (memory, processing time, etc.) are needed when uploading different image sizes? Any estimate about how many sites may be affected by that? Any suggestions on how to deal with it? Do you know what happens when post-processing of an uploaded image fails?

“Frankly for now it seems that this patch will have to be reverted and refactored in order to address this.”

Adam Silverstein responded to his concerns with clarification on why they chose the current approach, in anticipation of certain edge cases and eventually adding support for formats like AVIF once it is more widely supported:

I tend to agree with your assessment that all sub-sizes should be generated as WebP only, that was the shape of the original proposal. For the vast majority of use cases/users this will work the best. I would be open to considering this for the default (with some mitigations, see below).

The reason we decided to generate both formats was for backwards compatibility considerations for the few edge cases we identified where WebP images will might not work: namely emailed images (some older outlook/windows clients), Open Graph tags (some services don’t support WebP) and older Safari browsers. One possibility we considered would be to keep only the full sized JPEG so it is always available for those edge cases.

The “multi-mime” support here is built to generating multiple formats so you sites can provide a primary and fallback format with something like the picture element. This is less important for WebP since it is so widely supported, but would be helpful for adopting newer formats like AVIF by plugins or core.

Silverstein also said the option of generating images on the fly was something they need to figure out but “felt out of scope for this effort.”

In response to the complaint regarding the dramatic increase in resources for image uploads, Silverstein said they are relying on the “retry” mechanism to mitigate this problem.

“This change also doubles the number of times WordPress will retry’ the image regeneration, so while processing time will increase, I don’t think we’ll see a jump in failures necessarily,” he said. “I know we had trouble adding new sizes in the past, however that was before we added the retry mechanism.”

The team behind the WebP by default project is more focused on serving smaller image sizes on the frontend and considers the additional resource usage on upload a necessary sacrifice for WordPress users.

“The additional resources at upload time needs to be weighed against the reduced resources of serving the smaller WebP image, especially since serving typically occurs several orders of magnitude more frequently than uploading,” Silverstein said.

“If uploading fails after all the retries, the user has the same experience as currently: they are left with a broken, unusable image. That can probably be fixed, although I don’t think this change will increases failure rates.”

WordPress lead developer Dion Hulse also commented on the ticket to report issues with WebP conversions in the WordPress Photo Directory:

Just noting here, that these additional webp conversions appear to have been the leading cause of upload failures on the WordPress Photo Directory in recent weeks. See #meta6142 and tickets closed as duplicate of it.

The errors were generally along the lines of Allowed memory size of 256M bytes exhausted (tried to allocate 90M bytes (obviously with byte values) while attempting to perform the initial full-size-original jpeg -> webp conversion.

It hasn’t affected every upload, only that of certain images. Potentially related to the $quality value being passed for webp requests (IIRC the default of 82 was optimized for jpeg?).

Hulse disabled the JPEG to WebP conversion as a result of these errors, as the photo directory doesn’t currently use WebP but noted that it “might be a sign that it might be worth considering only generating webp’s for the resized images, rather than for the original file too.”

Silverstein said they are investigating the issues Hulse reported, as it may have exposed a bug.

Ozz circled back to recommend that making sub-sizes on demand would be a better approach that has faster processing of uploaded images and reduced space requirements, since the additional JPEG images would not be generated unless needed. He also noted that the “retry” for image post-processing “doesn’t work as well as expected.”

“The bad news is that if the post processing fails, chances are the originally uploaded file will remain,” Ozz said. “Then it will be used everywhere as most of the code in WP falls back to available sizes, and the only size will be the original. That means we will be serving huge (4MB – 8MB average) images. A serious drawback.”

Silverstein responded to Ozz’s suggestions, agreeing with many, and proposed two potential paths forward for the project:

  1. Keep the current multi-mime infrastructure, but change the defaults so only WebP files are generated, possibly up to a threshold size beyond which only JPEGs would be generated. Most existing work would continue; content filtering could possibly be removed.
  2. Revert the multi-mime infrastructure and switch back to a single mime approach, using WebP for images up to a threshold size and adjusting the compatibility layer to use the JPEGs we keep.

The Performance team is doing more research and has temporarily paused committing anything else until they receive feedback on the next direction for the project.

30

30 responses to “WebP by Default on Hold for 6.1 After New Objections From WordPress Lead Developers”

  1. I would love to see ALL image sizes get generated on the fly as needed. Pre-generating all the images creates so many extra files that never get used.

    • Agreed, all the extra generated images is crazy. And it sounds like core is concerned about that in this instance. But there are problem with OTF processing so not sure I’d vote for that.

      What I can’t believe is a solution like Delicious Brains developed and promoted for inclusion in WP core wasn’t considered. And they did this five years ago! https://deliciousbrains.com/introducing-wp-image-processing-queue/

      To me that seems the ideal solution. Images are generated and stored on the server but ONLY after they are needed. I don’t need a banner size crop of every single image I upload for example. 🙂

      • This would be the worst solution I can imagine.

        Server storage is cheap. CPU is not, so why would I add additional load to my webserver (especially the PHP daemon) for dealing this, if instead it can just do a very cheap operation (serve static file).

        • It’s not “additional [cpu] load”, it’s actually less. Deferred OTF image generation means you’re only generating thumbnails one at a time, not a dozen at a time which means a shorter cpu usage spike affecting other processes. And, only generating the thumbnails that are actually needed means a net of less cpu usage as well.

          There are important concerns with what happens in that approach with failures and retires, but those are not really any different than the concerns voiced by Ozz with failures with the current approach. One wouldn’t just keep retrying generating a thumb repeatedly, it’d quickly give up, ideally with error logging available to a lay user that the current approach doesn’t have.

          While file storage is somewhat cheap, it’s not always cheap. It’s why we do often end up offloading storage on large sites to things like S3, at scale storage costs do add up. Case and point just had to upgrade a site on Nexcess from $100/m to $250/m solely due to ticking over the 100GB mark.

          Finally, dealing with large numbers of files also has a performance impact. You don’t see that at 10-20,000 files in a directory, but you do at 100,000 in a directory. Simply doubling the number of thumbnails on disk will impact impact performance on large sites.

  2. Why the push for WebP? Does it REALLY save that much bandwidth? Unless it can be shown that WebP has SUBSTANTIAL advantages over other image formats, then let the site owner decide what format they want to use.

    Finally, what about photographers and artists that want their images in a specific format for a good reason? Are you going to force them to use WebP?

    This seems like a really ill-conceived idea that appears to be pushed by contributors with a vested interest in seeing WebP adoption increase (i.e., Google). It isn’t the place of WordPress to push adoption of a proprietary image format of one of the world’s largest corporations.

    • For me, png to webp saved me A LOT of sizes of the files. However JPG to webp was not good experience.

      I have close to 15,000,000 photos on one site, another 25,000,000.

      Yes, even 10 kb can add up

  3. Squarespace and Wix already do WebP conversion by default. Most modern browsers support WebP, older versions of IE do not. Hopefully, the development team will come to a good solution in time.

  4. This is an active discussion, and the feature is not on hold.

    Exactly what it’ll look like / what will ship in 6.1 is still being discussed, and the ticket is a great way to keep up to date!

    I’m glad that we’re working towards consensus, and making the feature the best that it can be.

    • I was thinking it was on hold because Ozz said, “Frankly for now it seems that this patch will have to be reverted and refactored in order to address this.” and it wasn’t yet clear if there would be blockers that would cause it to miss the deadline for 6.1.

      • Thanks for the reply!

        This thread from the media chat + comment from Ozz is what I was going with at the time “Yea, looks like WebP will be in 6.1 in any case (unless some -huge- problems/regressions surface now, but don’t think that will happen)”https://wordpress.slack.com/archives/C02SX62S6/p1661377197521529?thread_ts=1661350515.706809&cid=C02SX62S6

        I decided to say something because I’m a bit afraid that folks will disengage from the development and testing of the feature if they think it is “on hold.”

        I also don’t want folks to assume it’s not shipping, then be surprised/upset that it does.

        Development and discussion/consensus building is actively happening, and I hope that folks will continue to both test and provide feedback.

        I’m really glad to see that there has been so much discussion about this particular feature. The approach has changed and improved so much since the original proposal!

  5. Maybe it’s time to think about on-the-fly processing of images, too. It’s 2022 and WP might be the only platform that still clutters its upload folder with images at sizes that are never used, with the waste of disk space and server processing that this entails.

  6. The image rendering speed of Webp is faster than the jpg and also size matters here. However, issues for old browsers are still out there as many of them don’t support the webp extensions. Let’s see how it works in the future. Thanks for the update.

  7. Do you remember the phpThumb() library? It was ridden with vulnerabilities, and most of the ThemeForest themes used it. But the idea behind it was not bad, was it? Generating images on the fly, and then saving them in a cache folder.

    Could this be done with WebP? I think so. Resource -wise, I think it’s the best solution, if implemented in a smart and efficient way.

    • Of course but “Deferred On The Fly” generation is very different than TimThumb/phpThumb.

      For one, you could send any pixel dim you wanted to phpThumb/TimThumb and it’d attempt to generate it via a public API.

      It was, for a while, an easy way to DOS a site by asking it to generate 1000×1000, 1000×1001, 1000×1002, etc… images endlessly. Even though each one got cached you could ask for infinite image sizes.

      WP however would continue to have predefined thumbnail sizes and not open to requests for other sizes from the front end.

      Deferred OTF – would reduce CPU usage AND file storage. It really does make a lot of sense from a performance benefit point of view.

      Folks promoting this being “on by default” because that’d have the maximum benefit to existing sites, seem to ignore that face that most older large sites have image URLs hard coded in the content. They will not magically start serving the WebP version for old content just because WP has generated it.

      Almost no one objecting to this is objecting to WebP generation in core, or multi-mime. It’s solely an issue with the implementation that is on by default on existing sites and lacks a simple user facing option on the media page to enable/disable it because it has the potential to take down sites unexpectedly. The example the WordPress Photo Directory immediately starting to throw fatal errors and failing image uploads ought to give everyone pause. https://core.trac.wordpress.org/ticket/55443#comment:123

      If it’s a problem on that infrastructure… yikes.

  8. I am happy with converting to webp outside the WordPress then upload it just as webp.

    But webp is just another dominant of google, and hope webp will not have fate as amp where it looks like they’re about to leave it.

    What i am worry is if in the future maybe they will have webp2 which is more quality but more light and we will all be busy changing webp to webp2

  9. I’m all for supporting WebP (and SVG) in core. But I feel that any sort of automated file conversion should be opt-in. A simple checkbox could save a lot of hassle, IMO.

    But I’m happy that they’ve put this on hold. There’s nothing wrong with taking time to get the best possible solution. It’s a “nice-to-have” feature – not an emergency.

  10. I’m glad to see this slowing down, and hopefully pivoting. Moving to an “on first use” thumbnail generation is really the fundamental solution to all these problems. Until then, everything related to new image formats will just exacerbate the existing issues.

    Many of us have raised objections to the “WebP by Default” plans of the Performance Team… While some objections have been addressed, many others have been flat-out ignored or brushed off.

    https://wptavern.com/wordpress-performance-team-puts-controversial-webp-by-default-proposal-on-hold-after-critical-feedbackhttps://wptavern.com/wordpress-performance-team-revises-proposal-for-webp-by-defaulthttps://wptavern.com/webp-by-default-merged-into-core-for-wordpress-6-1

    On a near-daily basis, we have to deal with our clients’ sites running out of disk space, or backups failing, because they have too many files on their servers. Adding piles of extra WebP images, generated in the background, completely invisibly to most site owners, is going to make things significantly worse over time. (Having only a filter to opt-out is woefully insufficient; most site owners don’t even know what a filter is!)

    And if this does pick up again as proposed, it really needs to be opt-in, not opt-out. A setting in the media library — off by default — would give users control over their sites and allow everyone to move forward.

  11. It shocks me that nobody is talking about Safari. Safari does not support WebP except Big Sur. 80% of mac users have no Webp in their native browser Safari. That is normally fine because Safari will instead load the JPEG version of the image. Except when that doesn’t work due to an obscure bug.

    I’ve seen this happen on multiple clients websites it only affects around 1% of images but once it happens you basically have to restore every image to JPEG and update all paths / URLs to make sure images will load. JPEG images work fine, I’d prefer to stop generating countless image files and have a simple opt in for WebP.

    • Agreed, this has been missing from the discussions I’ve seen. Worse, it seems there’s currently no reliable data for macOS usage due to Apple blurring Catalina with Big Sur: https://bugs.webkit.org/show_bug.cgi?id=216593. Reading through the Trac ticket it feels like there’s a number of hand-wavy dismissals of potential issues, though this Safari one seems the most problematic.

      Anyone doing decent browser testing must be encountering this, so it’s hard to understand how this hasn’t been considered a blocker.

  12. I still think this option should be an opt-in instead of being default and force people to increase disk space tremendously. Many sites already have a image delivery/optimization system like Cloudflare, imagekit, optimole etc. That takes care of image optimization and WebP. So sites that already have such flow in place, they don’t want any WebP on disk for good reason.

    But on the fly image generation even for JPEG would be rad. But I’m not too optimistic about it though.

    I just hope no matter what the performance team decide to do, they make this an opt-in system and not default.

  13. The question I have not yet seen addressed is: since when is Google allowed to further its ambitions for world domination in WordPress over the objections of the community? Seems like a dangerous precedent to me.

  14. Quite honestly there are so many developers out there that just crunch a website together and do nothing about different image sizes, especially when a theme or plugin introduces them that I always found that there is a hell of a bloat regarding images. If the developers are so concerned about useless files, why don’t they tackle that issue that has been there since the beginning first? Like mentioned, processing images on the fly when they are needed to size that is actually needed is something that exists in other CMS for a long time. Webp is for now the new standard regarding core web vitals and as such it should be considered to be a standard format without having to install 5 other plugins. You could even go that far and just convert it when it’s uploading, discard the jpg and just store the webp.

  15. This article describes a typical iterative development workflow, except the premature merge before all acceptance criteria were met. Give it another iteration.

  16. Deferred thumbnail generation would be a huge improvement and would put an argument of bloated uploads dir on the sidelines for webp co-generation. I would happily take a penalty of loading full image util the thumbnail is ready just to have that benefit but further optimisation could be made to always create a 1024×1024 (or other size) to be used when a small thumbnail is requested and its not yet available to make this less of an issue.

    Over the years I’ve tried bunch of band aid fixes for image problem, ngx_pagepseed worked great for some time until I ran into issues with latest nginx versions. I’ve tried Cloudflare, but ran into issues where webp images are served to iPhones and those images were not displayed/rendered.

    Im wondering if willnorris/imageproxy + CF + some kind of css/js based webp loader would address the needs, but then again it’s a lot of effort for something that might end up working out of the box in WordPress, so doing and undoing everything seems like a lot of work with little benefit.

  17. Something that perhaps hasn’t been discussed, is some sites archive their photos into WordPress. Later, if the photo is needed elsewhere, they download the photo from their WordPress photo archive. Perhaps they want to download the original files; not WebP formats.

  18. Why so much emphasis on WebP when there are myriad other things that need to be fixed? They’ve introduced so many useless features like Duotone, that it’s time to make some real changes that’ll help real users.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Newsletter

Subscribe Via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

%d bloggers like this: