Developers at Yoast and Google Collaborate on Proposal to Add XML Sitemaps to WordPress Core

Developers at Google and Yoast are collaborating with other contributors on a proposal to add XML sitemaps to WordPress core. This capability has traditionally been handled by plugins, which provide a variety of implementations. The goal of this new feature project is to ship basic XML sitemaps in core while providing an XML Sitemaps API for plugin developers to extend. It would also update WordPress’ robots.txt file to reference the sitemap index.

Thierry Muller, a Developer Relations Program Manager at Google, published the details of the collaborators’ plans on WordPress.org. The sitemaps included in core would be enabled by default and would index the following content:

Homepage

Posts page

Core Post Types (Pages and Posts)

Custom Post Types

Core Taxonomies (Tags and Categories)

Custom Taxonomies

Users (Authors)

Although WordPress contributors have had multiple discussions regarding XML sitemaps on trac, Muller said there had not yet been a group to take ownership and get things moving forward. His proposal notes that 4 out of the top 15 plugins in the WordPress plugin directory ship with their own implementation of XML. Despite the demonstrated high demand for XML sitemaps, the right team for getting this added to core had not been assembled until now.

“XML Sitemaps is a standard across all search engines these days and any website with content which is intended to be indexed should have one,” Muller said. “Having XML Sitemaps in core would speed up content discoverability and indexing. We started discussing this topic last year with Yoast and agreed that it would be great to finally make it happen.”

So far the initiative has received a favorable response from the community and has already gained the support of WordPress project lead Matt Mullenweg.

“This makes a lot of sense, looking forward to seeing the v1 of this in core and for it to evolve in future releases and cement WordPress’ well-deserved reputation of being the best CMS for SEO,” Mullenweg commented on the proposal.

Performance is one of the primary concerns that the WordPress development community has expressed regarding adding XML sitemaps to core.

“As simple and straightforward as XML sitemaps are, they present some relatively significant performance challenges at scale,” Matthew Boynes, partner at Alley Interactive, said. “As one for-instance, how many urls are going to be in each paginated (sub-) sitemap? A sitemap index file is limited to 50,000 sitemaps, so even though each sitemap is limited to a maximum of 50,000 urls, generating 50,000 urls in one page request would be extremely difficult and non-performant to do on-the-fly.”

Muller and other contributors on the project have a strategy for addressing scalability concerns and are planning to keep these considerations on the forefront while building core’s implementation.

“As for everything else, engineering new features should always be done with security and performance in mind,” Muller said. “It starts at the high level architecture down to the granular details. Each Sitemap entry will only contain , and which should not overload the server request with a paginated approach. The number of entries per Sitemap is still to be defined, performance will definitely be a deciding factor.”

The XML Sitemaps project has a clearly defined scope. The initial implementation will not include features like image, video, or news sitemaps. It also will not include a UI for controls to exclude individual posts, pages, or custom post types from the sitemap. This is somewhat controversial but plugins can always extend core to offer these features until a more sophisticated version of sitemaps lands in core.

Plugin authors who offer XML sitemaps will need to re-architect their plugins with the new API in order to avoid conflicts and indexing errors.

“I can confidently say that we will make it as smooth as possible and work with plugin authors to avoid conflicts,” Muller said. “If anything else, having XML Sitemaps as well as a Core API to extend it will make their lives easier – plugin authors will have a standard way to extend it.”

This effort to get sitemaps added to WordPress core is not limited to those working at Google and Yoast. Muller encouraged authors of other XML sitemaps plugins to get involved or at least follow the project’s development to ensure their plugins are compatible once it is merged into core.