In reply to David Anderson.

Hi David,

I'm one of main developers on the Jetpack Elasticsearch systems. I've also worked a bit on search systems on WordPress.org (haven't really worked on Core yet). With that background I can give you my opinion.

But “more relevant results?” and “speed improvements” ? Those are arguable fixes for core defects., not new features.

This is a reasonable question you're asking and one I've asked myself. That said, doing something in Jetpack doesn't prevent it from happening in Core and maybe helps us learn what Core should do. From what I remember, the last big Core change to search was in 2013 when switching to relevance sorting: https://core.trac.wordpress.org/ticket/7394

I think one of the problems there was that they couldn't do as much as they would like because it caused performance problems on a lot of hosts. Jetpack of course gets around the performance part of the problem by offloading queries. The MySQL performance on a lot of hosts is so bad that doing a remote request (potentially across continents!) is significantly faster than querying the local DB.

On the relevance problem, MySQL has historically not been very good at search. I remember a good analysis a number of years ago (which
I can't find right now) comparing MySQL full text search and Lucene search (Elasticsearch is basically a wrapper of Lucene). By just switching over to Lucene (no extra tuning) the accuracy of the first result increased by 4x (precision@1 is the technical measurement).

Both of these are old-ish, anecdotal data points, and technology has improved, maybe Core could take another crack at it. Gutenberg certainly introduces some challenges for search because blocks are stored in post_content, so may increase the need for making changes. To me it feels like another table would be needed where content gets mirrored and somewhat denormalized. That is what we are doing for Elasticsearch indexing, and maybe that could be a model for Core to use. Maybe MySQL has also improved and needs some new investigation. Any such change would be challenging to support across all hosts though.

I think search isn't the only issue though, I think we need to be thinking about democratization of algorithms. Reverse chronological is obviously kinda limiting for engaging an audience. Why are Facebook/Twitter/Google/etc the only websites that can easily deploy an algorithmically organized "newsfeed" as their front page? How do we support algorithms that let small pages reorganize their content based on the user who is visiting? If I go to a site 6 times a day, why would it always show me the same content? If the site wants to, why shouldn't it try to engage me better than reverse chronological sorting is capable of?

I have ideas on how to do that with Elasticsearch. I'm not sure how we do it with Core on top of MySQL yet. I definitely think it is something we (Core) should pursue. I also think it will be very hard, and is not nearly as important for Core as the current work on Gutenberg and customization.