Measure Jetpack: An Independent Project Aimed at Measuring Jetpack’s Performance

Project Jetpack BenchmarkJust about every time we publish an article about Jetpack, one or more readers leaves a comment similar to the following, I don’t use Jetpack because it slows down my site. Many of the comments don’t link to or provide data that backs up their claim.

Outside of the benchmarks produced by the BruteProtect team late last year, there’s little evidence to support the claim that Jetpack negatively impacts a site’s performance. Arūnas Liuiza, a WordPress plugin developer, is working on a new project that hopes to solve the mystery of whether or not Jetpack causes sites to load more slowly.

Preparing the Testing Environment

In the next few weeks, Liuiza and his students will scour the WordPress plugin directory to find plugins with comparable features to Jetpack’s modules. The team is going to set up three identical WordPress sites with default Lorem Ipsum content.

One site will act as the benchmark, another will run Jetpack, and the third will run plugins similar to Jetpack’s modules. Liuiza explains how his students will perform the benchmark process:

We’ll start the measuring, with different plugins/modules activated and deactivated. We’ll be doing 1-to-1 comparisons, as well as some combos. We are going to measure load times, as well as some stats reported by Query Monitor such as memory usage, database query count etc.

Liuiza says benchmark results will be published in a series of posts on his blog and on the Advanced WordPress Facbook group. Liuiza is asking for the community’s help to locate plugins that are feature-comparable to the modules in Jetpack.

He’s also asking for advice on the testing methodology he and his students should use. There’s no timetable on when the results will be published as it’s considered a side project.

Benchmark Frontend Modules Only

Jetpack currently has 36 modules but some of them simply connect to services such as VaultPress or VideoPress. I think Liuiza and his team should concentrate on modules that potentially impact a site’s frontend performance as it doesn’t make sense to benchmark modules that are only for the backend.

I’m doubtful that an independent study of Jetpack’s performance will solve the issue once and for all, especially if the results show that it doesn’t have much of an impact on sites as people think. However, it will be nice to have another set of data to point people to when it comes up in discussions.

26 responses to “Measure Jetpack: An Independent Project Aimed at Measuring Jetpack’s Performance”

  1. Fun! It will probably be difficult for this to be conclusive but it will no doubt be fascinating… and what an awesome project to get students involved with. I wish I had thought of it :)

    I bet Jetpack comes out of this with a clean bill of health for most of the modules… with one or two that could be better.

  2. Yeah, will be interested in the results, and I’m already guessing the outcome will be positive.

    I just finished doing a workshop on Jetpack, and when I was preparing for it I installed it on a live site hosted at GoDaddy. I have no data to back me up, as that was not my intentions, but I can tell you from that experience I sure didn’t notice any obvious slowdowns.

    In fact, I was really impressed overall after digging into it much deeper than I ever have :)

  3. Anyone that’s run the P3 (Plugin Performance Profile) plugin knows that Jetpack is always the weightiest plugin they have, by far. If it isn’t then God help whichever one it is because it’d have to be a monster. But it also makes sense and may even be appropriate, considering the number of modules most people are going to be using at one time. If you only needed a couple of features that Jetpack offers, then it may be smarter to use dedicated plugins and CDN or just code it up instead if you’re really concerned about incremental gains in speed.

    • If you use different Jetpack modules, each one of them will indeed have an effect on your site’s loading speed – much like different plugins would. It’s up to you to determine if the modules are worth the additional resources required. And you can always disable the ones that you don’t want to use.

      It’s also worth noting that Jetpack does run some additional functions when you’re logged in. One example is the Stats module, which will run some additional queries to get information about Stats on every page load. Your readers, however, won’t trigger this extra query. So, P3 could show inflated results. I would consequently recommend that you run the P3 plugin in a different browser or in an incognito window, where you won’t be logged in.

      I’m excited to hear about this project and I’m looking forward to finding additional ways we can improve Jetpack!

      • Whether p3 is highly accurate or not, it’s not like jetpack is marginally heavier. I’ve run it on dozens of established sites, and for it to be responsible for 50-60% of total load time isn’t unusual. But as I said, that may be appropriate considering the number of and which modules are enabled. I also don’t use very many other plugins.

  4. As a team member of Jetpack I think this is fantastic. We’re constantly looking and working to improve Jetpack. We certainly offer a lot of features within our plugin. If this analysis shows that there are ways we can improve performance in certain areas, then that’s great. If not, that’s okay too. Either way, I’m excited to see what kind of results are found.

  5. I think their test sites must have lots of content in order to draw meaningful and representative results. If they just use a fresh install with several pages, then the test is considered biased.

    • A fair point. We’ll definitely have more contents than just “Hello world” stuff. Creating a good test environment is the second task on our ToDo list, just behind the search of good plugins to test against. We want this to be as objective as possible, so we’ll really try to make an environment that can highlight both the strengths and the weaknesses.
      In the spirit of this being a scientific project, after we assemble our test environment, we plan to make it available on GitHub, in case there will be someone who wants to try and reproduce our results.

      P.S. Big thanks to Jeff Chandler for covering this!

      • Great. What system are you using for collecting alternative plugin recommendation? You mention on your blog that you need help with that but didn’t mention how we should get involved.

        • I haven’t set up a “system” yet, but You can reach me via usual channels – email, Twitter, etc. My blog has all the relevant links in the header. Here in the comments is fine too, if Jeff does not mind :)

          • There are ~34 current Jetpack modules. It seems like a form or survey in which not only can users vote on already known plugins for any of the 34… but also add a new plugin to the list which can be voted on. Some sort of a self-growing array represented by radio inputs.

            The result would be not only a wide sampling of recommendations… but also some feedback about which are more popular and widely used.

            As I think of it, though, i’m not aware of any form service or plugin which allows users to add an item to the list of available choices.
            Interesting idea.

            • I guess that even if a few modules don’t have good alternatives, it’s still worth benchmarking 27-30 plugins against Jetpack, both one by one, some combinations and the full stack.

              +1 on the “lots of data to test” since performance impact is an exponential exercise and isn’t noticeable with a few entries.

              Would be nice checking in an environment of a social network (i.e. BuddyPress) or WooCommerce/EDD – just saying, knowing that it’s not a full-time research study.

  6. Finally they are gonna test this. But i can confirm this. If my website loads at f.e 4 seconds. The moment i enable jetpack out of the box with nothing done besides just enabling it it loads avarage 2 seconds slower. I tested this many times and have seen this result in users websites also. This is why i never used it besides all the great features it has. So i am curious on the test results. I hope they confirm my findings and in the end resolve that issue as it is a nice plugin.

  7. Ha! Didn’t mean to be ‘that guy’.

    Glad to see this project in the works, very timely. Curious to see the results.

    As I said, it’s been awhile since trying Jetpack, but my experience in the past was something like this:

    Page not loading, look in the console, ‘waiting for connection to wordpress.com’.

    Needing to set up a dummy wp.com site to enable functionality on my self hosted site also seemed unecessarily convoluted.

    Anyway, with all the great features that Jetpack offers, I would love to see data showing it was not adversely affecting site performance.

    • I too am looking forward to seeing the results here!

      I noticed your comment about:
      > Needing to set up a dummy wp.com site to enable functionality

      This is not true. All you need is a wp.com user account, which requires only a valid email address (linky).

      • Yeah, sorry, I should have said “dummy account” instead of “dummy site”.

  8. Looking forward to the results of these testing. Anecdotally, I’ve found Jetpack to be good on the performance front. As noted by other posters, the reason it gets a bad rap is often because it delivers so much functionality and / or people enable a load of modules they aren’t actually using.

  9. That would be a great experiment. A quick idea – a public spreadsheet with each component coming from Jetpack and an infinite number of columns where other WP folks can suggest other plugins that could serve as an alternative.

    Not hard to implement and allows for good collaboration taking a few minutes tops.

  10. I’m still learning….

    If you have Jetpack (& other plugins) installed but not activated, will that slow down your site?

    If you have images uploaded in Media Files but not used on the site, will that slow down your site?

    If you have several posts in draft mode, but not published, will that slow down your site?

      • If you have lots of images uploaded, then they take up space in the database, as they are attachments. Large databases, particularly those with lots of data in a single table, are slower than those containing less data.

        This makes logical sense, since if there is less data to sort through, you should be able to find it faster.

        @Bob, it is unlikely that you will need to worry about the number of attachments you have, unless it is a truly enormous number. In that situation, you could work around this issue by uploading the images, but not attaching them. I’ve never worked on a project where this was necessary though. These sorts of problems normally arise when you are doing stuff like logging, and you end up with many millions of records, usually for inane stuff that you didn’t really need to store anyway.

        In normal usage, I wouldn’t worry too much about the stuff you described.

        Having plugins installed and not activated is mostly a security footprint problem.

  11. I consider myself a strong critic of JetPack, but performance is not something I’d have guessed it had a problem with. The code itself has always looked quite solid to me.

Newsletter

Subscribe Via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

%d bloggers like this: