Speed Up Your WordPress Site with Pound, Varnish, Nginx and mod_pagespeed

tomaz

This post was contributed by Tomaž Zaman. He is the founder of a Danish startup called Codeable, a WordPress-only outsourcing service on a mission to help WordPress companies and enthusiasts from all over the world effortlessly scale their businesses. He spends his free time with his wife and four kids.

A vast majority of online articles about speeding up your WordPress site mention and use a plugin called W3 Total Cache (or W3TC for short). Rightfully so, because it’s a great, all-in-one solution to get your WP site fast with relatively small amount of work.

However, I’m not really a fan of all-in-one solutions, mainly because they bring a lot of complexity to the table, and as far as WordPress plugins go, possibly introduce compatibility issues and upgrade headaches.

That is why I decided to look for alternatives that would still make our (secure) WordPress site really fast and a friend recommended to look at mod_pagespeed, which turned out to be a great solution for making our WordPress really fast, with minimal effort.

Prerequisites

This article assumes you have at least basic knowledge of linux (all our examples are Ubuntu-based), know how to use the shell, and most importantly, you host your site on your own VPS. Shared hosting isn’t going to cut it, since we need to set up a customized version of nginx, which supports mod_pagespeed.

There are plenty of awesome and relatively cheap VPS providers out there, I’d personally recommend Digital Ocean.

Note that all commands that start with the dollar sign ($) indicate it’s a unix command, and you don’t actually enter that dollar sign into the command.

Main ingredients

As the title suggests, we’ll need a couple of programs installed on our server; The main one (for caching purposes) is called Varnish, which basically stores all your HTML output onto a temporary folder on disk and serves that instead of delegating requests to WordPress. It has one flaw, however – it does not support SSL termination, which is why we need Pound for.

The last two components we need are Nginx (it’s a web server like Apache) and a PHP process manager called PHP-FPM, since Nginx does not support modules.

Of course you also need to generate a certificate key and buy a certificate for this setup to work. Entry-level certificates are quite cheap, some starting with less that $10/year, which makes it a no-brainer.

Getting started

Provided you have a clean, fresh VPS ready, SSH on to it. Protip: If you use Linux or Mac OS locally, you can add the following a shortcut to ~/.ssh/config(create this file if it does not exist):

Host MyHost
  HostName [host-ip]
  Port [host-port]
  User [host user, usually root]
  IdentityFile [path/to/your/ID_RSA, usually ~/.SSH/ID_RSA]

This allows you to run the following command to log into your server shell: $ ssh MyHost.

Once logged in, it’s time to install all the necessary components that we’ll need via a package manager, in our case apt-get.

Run the following command: $ sudo apt-get install php5-fpm php5-cli php5-mysql varnish pound mariadb-server-5.5 unzip You’ll be prompted to enter the root password for MariaDB, which is a drop-in replacement for MySQL, twice. We did not install Nginx at this point because the version that comes as the default, does not support mod_pagespeed, which we need, so we’ll install it manually. To do that, visit the official documentation and follow the guide, with one exception, replace this line:

./configure --add-module=$HOME/ngx_pagespeed-release-${NPS_VERSION}-beta

with this one:

./configure --add-module=$HOME/ngx_pagespeed-release-${NPS_VERSION}-beta --with-http_gzip_static_module --with-http_realip_module

This will make sure you also include modules that will help you GZIP static assets and allow you to track correct IPs in your WordPress, respectively. Once done, you should have Nginx installed in /usr/local/nginx/.

If you try to visit the URL (or IP) of your server at this point, nothing will happen, because we haven’t configured all the components properly yet. Once we do, the communication schema between components on our server will look something like this:

diagram

Pound

Let’s start with the outermost component, which should listen on two standard ports, 80 and 443 (HTTP and HTTPS, respectively). Edit the file /etc/pound/pound.cfg and put the following contents in:

User            "www-data"
Group           "www-data"
LogLevel        1
Alive           30
Control "/var/run/pound/poundctl.socket"

ListenHTTP
    Address  0.0.0.0
    Port     80
    # This part makes sure you redirect all HTTP traffic to HTTPS
    Service
        HeadRequire "Host: yourdomain.com"
        Redirect "https://yourdomain.com"
    End
End
ListenHTTPS
    HeadRemove "X-Forwarded-Proto"
    AddHeader  "X-Forwarded-Proto: https"
    Address    0.0.0.0
    Port       443
    Cert       "/etc/ssl/yourdomain.com/yourdomain.com.pem"
    # This service removes the WWW-part
    Service
        HeadRequire "Host: www.mydomain.com"
        Redirect "https://yourdomain.com"
    End
    # The main service which passes requests to Varnish
    Service
        HeadRequire "Host: yourdomain.com"
        BackEnd
            Address 127.0.0.1
                    # 6081 is the default Varnish port
            Port    6081
        End
    End
End

Before restarting pound with this new configuration, you’ll also need to have a PEM certificate file in the defined folder, which you can create by following these instructions. You’ll also need to edit /etc/default/pound and set startup=1.

That’s it! Pound is now configured and after running $ service pound restart (on Ubuntu, other distributions may have different commands for service manipulation) you should be able to visit the configured domain. Of course, you’ll get an error (503 Service Unavailable), but that’s okay, we’re not done yet!

Varnish

Next, we need to configure Varnish. Edit the file `/etc/varnish/default.vcl and put the following in (delete everything else):

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

acl purge {
  "127.0.0.1";
  "localhost";
}

sub vcl_recv {
    if (req.request == "PURGE") {
      if (!client.ip ~ purge) {
        error 405 "Not allowed.";
      }
      ban("req.url ~ "+req.url+" && req.http.host == "+req.http.host);
      error 200 "OK";
    }

    # only using one backend
    set req.backend = default;

    # set standard proxied ip header for getting original remote address
    set req.http.X-Forwarded-For = client.ip;

    # logged in users must always pass
    if( req.url ~ "^/wp-(login|admin)" || req.http.Cookie ~ "wordpress_logged_in_" ){
        return (pass);
    }

    # don't cache search results
    if( req.url ~ "\?s=" ){
        return (pass);
    }

    # always pass through posted requests and those with basic auth
    if ( req.request == "POST" || req.http.Authorization ) {
         return (pass);
    }

    # else ok to fetch a cached page
    unset req.http.Cookie;
    return (lookup);
}



sub vcl_fetch {

    # remove some headers we never want to see
    unset beresp.http.Server;
    unset beresp.http.X-Powered-By;

    # only allow cookies to be set if we're in admin area - i.e. commenters stay logged out
    if( beresp.http.Set-Cookie && req.url !~ "^/wp-(login|admin)" ){
        unset beresp.http.Set-Cookie;
    }

    # don't cache response to posted requests or those with basic auth
    if ( req.request == "POST" || req.http.Authorization ) {
         return (hit_for_pass);
    }

    # only cache status ok
    if ( beresp.status != 200 ) {
        return (hit_for_pass);
    }

    # don't cache search results
    if( req.url ~ "\?s=" ){
        return (hit_for_pass);
    }

    # else ok to cache the response
    set beresp.ttl = 24h;
    return (deliver);
}



sub vcl_deliver {
    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
    }
    else {
        set resp.http.X-Cache = "MISS";
    }
    unset resp.http.Via;
    unset resp.http.X-Varnish;
}

sub vcl_hit {
  if (req.request == "PURGE") {
    purge;
    error 200 "OK";
  }
}

sub vcl_miss {
  if (req.request == "PURGE") {
    purge;
    error 404 "Not cached";
  }
}


sub vcl_hash {
    hash_data( req.url );
    if ( req.http.host ) {
        hash_data( regsub( req.http.host, "^([^\.]+\.)+([a-z]+)$", "\1\2" ) );
    } else {
        hash_data( server.ip );
    }
    # ensure separate cache for mobile clients (WPTouch workaround)
    if( req.http.User-Agent ~ "(iPod|iPhone|incognito|webmate|dream|CUPCAKE|WebOS|blackberry9\d\d\d)" ){
        hash_data("touch");
    }
    return (hash);
}

I won’t go into much details here, because official Varnish documentation does a much better job at that. Three things worth mentioning here are the lines that PURGE the cache (delete it), which comes in handy to have within WordPress so that updated parts automatically initiate cache removal. There are several plugins for that, we use Better WP Varnish, but any will do.

The second thing to mention is cookie handling. This configuration prevents logged in users to see cached versions of WordPress for easier debugging and previewing.

Last, we also use a custom header (X-Cache), which equals to either HIT or MISS so you can easily debug and see whether Varnish fetched a cached version (HIT) or not.

As with Pound, it’s now time to restart Varnish to load the new configuration: $ service varnish restart.

Nginx and mod_pagespeed

Almost there! The final piece of the puzzle is Nginx. Because we didn’t use the package manager to install it, we first need to set up an init script that will register Nginx as a daemon (a program or a process that runs in the background). In order to do that, run the following commands:

$ cd /etc/init.d
$ wget https://raw.githubusercontent.com/JasonGiedymin/nginx-init-ubuntu/master/nginx
$ chmod +x nginx
$ update-rc.d nginx defaults

This downloads the init script, makes it executable (because it’s essentially a program) and registers it with the operating system so it can start at boot time.

Once that’s sorted, we need to configure Nginx by editing the file /usr/local/nginx/conf/nginx.conf. Locate the first server block in the file, and replace it with the following code (this assumes you have your WordPress installed in /srv/www/yourdomain.com):

server {
        listen       8080;
        server_name  test.com;

        root /srv/test.com;
        index index.php index.html;

        location / {
                try_files $uri $uri/ /index.php?$args;
        }

        location ~ .php$ {
                try_files $uri /index.php;
                include fastcgi_params;
                fastcgi_pass unix:/var/run/php5-fpm.sock;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_index index.php;
        }

        pagespeed On;
        pagespeed ModifyCachingHeaders on;

        # Needs to exist and be writable by nginx.
        pagespeed FileCachePath "/var/cache/ngx_pagespeed/";
        pagespeed LoadFromFile "https://yourdomain.com" "/srv/yourdomain.com";
        pagespeed MapOriginDomain "http://localhost" "https://yourdomain.com";
        pagespeed EnableFilters rewrite_css,combine_css,trim_urls;
                pagespeed DisableFilters sprite_images,convert_jpeg_to_webp,convert_to_webp_lossless;

}

The top half is just basic configuration for our virtual host which enables rewrites and delegates PHP processing to PHP-FPM (listening on a socket).

What’s interesting here is the pagespeed part. After turning it on, we need to properly map URLs so that mod_pagespeed will know when to trigger its magic. After that, we enable/disable various filters and it’s up to you to which level you want mod_pagespeed to modify/optimize your HTML so go ahead and play with them. My favorites are ones that automatically combine CSS and Javascript files, optimize images and rename them according to their last-modified time, so we can have a far-future expiration date.

There’s only one caveat: Make sure to disable convert_jpeg_to_webp filter. The name of the filter is pretty descriptive but can pose a problem because of Varnish. If the very first visitor of your website uses Chrome, Varnish will pass the request on to Nginx and it will detect that and serve images in WEBP format, filling the cache with them. All subsequent users will then get that cached version, but as WEBP is not yet widely supported users with other browsers won’t be able to see the images!

Now it’s time to see if our effort was fruitful; Restart Nginx to load the new configuration ($ service nginx restart) and visit the website. UH-OH! You probably don’t see what you expected, right? That’s because Varnish is working, and probably serving erroneous version in case you visited it before. The fix is simple, just clear all Varnish cache ($ varnishadm "ban req.url ~ /") and you should be able to see the proper output.

To verify, you might want to inspect the response locally: $ curl -i https://yourdomain.com. In the response you should see two headers: X-Cache and X-Page-Speed, which indicates you’re now a webmaster of a considerably faster WordPress website!

What about benchmarks?

I’ve been asked about benchmarking the results, but in this case, I don’t believe it would make much sense. There’s plenty of Varnish benchmarks out there, but mod_pagespeed isn’t just about speed (despite it’s name). As per it’s documentation it can do a whole lot to optimize your website output in terms of asset concatenation, minification and serving.

That being said, if you still prefer having hard numbers for this, feel free to test it out and report back and I’ll be more than happy to update this article with your findings.

Conclusion

What I didn’t address in this article is WordPress installation, GZIPping, error handling and various possible corner cases, but I think this tutorial should give you a good base to start off with on the path to improved user experience.

35 Comments


  1. Tomaz, thanks for this guide. One question, is the SSL and Pound for SPDY? or ….

    ok two questions… why not redis instead of varnish?

    Report


    1. You’re welcome!

      SSL and Pound are used to serve the website through a secure connection. SPDY could be used in this configuration as well, it would probably need to be compiled into nginx and configured, but it’s doable.

      Redis and Varnish are two very different technologies. Varnish is a highly optimized http caching layer (doesn’t even support SSL on it’s own), while Redis is a cache store (a database engine). While I can imagine Redis performing the same function, it would require a lot of development to get there.

      Report


      1. I haven’t read the articles, but they’re probably for providing object caching to WordPress, not for front-end static page caching. You might be able to use them with something like Batcache, but it would still be slower than Varnish or Nginx static caching since they’re further up the stack.

        Report


  2. PageSpeed does amazing things and well however I think it’s not suitable for small VPS because of the CPU consumption required. In many cases, optimizing once for good the assets with plugins or not is more suitable. However on big VPS it really doe an impressive job.

    Why choosing Pound over an SSL offload on nginx, then you could have added the SPDY support to it?

    Report


  3. Why use Pound? If you are already using Nginx then you can use it as the end point, so: Browser -> Nginx (listen 443, proxy to Varnish) -> Varnish -> Nginx (proxy to PHP-FPM)

    Adding Pound is simply adding another point of failure, and rather misses the point of what Nginx does :)

    It’s probably also worth making sure to emphasis that people understand Varnish is one of those things that can completely screw things up if you don’t understand what is going on so before following any tutorial like this it’s really important to have a read, and a play on a non critical site.

    Report


    1. Just to follow up on my original comment, another big advantage to using Nginx instead of Pound, is that a lot of the bit’s done by mod_pagespeed are on a per client basis, such as image substitution for example so you can do these on the front nginx end point, and then reuse mod_pagespeed on the back for concatenation and minfying.

      Report


    2. Using Pound in front is because Varnish don’t recognize the SSL request. So. It’s kind like a encode module for Varnish.

      Report


      1. Hi Nel, As I mentioned above Pound is NOT needed in this scenario, Nginx can used where Pound is used and then used after varnish as well.

        So: Nginx -> Varnish -> Nginx -> PHP

        From a site performance point of view this is a no brainer, you can enable SPDY out of the box, you can make full use of mod_pagespeed rather then just its non user specific aspects.

        From a security stand point it’s also more useful as you can utilise Naxsi or similar WAF on the front end, you also can route unwated traffic through to things like Fail2Ban. (you can do some of these with the above setup but not as effectively)

        From a management point of view Pound is an extra thing to worry about and maintain.

        Report


      2. You could even replace Varnish (or at least some aspects of it) with nginx (i.e. fastcgi_cache).

        Report


      3. Tim, thanks for the suggestion. Truth be told, it didn’t even occur to me to try nginx on two points. I’ll try it out!

        Report


  4. Why use varnish/pound instead of nginx’ fastcgi_cache?

    Report


  5. This is an excellent write-up in the “advanced” category, I’m glad to see articles like this on the Tavern!
    Tomaž, at my company we use a similar configuration sans Varnish and Pound. Nginx has all the functionality that Pound has when it comes to SSL and it’s built-in FastCGI cache can substitute Varnish. I believe in keeping it simple and I also think that less software means faster response time and fewer points of failure, not to mention hardware resource requirements :)

    Report


  6. Does this setup works with HHVM instead of PHP-FPM? I’m already using Nginx.

    Report


    1. I’m not sure about this exact setup, but yes, HHVM and Nginx work extremely well together for an incredible amount of performance. As others in the comments have mentioned, if you use Nginx’s built in capabilities you don’t really need Pound or Varnish. There may be some issues with some plugins but core and the vast majority of plugins will run well on HHVM and Nginx.

      Report


  7. Hi!

    Can I know with website have this setup, so I can run a couple of test with pagespeed insight and also gtmetrix and see the speed and perfomance of the setup running… ??

    Regards…

    Report


    1. Those tools are for measuring on page speed improvements. They are not very useful for measuring the types of performance increases obtained through this sort of setup.

      Something like Apache Bench would be more helpful, but I don’t think anyone’s going to want you testing that out on their website.

      Report


  8. Nice article. I think pound and varnish are not required. Nginx has built-in support for SPDY.

    Also, on fresh VPS, whole thing can be automated in 3 commands using EasyEngine – http://github.com/rtCamp/easyengine

    1. install easyengine

    wget -qO ee rt.cx/ee && sudo bash ee

    2. install nginx, php, mysql, postfix, etc

    ee stack install

    3. setup wordpress at example.com with nginx fastcgi_cache

    ee site create example.com --wpfc

    Report


    1. Hi Rahul, I a big fan of your article concerning Nginx :)
      From what I have seen Varnish and Nginx both have similar performance but I was wondering how flexible are the rules on nginx concerning the cache.

      On varnish, there are few tricks to strip the cookies or to make them part of the hash. How do you handle cookies and logged users on nginx?

      Report


      1. Hi Michael, thanks for your appreciation. :-)

        You can use nginx’s 3rd party-module https://github.com/openresty/headers-more-nginx-module to intercept cookies as well as other HTTP headers.

        For logged in users – nginx can use WordPress cookie `wordpress_logged_in` to bypass cache. Example – https://github.com/rtCamp/easyengine/blob/master/config/nginx/common/wpfc.conf#L21-23

        I never used Varnish so it will be incorrect on my part to compare it’s flexibility with Nginx. But is last 5 years, we haven’t come across any case where nginx cache failed to deliver.

        Nginx core + 3rd party modules together provides many different cache solutions.

        We maintain custom builds of nginx on Launchpad – https://launchpad.net/~rtcamp/+archive/ubuntu/nginx . With these builds, you may try many 3rd party nginx modules without compiling nginx from source on Ubuntu.

        Report


      2. It works pretty much like varnish for logged in users with cookies and header invalidation.

        I guess the only thing missing in fastcgi cache is the ESI for fragment caching mgt. But I could wrong.

        I will definitely look into it Rahul as I would be happy to drop varnish in some projects if there is no disadvantages.

        It’s just I am so used to using both varnish and nginx that it became a habit :)

        Tnx for the feedback Rahul.

        Report


      3. For ESI, there is a module – https://github.com/pintsized/ledge . Our launchpad build doesn’t have it but.

        We use fastcgi_cache for simple content websites where writers/site-admins are only logged in users. For membership-site or bbpress forums, where logged in users percentage is higher, memcache-based object-cache and database-cache (using a plugin like W3 Total Cache) is good choice.

        Using HHVM can speed up things so much that you will stop wondering about other caching mechanism! Only downside is, hhvm doesn’t like all WordPress plugins, so it is recommended to use FPM as backup php-interpreter. Reference – https://github.com/rtCamp/easyengine/wiki/HHVM-with-EE-2.x.x#use-hhvm-and-adding-php5-fpm-as-fallback

        Report


  9. Thanks Rahul, That’s some precious info there. I haven’t heard about this ESI module before.

    Report


  10. Excellent article! I’ve recently been experimenting using Digital Ocean to run Litespeed and MariaDB. DO is such a great platform for learning new tools. Nginx is next on the list…so, thanks!

    Report


  11. Hi!

    This is a very helpful and straightforward article, and the first one that advise to use nginx (instead of apache) and wordpress together.

    Regarding to the caveat…

    You may enable convert_jpeg_to_webp filter. To avoid having problems with varnish cache you should add the next lines to vcl_fetch:

    if ( req.http.X-Accept ) {
    hash_data (req.http.X-Accept);
    }

    And, in vcl_recv:

    if ( req.http.Accept ~ “.*image/webp.*” ) {
    set req.http.X-Accept = “GoogleImageFormat”;
    }
    if ( req.http.Accept !~ “.*image/webp.*” ) {
    set req.http.X-Accept = “other”;
    }

    With that code, varnish will make a difference between Chrome a firefox browsers.

    Report


  12. My vote goes to mod_pagespeed. I’m using it with my VPS server and it works like a charm!

    Report


  13. Hello ,

    I am using pound>varnish+turpentine>nginx, but when i did curl to my test domain, i didnt get the headers of varnish or turpentine, any suggestion would be appriciated

    [root@localhost conf.d]# curl -I varnish.zeondemo.com
    HTTP/1.1 200 OK
    Content-Type: text/html; charset=UTF-8
    P3p: CP=”CAO PSA OUR”
    LOCAL: varnish-emedco
    Date: Thu, 13 Aug 2015 17:47:02 GMT
    Age: 7
    Connection: keep-alive

    Report

Comments are closed.