Varnish: the secret weapon for boosting page speed

Before We Begin

Request processing (especially dynamic) with web servers such as Apache can be a heavy strain on system resources.

That’s why more and more people started using lightweight web servers like nginx or lighttpd; they can work as a frontend for static content paired with Apache or independently. But there is another way to improve your site: using a web server accelerator such as Varnish Cache – a solution from Varnish Software.

There are a few ways to learn all the specifics of Varnish Cache processing: reading the Varnish Book, the Official documentation and a lot of practice and experimentation. We highly recommend starting with the first two. This article is based on all three, materials found at stackoverflow and other professional sites, and personal experience.


What is Varnish?

VARNISH is an open source caching reverse HTTP proxy, sometimes referred to as a web application accelerator. A reverse proxy is a proxy server that appears to clients as an ordinary server. It stores (caches) files or fragments of files in memory that are used to reduce the response time and network bandwidth consumption.

Varnish is available under a two-clause BSD licence. Commercial support is available from Varnish Software. Version 1.0 of Varnish was released in 2006, Varnish 2.0 in 2008, Varnish 3.0 in 2011 and Varnish 4.0 in 2014.

Varnish is flexible because you can configure it by writing your own caching policies in its Varnish Configuration Language (VCL). VCL is a domain specific language based on C. VCL is then translated to C code and compiled.


How does Varnish work?

Let’s imagine that the client sends a request to your server. Varnish receives it and tries to find an appropriate response in the cache storage. If it finds one and its content is still valid, then we have a response we can deliver to the client. Otherwise Varnish will forward the request to the backend, fetch the response, store it in the cache and send the response to the client. Next time the same request is made, Varnish will have the response in cache storage.

It’s easy to represent this with a simple diagram:

varnish cache flowchart


Installation and basic configuration of Varnish


You can easily find Varnish binary packages or compile it for yourself from source code. Use the official site for grabbing a current version. Installation can be done with the system package manager, but please note that this might not be the latest version of Varnish. For example under Debian 8 (“jessie”) do the following as root:


apt-get install apt-transport-https
curl | apt-key add -
echo "deb jessie varnish-4.1" >> /etc/apt/sources.list.d/varnish-cache.list
apt-get update
apt-get install varnish


If compiling from source, make sure that you already have dependencies installed. Please check the official documentation for a list.

The next step is configuration. First, let’s look at the configuration file
/etc/default/varnish (Debian)

/etc/sysconfig/varnish (Red Hat)


This holds a lot of useful and important settings for the Varnish daemon, such as locked shared memory, default TTL, numbers of threads, etc.

Here is a simple example:


             -f ${VARNISH_VCL_CONF} \
             -t ${VARNISH_TTL} \
             -p esi_syntax=0x2 \
             -p cli_buffer=16384 \
             -p http_resp_hdr_len=131072 \
             -p thread_pools=4 \
             -p thread_pool_add_delay=2 \
             -p thread_pool_min=15 \
             -p thread_pool_max=50 \
             -p thread_pool_timeout=3600 \
             -u varnish -g varnish \
             -S ${VARNISH_SECRET_FILE} \
             -s ${VARNISH_STORAGE}"


In VARNISH_VCL_CONF we define a path to a second file, which describes request handling and the caching process in the Varnish Configuration Language (VCL). VCL is translated to C and compiled into binary code which is then executed when requests arrive. This feature makes Varnish a powerful HTTP processor not only for caching, since VCL has inherited a lot from C and it reads much like a simple C or Perl script. We will look on it closer in next chapter.


A look inside

Let’s look inside request processing in Varnish v.4!

To start, here’s a simplified diagram of the Varnish Finite State Machine and a brief description of each of its processes:


varnish processes



The start of all processes, called just after the request has been received and parsed. It determines the next steps for incoming requests – will they be modified, cached or not, should anything be forwarded for processing by the backend or simply return a response from cache, etc.


Called after vcl_recv to create a hash value for the request that will be used as a lookup key for the object in Varnish. Passes control to the functions, depended on vcl_recv result.


Called when a cache lookup is successful. Its result determines how to handle the rest of the request.


Called after a cache lookup if the requested document was not found in cache or if vcl_hit returned fetch.


Called upon entering pass mode. In this mode, the request is passed on to the backend, and the backend’s response is passed on to the client, but is not entered into the cache.


Called upon entering pipe mode. In this mode, the request is passed on to the backend, and any further data from both the client and backend is passed on unaltered until either end closes the connection.


Backend servers

In the descriptions above, we referred to the backend several times. In Varnish terminology “backend” refers to any server(s) that provide cacheable content for Varnish, the one(s) that Varnish accelerates.

To tell Varnish where it can find a backend, define it in the VCL file like this:


backend default {
        .host = "";
        .port = "8080";



In real life we could encounter server crashes or other problems in the backend. To ensure continuity of work we will define a set of parameters called “probe” for checking the backend’s state.


backend default{
        .host = "";
        .port = "8000";
        .probe = {
                .url = "/alive/";
                .timeout = 2s;
                .interval = 3s;
                .window = 10;
                .threshold = 9;



Now Varnish will send a simple GET request to “/alive/” url every 3 seconds, with a timeout limit of 2 seconds. Since we set “window” to 10 and “threshold” to 9, the backend will be considered healthy if at least 9 of the last 10 requests succeeded. Varnish will not send traffic to hosts that are marked as unhealthy.

Great, now when Varnish needs content it will know where to get it by using a connection to a default backend. But what if you have multiple backend servers or maybe you have some special policy for some requests? We could define a new backend called ‘’magic’’ and add a new rule for processing requests:



backend magic {
        .host = "";
        .port = "8000";
sub vcl_recv {
        if (req.url ~ "^/magic/") {
                set req.backend_hint = magic;
        } else {
                set req.backend_hint = default;



You are free to define your own method of specifying a backend for a particular request, based on rules, parameters or randomly. You can also group several backend servers together into a “director”. This requires you to load a VMOD, a Varnish module, and then call certain actions in vcl_init.



import directors;    # load the directors
backend default {
        .host = "";
        .port = "8080";
backend magic {
        .host = "";
        .port = "8000";
sub vcl_init {
        new foobar = directors. round_robin();
sub vcl_recv {
        set req.backend_hint = foobar.backend();


This is a good way to use Varnish for load balancing. Here we are using a round-robin director, but there is also a random director which distributes requests randomly.


Let’s talk about Edge Side Includes (ESI)

Sometimes we need to display unique information to each user, like a user’s name, contents of their cart or a list of past orders etc. Caching such data can be a huge problem, that’s why one of the best things implemented in Varnish are ESI blocks.

Using ESI we can break our page up into smaller pieces and decide which of them should be cached or not, then assembling them into a response. These fragments can have individual cache policies, for example if you have a list of the 5 most popular products in your online store, this list can probably be cached for a time and included in other pages, while other fragments might never be cached and must always be loaded dynamically.


varnish esi diagram


With this strategy you can increase your hit rate and reduce the load on your servers.

ESI only implements a small subset of the full Varnish functionality, but it is enough. You can use three ESI statements:

  1. esi:include
  2. esi:remove
  3. <!–esi …–>

Varnish will not process ESI instructions in HTML comments.

Let’s look at some examples.



We will make a simple example. This is user.php file:

// Including some libraries
$user = $this->getUser();
echo $user->getName();



Now some HTML file that has an ESI include statement. This is test.html:

        Hello <esi:include src="user.php"/>!


And the last step before ESI will work is activation ESI processing in VCL.


sub vcl_backend_response {
        if (bereq.url == "/test.html") {
               set beresp.do_esi = true;
               set beresp.ttl = 24 h;   
        } elseif (bereq.url == "/user.php") {
               set beresp.ttl = 1m;




ESI:remove and <!–esi … –>


But what should we do if ESI is not available? For that there are the <esi:remove> and <!–esi … –> constructs. You can use it to present content whether or not ESI is available, for example you can include content when ESI is available or link to it when it is not.

The ESI processors will remove the starting “<!–esi” and the ending “-→” tags when the page is processed, while still processing the contents. If the page is not processed, it will remain intact, becoming a HTML/XML comment tag. ESI processors will remove <esi:remove> tags and all content contained in them, allowing you to only render the content when the page is not being ESI-processed. For example:

  <a href="">The license</a>
<p>The full text of the license:</p>
<esi:include src="" />


Who uses Varnish Cache

Based on information from Wikipedia, Varnish is used in around a tenth of the TOP 10K sites – online newspapers, social media and media content sites; here’s short a list of the most famous: Wikipedia, Facebook, Twitter, Vimeo, The New York Times, The Guardian, The Hindu, and Tumblr.


Use Varnish in Magento


Magento is the top ecommerce platform in the world. It’s known for being flexible, but not so much for being fast. If you want to use Varnish with Magento 1, you should find the ready solution for it or if you have a lot of time: Do-It-Yourself. You just need to learn how to use Magento, understand HTTP Headers and get a hold of reverse proxy caching and Varnish Cache specifically.

"There are only two hard things in Computer Science: cache invalidation and naming things."Phil Karlton

In case of Magento 2 – everything is not so bad. Magento 2 officially supports Varnish versions 3.0.5 or later or any Varnish 4.x version. Just follow the tutorial in the documentation, but we’ll summarize it for the sake of completeness:

  • Install Varnish and test it by accessing any Magento page to see if you’re getting an HTTP response header that indicates Varnish is working.
  • Install the Magento software and use the Magento Admin to create a Varnish configuration file.
  • Replace your existing Varnish configuration file with the one generated by the Admin.
  • Test everything again and if there is nothing in your /var/page_cache directory, you’ve successfully configured Varnish with Magento!

Finally, remember that your (or your customers’) ecommerce site is the last possible place on the web where you should be testing random VCL snippets and unknown PHP code.



6 replies

Trackbacks & Pingbacks

  1. […] Slow page load influences search engine rankings and it scares off potential customers. […]

  2. […] Varnish is a reverse caching proxy, that can dramatically decrease page load times. It sits between visitors’ browsers and the backend server, intercepting each request and attempting to provide a previously cached response from its own static storage while requesting only the dynamic elements from the backend. The developers of Magento recognize its power and Magento 2 uses this service by default, but stores on Magento 1 can benefit from it as well. In this presentation Tamás will show you how. […]

  3. […] Putting all the pieces together on a slow website is a sure fire way to lose customers and sales. Boost your load times with these tips: 8 Effective Ways to Really Boost Magento Page Speed! + Varnish: the secret weapon for boosting page speed) […]

  4. […] Varnish is one of the most popular caching solutions for serving dynamic content. We have covered this topic in one of our previous posts: Varnish: the secret weapon for boosting page speed […]

  5. […] bad for you both on desktop and on mobile. Of course there are remedies for that, like using Varnish cache, but it won’t solve user experience […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published.