Introducing varnishtest


When I first started working with Varnish, my only concerns were mostly configuration and a bit of administration. Depending on your needs (which can impressively become complex sometimes) you can reach the limits of Varnish in terms of features (especially if your needs are not directly linked to HTTP caching). In this case, you will probably need a module or create your own, and I’ve already shown you how to make one. I already knew that Varnish comes with a test framework, but I didn’t expect it could also enable TDD for VMODs. I was lost when I first tried to read a test case, but I quickly found that the varnishtest framework is quite powerful and easy to use!

A test framework ?

When I stated earlier that Varnish comes with a test framework, I meant that you do have the varnishtest program when you install Varnish. It’s not just a tool for the Varnish development team, nor a tool for VMOD creators. It’s also a tool you can use to reproduce a bug when you file a bug report. If you look at the tests in Varnish’s source tree, you can see a README:

As of Varnish 3.0.3, released a few days ago, there are 294 tests you can study before writing your own. And today I’ll save you some time by showing the important parts of the VTC language.

The Varnish Test Case language

Many things in Varnish’s architecture are impressive for me. The design of a DSL for the configuration is one example, but I didn’t expect I would find another DSL for testing when I first used with Varnish. The VTC is really easy to both write and read, but you need a small mental shift (at least I needed) because it doesn’t follow the set up/test/assert/tear down or given/when/then patterns. Depending on your scenario, there might be test preparations, executions and assertions all over the place. Unlike the VCL, the VTC is not compiled but simply interpreted on the fly.

First of all, a test has a name:

Starting a varnishd instance

The thing about varnishtest is that it launches a real Varnish instance and interacts with it. It launches the varnishd instance, controls it through the management process and makes assertions based on the shared memory logs. It makes tests really slow (like a hundred milliseconds slow every time an expected value has yet to be found) but it actually tests the real product.

The Varnish instance needs a name. You can optionally configure it or just rely on the default VCL. If you are testing a VMOD, you can import it with an absolute file name.

The instance can be started later…

If you fork the libvmod-example module, your autotools configuration will run varnishtest with the vmod_topbuild macro. It works like Java system properties in the command line: -Dkey=value.

The -vcl+backend directive automatically injects the backend into the VCL, which most of the time is what you want to do. Sometimes you might have several backends (load balanced for instance) so you might want to do it manually:

Now let’s learn how varnishtest can mock backends and run clients.

Mocking the backend

The syntax for backends and clients is similar to the syntax for Varnish instances except that instead of VCL, you’ll find actions and assertions.

In this case, we are starting a server which answers a response to exactly one request. It could also answer to more requests, it doesn’t matter. It would be a problem if the server received more requests than expected, it would timeout and the Varnish instance would send a 503 Unavailable response (yet, this is a behaviour you might need to test). Those can be seen as actions: receiving a request (rx) and transmitting (tx) a response.

Answering up to two requests:

In addition to actions, you can add assertions. You can expect the server to receive a request with a given url, a specific header… Remember that your client’s request goes through a Varnish instance first and the backend request might change in some way you want to check.

It is of course possible to change the empty default response depending on your use cases.

Once you start a server named s1, macros are automatically made available:

${s1_addr} : the IP address (

${s1_port} : the port (randomly chosen)

${s1_sock} : the socket

It is also possible to use a real backend instead of just a mock. Remember that varnishtest launches a real varnishd instance. You could integrate Varnish in your CI environment and run your own integration or regression tests against your actual application.

Running client requests

Once you have a server and a Varnish, you can start running clients and sending requests. The syntax is the same as for the servers, but instead of expecting requests and answering responses, clients send requests and expect responses.

And it’s easy to run the same client’s scenario multiple times:

And of course a single client can send several requests:

We’ve seen that servers and clients are made to embed a scenario with both actions and expectations, embeded in the server or client body. Since a Varnish instance’s body contains VCL, is it possible to add assertions ? The answer is yes.

Varnish assertions

Unlike clients or servers, Varnish assertions are put outside the instance’s declaration body. What can you actually test with Varnish ? Since it’s a proxy to your backend, checking requests and responses is irrelevant. Instead, you have access to the counters exposed by varnishstat at any time (it doesn’t mean that all counters are relevant for testing purpose).

Now, how does it look like when you write a test case that does all this ?

A more advanced example


Test preparations, executions and assertions all over the place, didn’t I warn you ? Yes I did, however, it doesn’t mean that you can’t write readable tests. On the contrary, the varnishtest DSL is really expressive, especially once you know how to read it. It’s also easy to write, even though your test case is exploded (servers + varnish + clients), the isolation of those components makes it simpler IMHO. I should probably mention that varnishtest is not suited for performance tests by design.

This is only an introduction to varnishtest but I hope this will answer questions you might ask yourself when writing your first test cases.

Partagez cet article.

A propos de l'auteur

Ajouter un commentaire