Introducing varnishtest

When I first started working with Varnish, my only concerns were mostly configuration and a bit of administration. Depending on your needs (which can impressively become complex sometimes) you can reach the limits of Varnish in terms of features (especially if your needs are not directly linked to HTTP caching). In this case, you will probably need a module or create your own, and I’ve already shown you how to make one. I already knew that Varnish comes with a test framework, but I didn’t expect it could also enable TDD for VMODs. I was lost when I first tried to read a test case, but I quickly found that the varnishtest framework is quite powerful and easy to use!

A test framework ?

When I stated earlier that Varnish comes with a test framework, I meant that you do have the varnishtest program when you install Varnish. It’s not just a tool for the Varnish development team, nor a tool for VMOD creators. It’s also a tool you can use to reproduce a bug when you file a bug report. If you look at the tests in Varnish’s source tree, you can see a README:

Test-scripts for varnishtest
============================
Naming scheme
-------------
	The intent is to be able to run all scripts in lexicographic
	order and get a sensible failure mode.
	This requires more basic tests to be earlier and more complex
	tests to be later in the test sequence, we do this with the
	prefix/id letter:
		[id]%05d.vtc
	id ~ [a] --> varnishtest(1) tests
	id ~ [b] --> Basic functionality tests
	id ~  --> Complex functionality tests
	id ~ [e] --> ESI tests
	id ~ [g] --> GZIP tests
	id ~ [m] --> VMOD tests
	id ~ [p] --> Persistent tests
	id ~ [r] --> Regression tests, same number as ticket
	id ~ [s] --> Slow tests, expiry, grace etc.
	id ~ [t] --> sTreaming tests
	id ~ [v] --> VCL tests: execute VRT functions

As of Varnish 3.0.3, released a few days ago, there are 294 tests you can study before writing your own. And today I’ll save you some time by showing the important parts of the VTC language.

The Varnish Test Case language

Many things in Varnish’s architecture are impressive for me. The design of a DSL for the configuration is one example, but I didn’t expect I would find another DSL for testing when I first used with Varnish. The VTC is really easy to both write and read, but you need a small mental shift (at least I needed) because it doesn’t follow the set up/test/assert/tear down or given/when/then patterns. Depending on your scenario, there might be test preparations, executions and assertions all over the place. Unlike the VCL, the VTC is not compiled but simply interpreted on the fly.
First of all, a test has a name:

varnishtest "Test example vmod"

Starting a varnishd instance

The thing about varnishtest is that it launches a real Varnish instance and interacts with it. It launches the varnishd instance, controls it through the management process and makes assertions based on the shared memory logs. It makes tests really slow (like a hundred milliseconds slow every time an expected value has yet to be found) but it actually tests the real product.
The Varnish instance needs a name. You can optionally configure it or just rely on the default VCL. If you are testing a VMOD, you can import it with an absolute file name.

varnish v1 -vcl+backend {
	import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
	sub vcl_deliver {
		set resp.http.hello = example.hello("World");
	}
} -start

The instance can be started later…

varnish v1 -vcl+backend {
	import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
	sub vcl_deliver {
		set resp.http.hello = example.hello("World");
	}
}
[...]
varnish v1 -start

If you fork the libvmod-example module, your autotools configuration will run varnishtest with the vmod_topbuild macro. It works like Java system properties in the command line: -Dkey=value.
The -vcl+backend directive automatically injects the backend into the VCL, which most of the time is what you want to do. Sometimes you might have several backends (load balanced for instance) so you might want to do it manually:

varnish v1 -vcl {
	import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
	# you'll learn how to run a "s1" server
	backend default {
		.host = "${s1_addr}";
		.port = "${s1_port}";
	}
	sub vcl_deliver {
		set resp.http.hello = example.hello("World");
	}
} -start

Now let’s learn how varnishtest can mock backends and run clients.

Mocking the backend

The syntax for backends and clients is similar to the syntax for Varnish instances except that instead of VCL, you’ll find actions and assertions.

server s1 {
       rxreq
       txresp
} -start

In this case, we are starting a server which answers a response to exactly one request. It could also answer to more requests, it doesn’t matter. It would be a problem if the server received more requests than expected, it would timeout and the Varnish instance would send a 503 Unavailable response (yet, this is a behaviour you might need to test). Those can be seen as actions: receiving a request (rx) and transmitting (tx) a response.
Answering up to two requests:

server s1 {
       rxreq
       txresp
       rxreq
       txresp
} -start

In addition to actions, you can add assertions. You can expect the server to receive a request with a given url, a specific header… Remember that your client’s request goes through a Varnish instance first and the backend request might change in some way you want to check.

server s1 {
       rxreq
       expect req.url == "/"
       txresp
} -start

It is of course possible to change the empty default response depending on your use cases.

server s1 {
       rxreq
       expect req.url == "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
       txresp -body "A great article :p"
       rxreq
       expect req.url == "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
       txresp -body "The second part ;)"
} -start

Once you start a server named s1, macros are automatically made available:

${s1_addr} : the IP address (127.0.0.1)

${s1_port} : the port (randomly chosen)

${s1_sock} : the socket

It is also possible to use a real backend instead of just a mock. Remember that varnishtest launches a real varnishd instance. You could integrate Varnish in your CI environment and run your own integration or regression tests against your actual application.

Running client requests

Once you have a server and a Varnish, you can start running clients and sending requests. The syntax is the same as for the servers, but instead of expecting requests and answering responses, clients send requests and expect responses.

client c1 {
	txreq -url "/"
	rxresp
	expect resp.http.hello == "Hello, World"
} -run

And it’s easy to run the same client’s scenario multiple times:

client c1 {
	txreq
	rxresp
}
[...]
client c1 -run
client c1 -run
client c1 -run

And of course a single client can send several requests:

clent c1 {
	txreq -url "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
	rxresp
	expect resp.bodylen == 18
	txreq -url "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
	rxresp
	expect resp.bodylen == 18
} -run

We’ve seen that servers and clients are made to embed a scenario with both actions and expectations, embeded in the server or client body. Since a Varnish instance’s body contains VCL, is it possible to add assertions ? The answer is yes.

Varnish assertions

Unlike clients or servers, Varnish assertions are put outside the instance’s declaration body. What can you actually test with Varnish ? Since it’s a proxy to your backend, checking requests and responses is irrelevant. Instead, you have access to the counters exposed by varnishstat at any time (it doesn’t mean that all counters are relevant for testing purpose).

varnish v1 -expect the_counter == the_value

Now, how does it look like when you write a test case that does all this ?

A more advanced example

varnishtest "Test example vmod even more"
server s1 {
       rxreq
       expect req.url == "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
       txresp -body "A great article :p"
       rxreq
       expect req.url == "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
       txresp -body "The second part ;)"
} -start
varnish v1 -vcl+backend {
	import example from "${vmod_topbuild}/src/.libs/libvmod_example.so";
	sub vcl_deliver {
		set resp.http.hello = example.hello("World");
	}
} -start
client c1 {
	txreq -url "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1"
	rxresp
	expect resp.http.hello == "Hello, World"
	expect resp.bodylen == 18
	txreq -url "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2"
	rxresp
	expect resp.http.hello == "Hello, World"
	expect resp.bodylen == 18
}
varnish v1 -expect cache_miss == 0
varnish v1 -expect cache_hit == 0
client c1 -run
varnish v1 -expect cache_miss == 2
varnish v1 -expect cache_hit == 0
client c1 -run
client c1 -run
client c1 -run
varnish v1 -expect cache_miss == 2
varnish v1 -expect cache_hit == 6

Conclusion

Test preparations, executions and assertions all over the place, didn’t I warn you ? Yes I did, however, it doesn’t mean that you can’t write readable tests. On the contrary, the varnishtest DSL is really expressive, especially once you know how to read it. It’s also easy to write, even though your test case is exploded (servers + varnish + clients), the isolation of those components makes it simpler IMHO. I should probably mention that varnishtest is not suited for performance tests by design.
This is only an introduction to varnishtest but I hope this will answer questions you might ask yourself when writing your first test cases.