Introducing varnishtest
When I first started working with Varnish, my only concerns were mostly configuration and a bit of administration. Depending on your needs (which can impressively become complex sometimes) you can reach the limits of Varnish in terms of features (especially if your needs are not directly linked to HTTP caching). In this case, you will probably need a module or create your own, and I’ve already shown you how to make one. I already knew that Varnish comes with a test framework, but I didn’t expect it could also enable TDD for VMODs. I was lost when I first tried to read a test case, but I quickly found that the varnishtest
framework is quite powerful and easy to use!
A test framework ?
When I stated earlier that Varnish comes with a test framework, I meant that you do have the varnishtest
program when you install Varnish. It’s not just a tool for the Varnish development team, nor a tool for VMOD creators. It’s also a tool you can use to reproduce a bug when you file a bug report. If you look at the tests in Varnish’s source tree, you can see a README:
Test-scripts for varnishtest ============================ Naming scheme ------------- The intent is to be able to run all scripts in lexicographic order and get a sensible failure mode. This requires more basic tests to be earlier and more complex tests to be later in the test sequence, we do this with the prefix/id letter: [id]%05d.vtc id ~ [a] --> varnishtest(1) tests id ~ [b] --> Basic functionality tests id ~ --> Complex functionality tests id ~ [e] --> ESI tests id ~ [g] --> GZIP tests id ~ [m] --> VMOD tests id ~ [p] --> Persistent tests id ~ [r] --> Regression tests, same number as ticket id ~ [s] --> Slow tests, expiry, grace etc. id ~ [t] --> sTreaming tests id ~ [v] --> VCL tests: execute VRT functions
As of Varnish 3.0.3, released a few days ago, there are 294 tests you can study before writing your own. And today I’ll save you some time by showing the important parts of the VTC language.
The Varnish Test Case language
Many things in Varnish’s architecture are impressive for me. The design of a DSL for the configuration is one example, but I didn’t expect I would find another DSL for testing when I first used with Varnish. The VTC is really easy to both write and read, but you need a small mental shift (at least I needed) because it doesn’t follow the set up/test/assert/tear down or given/when/then patterns. Depending on your scenario, there might be test preparations, executions and assertions all over the place. Unlike the VCL, the VTC is not compiled but simply interpreted on the fly.
First of all, a test has a name:
varnishtest "Test example vmod"
Starting a varnishd instance
The thing about varnishtest
is that it launches a real Varnish instance and interacts with it. It launches the varnishd
instance, controls it through the management process and makes assertions based on the shared memory logs. It makes tests really slow (like a hundred milliseconds slow every time an expected value has yet to be found) but it actually tests the real product.
The Varnish instance needs a name. You can optionally configure it or just rely on the default VCL. If you are testing a VMOD, you can import it with an absolute file name.
varnish v1 -vcl+backend { import example from "${vmod_topbuild}/src/.libs/libvmod_example.so"; sub vcl_deliver { set resp.http.hello = example.hello("World"); } } -start
The instance can be started later…
varnish v1 -vcl+backend { import example from "${vmod_topbuild}/src/.libs/libvmod_example.so"; sub vcl_deliver { set resp.http.hello = example.hello("World"); } } [...] varnish v1 -start
If you fork the libvmod-example module, your autotools configuration will run varnishtest
with the vmod_topbuild
macro. It works like Java system properties in the command line: -Dkey=value
.
The -vcl+backend
directive automatically injects the backend into the VCL, which most of the time is what you want to do. Sometimes you might have several backends (load balanced for instance) so you might want to do it manually:
varnish v1 -vcl { import example from "${vmod_topbuild}/src/.libs/libvmod_example.so"; # you'll learn how to run a "s1" server backend default { .host = "${s1_addr}"; .port = "${s1_port}"; } sub vcl_deliver { set resp.http.hello = example.hello("World"); } } -start
Now let’s learn how varnishtest
can mock backends and run clients.
Mocking the backend
The syntax for backends and clients is similar to the syntax for Varnish instances except that instead of VCL, you’ll find actions and assertions.
server s1 { rxreq txresp } -start
In this case, we are starting a server which answers a response to exactly one request. It could also answer to more requests, it doesn’t matter. It would be a problem if the server received more requests than expected, it would timeout and the Varnish instance would send a 503 Unavailable
response (yet, this is a behaviour you might need to test). Those can be seen as actions: receiving a request (rx
) and transmitting (tx
) a response.
Answering up to two requests:
server s1 { rxreq txresp rxreq txresp } -start
In addition to actions, you can add assertions. You can expect the server to receive a request with a given url, a specific header… Remember that your client’s request goes through a Varnish instance first and the backend request might change in some way you want to check.
server s1 { rxreq expect req.url == "/" txresp } -start
It is of course possible to change the empty default response depending on your use cases.
server s1 { rxreq expect req.url == "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1" txresp -body "A great article :p" rxreq expect req.url == "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2" txresp -body "The second part ;)" } -start
Once you start a server named s1
, macros are automatically made available:
${s1_addr}
: the IP address (127.0.0.1)
${s1_port}
: the port (randomly chosen)
${s1_sock}
: the socket
It is also possible to use a real backend instead of just a mock. Remember that varnishtest
launches a real varnishd
instance. You could integrate Varnish in your CI environment and run your own integration or regression tests against your actual application.
Running client requests
Once you have a server and a Varnish, you can start running clients and sending requests. The syntax is the same as for the servers, but instead of expecting requests and answering responses, clients send requests and expect responses.
client c1 { txreq -url "/" rxresp expect resp.http.hello == "Hello, World" } -run
And it’s easy to run the same client’s scenario multiple times:
client c1 { txreq rxresp } [...] client c1 -run client c1 -run client c1 -run
And of course a single client can send several requests:
clent c1 { txreq -url "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1" rxresp expect resp.bodylen == 18 txreq -url "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2" rxresp expect resp.bodylen == 18 } -run
We’ve seen that servers and clients are made to embed a scenario with both actions and expectations, embeded in the server or client body. Since a Varnish instance’s body contains VCL, is it possible to add assertions ? The answer is yes.
Varnish assertions
Unlike clients or servers, Varnish assertions are put outside the instance’s declaration body. What can you actually test with Varnish ? Since it’s a proxy to your backend, checking requests and responses is irrelevant. Instead, you have access to the counters exposed by varnishstat
at any time (it doesn’t mean that all counters are relevant for testing purpose).
varnish v1 -expect the_counter == the_value
Now, how does it look like when you write a test case that does all this ?
A more advanced example
varnishtest "Test example vmod even more" server s1 { rxreq expect req.url == "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1" txresp -body "A great article :p" rxreq expect req.url == "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2" txresp -body "The second part ;)" } -start varnish v1 -vcl+backend { import example from "${vmod_topbuild}/src/.libs/libvmod_example.so"; sub vcl_deliver { set resp.http.hello = example.hello("World"); } } -start client c1 { txreq -url "/index.php?post/2012/07/25/RPM-Maven-Jenkins-1" rxresp expect resp.http.hello == "Hello, World" expect resp.bodylen == 18 txreq -url "/index.php?post/2012/08/02/RPM-Maven-Jenkins-2" rxresp expect resp.http.hello == "Hello, World" expect resp.bodylen == 18 } varnish v1 -expect cache_miss == 0 varnish v1 -expect cache_hit == 0 client c1 -run varnish v1 -expect cache_miss == 2 varnish v1 -expect cache_hit == 0 client c1 -run client c1 -run client c1 -run varnish v1 -expect cache_miss == 2 varnish v1 -expect cache_hit == 6
Conclusion
Test preparations, executions and assertions all over the place, didn’t I warn you ? Yes I did, however, it doesn’t mean that you can’t write readable tests. On the contrary, the varnishtest
DSL is really expressive, especially once you know how to read it. It’s also easy to write, even though your test case is exploded (servers + varnish + clients), the isolation of those components makes it simpler IMHO. I should probably mention that varnishtest
is not suited for performance tests by design.
This is only an introduction to varnishtest
but I hope this will answer questions you might ask yourself when writing your first test cases.