Creating a Varnish 4 module
Don’t let the title fool you, Varnish 4 hasn’t been released yet. However, the master branch (which holds Varnish 4 developments) in its current state should be close enough to what the first release will look like. I’ve been bitten by API changes several times, but that’s the deal when writing code against work in progress[1].
Note
[1] Last tested against the revision d0c0ee9
What’s new in Varnish 4 ?
I will simply copy/paste an answer from Varnish Software (blame my laziness on this one):
The 4.0 release is underway and should be released after the summer, with a prerelease during the summer, hopefully. The main new features are:
- New logging framework (see this blog post https://www.varnish-software.com/blog/varnishlog-query-language)
- Increased performance (multiple acceptor threads, amongst others)
- HTTP 2.0 compatible architecture. HTTP 2.0 is still far away, but the internals are being reworked in order to fit with what the specs are going to look like
- Directors in vmods. You can now write a director in a vmod. Should make it a lot easier to write a new director.
In addition there are numerous improvements that will allow us to add features quicker.
The last point about directors is the one I’m interested in in this post. The fact that directors are now declared programmatically offers even more flexibility. You can now make Varnish serve static contents by simply writing a director that fetches resources from the filesystem instead of over HTTP, or can you ? Anyway, this changeset doesn’t really give any clue on what’s new for VMOD developers.
What’s new for VMOD developers ?
The short answer is:
Most probably more than I could tell you.
So I’ll do my best to list and describe significant changes for VMODs, but I do not claim completeness. As my experience with VMODs is a bit limited to those I’ve built, so is my knowledge of Varnish internals. I’ll try to point changes in the code related to the changes listed above.
New logging framework
I don’t know whether it is actually related, but the logging API changed. With Varnish 3 you would use WSP
and WSPR
macros:
struct sess *sp; [...] WSP(sp, SLT_Debug, "%s", "printf-like logging");
Varnish 4 introduces two equivalent functions, respectively VSLb
and VSLbt
. It works exactly the same, besides being functions instead of macros, except that the sess
pointer is replaced by a vsl_log
pointer:
struct vsl_log *vsl; [...] VSLb(vsl, SLT_Debug, "%s", "printf-like logging");
With this, varnishtest
introduces a new expectlog
instruction and a new -errvcl
argument that seems to replace the -badvcl
one for Varnish servers, with the expected message explaining why the VCL won’t compile.
Architecture
The changes in the architecture to allow alternative protocols like SPDY (which is the starting point of the HTTP 2.0 draft) also mean a lot of changes in the code.
Function signatures
Varnish 3 passes a sess
pointer to the VMOD functions, it is replaced by a vrt_ctx
pointer in Varnish 4. This structure is described as a composite object, it holds other objects (req, bereq, …) needed throughout the VCL.
Includes
Some files have moved, including headers. This is not a big deal, but at least I’ve noticed a new bin/varnishd/cache/
directory containing the cache.h
header I use a lot, and all corresponding cache_*.c
.
Objects
This time I’m talking about objects as in Object-Oriented Programming. The kind of programming that talks to the enterprisey developer I am. With Varnish 4, you can now manipulate objects in your VCL. But don’t run away, it’s just a means to keep track of state.
With Varnish 3, you can make stateful functions in your VMODs, and you would use Private Pointers to keep track of state. With Varnish 4, you can also declare objects that will encapsulate both state and behavior (nothing fancy like inheritance or whatever sounds too much enterprisey). The key difference is that it creates a named variable in the VCL, where private pointers OTOH must be handled entirely in the VMOD space (in any case Varnish does the housekeeping).
Directors in VMODs
This change means a lot for both Varnish users and VMOD creators. Instead of declaring backends and directors declaratively, you need to declare them in the vcl_init
and vcl_fini
functions. You’ll lose a bit of readability, a small cost for the flexibility it gives : you can now have not only backends in a director, but also other directors.
VCL
From the man page:
A director is a logical group of backend servers clustered together for redundancy. The basic role of the director is to let Varnish choose a backend server amongst several so if one is down another can be used.
With Varnish 3 you create directors declaratively in similar fashion to backends:
backend node1 { .host = "192.168.1.1"; .port = "http"; } backend node2 { .host = "192.168.1.2"; .port = "http"; } director my_cluster round-robin { { .backend = node1; } { .backend = node2; } } sub vcl_recv { set req.backend = my_cluster; }
With Varnish 4 it’s different, you would do it this way:
import directors; backend node1 { .host = "192.168.1.1"; .port = "http"; } backend node2 { .host = "192.168.1.2"; .port = "http"; } sub vcl_init { new rr = directors.round_robin(); rr.add_backend(node1); rr.add_backend(node2); } sub vcl_recv { set req.backend = rr.backend(); }
You may remember I said earlier that you can now put directors behind other directors. This allows new network topologies behind your Varnish like active/passive clusters with a combination of fallback and round robin directors.
import directors; backend active_node1 { .host = "192.168.1.1"; .port = "http"; } backend active_node2 { .host = "192.168.1.2"; .port = "http"; } backend passive_node1 { .host = "192.168.2.1"; .port = "http"; } backend passive_node2 { .host = "192.168.2.2"; .port = "http"; } sub vcl_init { new active_rr = directors.round_robin(); active_rr.add_backend(active_node1); active_rr.add_backend(active_node2); new passive_rr = directors.round_robin(); passive_rr.add_backend(passive_node1); passive_rr.add_backend(passive_node2); new fb = directors.fallback(); fb.add_backend(active_rr); fb.add_backend(passive_rr); } sub vcl_recv { set req.backend = fb.backend(); }
Build scripts
Since directors have been moved to a director VMOD, the std is not the sole VMOD in the Varnish source tree anymore (there’s also a new debug module).
The VMOD build scripts have been moved to lib/libvcl/
, a small change to do in your makefile.
Objects
To declare objects, in the VCC, the syntax looks like:
Module blog Object my_object( [...] ) { Method VOID .do_something( [...] ); Method STRING .return_something( [...] ); } # [...] means you can add parameter types
The generated header looks like:
/* * NB: This file is machine generated, DO NOT EDIT! * * Edit vmod.vcc and run vmod.py instead */ struct vrt_ctx; struct VCL_conf; struct vmod_priv; struct vmod_blog_my_object; VCL_VOID vmod_my_object__init(const struct vrt_ctx *, struct vmod_blog_my_object **, const char *[, ...]); VCL_VOID vmod_my_object__fini(struct vmod_blog_my_object **); VCL_VOID vmod_my_object_do_something(const struct vrt_ctx *, struct vmod_blog_my_object *[, ...]); VCL_STRING vmod_my_object_return_something(const struct vrt_ctx *, struct vmod_blog_my_object *[, ...]);
Well, this is C, so you don’t really have objects. Instead, there is a vmod_<module_name>_<object_name>
structure you need to define, this is where you keep track of state. The structure is then transmitted to the object methods, that is to say functions following the vmod_<module_name>_<object_name>_<method_name>
naming scheme.
Varnish expects two special functions (init
and fini
) which act as constructor and destructor. Free nitpick, the destructor call doesn’t seem to be implemented yet (more on that later).
In your VCL, you would use the new keyword new
to instantiate a named object:
sub vcl_init { new my_instance = blog.my_object( [...] ); } sub vcl_recv { my_instance.do_something( [...] ); }
This is stateful made easy !
A word on objects
The funny thing with Varnish is that the code base looks Object Oriented-ish. A lot of objects consists in a structure with a magic
field that serves as a sort-of weak runtime type system. Let’s take the same example as in my previous post: the workspace structure.
First, there is the structure, holding the state:
struct ws { unsigned magic; #define WS_MAGIC 0x35fac554 unsigned overflow; /* workspace overflowed */ const char *id; /* identity */ char *s; /* (S)tart of buffer */ char *f; /* (F)ree/front pointer */ char *r; /* (R)eserved length */ char *e; /* (E)nd of buffer */ };
And then you get all the WS_*
functions that serve as methods, they all take a struct ws*
as the first argument (call that this
or self
or whatever your favorite OOP language chose to reference the current instance). The WS_Init
function acts as some sort of constructor.
Of course it doesn’t provide the same level of visibility than say, Java. And I could easily mess with a ws
structure and perform direct writes to its fields ignoring the methods. That is not the case for the workspace API, but private methods are emulated with static functions. If there were private methods for ws
objects, the function names would probably be in lower case (eg. static ws_garbage_collect
).
It’s also probably worth mentioning the CHECK_OBJ_NOTNULL
macro that takes a pointer and a magic number, and check whether the pointer is not null and seems to reference the right structure (there’s no guarantee, but still a valuable check). And also the CAST_OBJ_NOTNULL
is similar but allows to copy and cast the reference to another pointer. Two other macros come handy for such structures, ALLOC_OBJ
and FREE_OBJ
. Better know them when working with backends.
Working with backends
Varnish 4 introduces a new type on the VMOD side: VCL_BACKEND
. The name is a bit misleading though, because it isn’t defined as a pointer to a backend
structure, but to a director
structure instead. This is because backends are abstracted to directors, which gives the flexibility to put a director behind another director as shown above.
As for backends themselves, they are wrapped into a vdi_simple
structure, which is a single-backend director.
Creating a Varnish 4 module then ?
After this short introduction, it is finally time to build the VMOD, but what VMOD ?
The goal
The one thing I’d always wanted to do since I know Varnish is to use it to serve static contents. But this is Varnish Cache, and by design it is meant to only speak HTTP. I’ve even written a VCL to do that, using std.fileread
, but it doesn’t work with binary files, and it didn’t work properly even with text files, and it was fun but pointless. So this time, I’ve tried to leverage the new programmatic directors feature to achieve this, but it failed. The VMOD is called libvmod-fsdirector
.
What The Fail ?
A quick side note to explain how and why I’ve failed. In order to create a director, you need a director
structure:
struct director { unsigned magic; #define DIRECTOR_MAGIC 0x3336351d const char *name; char *vcl_name; vdi_getfd_f *getfd; vdi_fini_f *fini; vdi_healthy *healthy; void *priv; };
It looks quite self-describing in my opinion. You need a function to provide a file descriptor that your director knows how to pick. Another function is used to know whether the director is healthy or not (it depends on you definition of healthy). You are also given a private pointer to keep track of your state and you have to provide a function to clean it up. If you want to write a director like the built-in ones, you can’t reuse the function that take care of all the boilerplate.
So this looks fairly simple, right ? Yes but… There always is a “but”. The problem here is that I’m trying to build a backend through the director facility. I’m trying to create a server thread AND a varnish backend, all with the director API. If you haven’t spotted the problem, please re-read the previous statement.
First attempt – naive
Create an object with a method that returns a VCL_BACKEND
, just like built-in directors. Everything is created once in the constructor. Since we’re not actually creating a director per-se (we are not putting existing backends or directors behind it) we need a dummy backend.
import fsdirector; backend dummy { .host = "127.0.0.1"; .port = "8080"; } sub vcl_init { new fs = fsdirector.file_system(); } sub vcl_recv { set req.backend = fs.backend(); }
This one failed because I’ve tried to programmatically create the backend, register it, and return it as a genuine native backend. It probably failed because registration needed a CLI
object. Anyway, it was fun to try, and at least it made obvious to me that I wasn’t trying to build a director at all.
Second attempt – pragmatic
Declare a backend in the VCL, pass it to the VMOD so that a server thread can be created accordingly. No director involved. I’ve also tried to enhance the backend, but it failed. If the backend had a nul port, I would bind a dynamic port in the server thread and change the port in the vrt_backend
structure but it didn’t work. So instead I can only declare a backend with the desired port and the server thread will listen to this very port.
import fsdirector; backend static { .host = "127.0.0.1"; .port = "8080"; } sub vcl_init { new fs = fsdirector.file_system(static, "/var/www"); }
This second attempt which succeded (minus dynamic binding) is the actual goal this module is trying to reach. We just have to build it then.
The build
Since we’re going to make VMOD for Varnish 4, we need a few changes in the build system. I’m talking of course about libvmod-example‘s build system.
autoconf
In the configure.ac
file, we need to ensure we have a Varnish *4* source tree. At least I have decided I wanted to make sure of that, this part is actually optional.
You can use this to
check the new VMOD facility:
AC_CHECK_FILE( [$VARNISHSRC/lib/libvcl/vmodtool.py], [], [AC_MSG_FAILURE(["$VARNISHSRC" is not a Varnish 4 source directory])] )
automake
In the src/Makefile.am
, we can then use this very script to generate the vcc_if.*
files:
vcc_if.c vcc_if.h: vmod_fsdirector.vcc @PYTHON@ $(VARNISHSRC)/lib/libvcl/vmodtool.py vmod_fsdirector.vcc
I’ve also decided to tweak the includes to add varnishd
in my include path in order to (lazily) include "cache/cache*.h"
instead of "bin/varnishd/cache/cache*.h"
.
INCLUDES = -I$(VARNISHSRC)/include \ -I$(VARNISHSRC)/bin/varnishd \ -I$(VARNISHSRC)
And since it’s using libmagic
which definitely *is* magic, let’s add the flag:
libvmod_fsdirector_la_LIBADD = -lmagic
vcc
As stated before, I didn’t manage to create a proper director, so instead of having something similar to what we find in vmod-directors
, we simply declare an object without methods (only a constructor and the implicit destructor).
Module fsdirector Object file_system(BACKEND, STRING) { }
The implementation
Unlike the name suggests, fsdirector does not provide a new director, this approach failed. Probably my wrongdoing… So instead, this VMOD spawns a thread that will listen to requests and serve responses from the filesystem. You only need to declare a backend on localhost, and pass it to the constructor and the module creates a server thread according to the backend’s properties.
backend static { .host = "127.0.0.1"; .port = "8080"; } sub vcl_init { new fs = fsdirector.file_system(static, "/var/www"); }
With this sample, we create a server thread that listens on the port 8080
and serves files from /var/www
.
The object
This is probably not the best example, since the object has no method, but the point is still valid since there implicitly is a destructor. Since we declare an object, we need to provide the structure that will hold the state between method calls. In this module it looks like this:
struct vmod_fsdirector_file_system { unsigned magic; #define VMOD_FSDIRECTOR_MAGIC 0x94874A52 const char *root; struct vdi_simple *vs; int sock; struct vss_addr **vss_addr; char port[6]; char sockaddr_size; struct sockaddr_in sockaddr; pthread_t tp; struct worker *wrk; struct http_conn htc; magic_t magic_cookie; char *thread_name; char *ws_name; };
The first field is not required, but doesn’t cost much and can be used for defensive programming to make sure you are given the right structure. There are various fields, some of them using Varnish data structures, such as the vile vdi_simple
.
The vdi_simple
gate
As I explained before, we are not supposed to manipulate backends directly, they are wrapped inside a vdi_simple
structure so that everything can be handled as directors. This is a nice approach, except that it prevents from reading the backend’s data in the VMOD space. So I came up with a brittle workaround that consists in copying the structure in the VMOD’s source:
/*-------------------------------------------------------------------- * Stolen from bin/varnishd/cache/cache_backend.c */ struct vdi_simple { unsigned magic; #define VDI_SIMPLE_MAGIC 0x476d25b7 struct director dir; struct backend *backend; const struct vrt_backend *vrt; };
Notice the backend
and vrt_backend
structure ? That’s a lot of code to read that piles up to understand which is responsible for what. As for the vdi_simple
object, I can retrieve such a structure from the priv
field of the director
structure, the one I get from the VCL.
// include/vrt.h typedef struct director * VCL_BACKEND; // bin/varnishd/cache/cache_backend.h struct director { unsigned magic; #define DIRECTOR_MAGIC 0x3336351d const char *name; char *vcl_name; vdi_getfd_f *getfd; vdi_fini_f *fini; vdi_healthy *healthy; void *priv; };
So how do we use all that in the constructor ? Let’s write the implementation of the generated signature of vmod_file_system__init
:
VCL_VOID vmod_file_system__init(const struct vrt_ctx *ctx, struct vmod_fsdirector_file_system **fsp, const char *vcl_name, VCL_BACKEND be, const char *root) { struct vmod_fsdirector_file_system *fs; struct vdi_simple *vs; AN(ctx); AN(fsp); AN(vcl_name); AZ(*fsp); CHECK_OBJ_NOTNULL(be, DIRECTOR_MAGIC); CAST_OBJ_NOTNULL(vs, be->priv, VDI_SIMPLE_MAGIC); ALLOC_OBJ(fs, VMOD_FSDIRECTOR_MAGIC); AN(fs); *fsp = fs; fs->vs = vs; fs->thread_name = malloc(sizeof("fsthread-") + strlen(vcl_name)); fs->ws_name = malloc(sizeof("fsworkspace-") + strlen(vcl_name)); AN(fs->thread_name); AN(fs->ws_name); sprintf(fs->thread_name, "fsthread-%s", vcl_name); sprintf(fs->ws_name, "fsworkspace-%s", vcl_name); AN(root); assert(root[0] == '\0' || root[0] == '/'); fs->root = root; [...] server_start(fs); }
First, we have a few checks on the parameters, nothing unusual. The fsp
pointer is expected to be NULL
since it’s the duty of the constructor to instantiate it.
Then, we check the given director holds a vdi_simple
object, and thus a backend, not a true director. Again this is might suffer breaking changes in the future.
The next step allocates an fsdirector
object, which implies a structure starting with a magic
field. This not only saves me complexity by reusing Varnish’s object utilities but I can also reuse the CHECK_OBJ_NOTNULL
macro with my own structure.
For the root
argument, I went super lazy and only checked for either an empty string or something that looks like an absolute path, nothing more. And finally, I can start the server thread.
So of course, anything allocated by the constructor must be freed by the destructor:
VCL_VOID vmod_file_system__fini(struct vmod_fsdirector_file_system **fsp) { struct vmod_fsdirector_file_system *fs; void *res; // XXX It seems that the destructor is not called yet. // A little reminder then... abort(); fs = *fsp; *fsp = NULL; CHECK_OBJ_NOTNULL(fs, VMOD_FSDIRECTOR_MAGIC); AZ(pthread_cancel(fs->tp)); AZ(pthread_join(fs->tp, &res)); assert(res == PTHREAD_CANCELED); [...] free(fs->thread_name); free(fs->ws_name); free(fs->wrk->aws); FREE_OBJ(fs->wrk); FREE_OBJ(fs); }
I haven’t tested this yet for obvious reasons. Anyway, this is just a matter of time until this is implemented. Now for the vdi_simple
usage, here is the server_start
function called in the constructor:
static void server_start(struct vmod_fsdirector_file_system *fs) { struct vdi_simple *vs; const struct vrt_backend *be; vs = fs->vs; be = vs->vrt; AN(VSS_resolve(be->ipv4_addr, be->port, &fs->vss_addr)); fs->sock = VSS_listen(fs->vss_addr[0], be->max_connections); assert(fs->sock >= 0); WRK_BgThread(&fs->tp, fs->thread_name, server_bgthread, fs); }
That’s just a lazy reuse of Varnish’s code to open a socket and start a worker thread (for me it means a thread with a workspace). I’ve written this module for fun, and to dig into Linux system calls and Varnish’s code. This time Varnish won, even though I initially wrote it with the socket and pthread APIs.
The server thread
The worker thread is just a wrapper to a pthread which receives a worker
structure and does whatever a worker does… The previous statement should make it obvious that choosing a Varnish worker thread over a standard pthread was just driven by fun and curiosity…
The other big deal with this server thread is to actually listen to HTTP requests. So after a long thinking (of about half a second) I decided to look at what Varnish has to offer. And surprisingly, I managed to use an http_conn
structure and some HTTP1_*
functions. And without trying too hard, I had a simple HTTP listener:
static void * server_bgthread(struct worker *wrk, void *priv) { struct vmod_fsdirector_file_system *fs; struct sockaddr_storage addr_s; socklen_t len; struct http_conn *htc; int fd; enum htc_status_e htc_status; CAST_OBJ_NOTNULL(fs, priv, VMOD_FSDIRECTOR_MAGIC); assert(fs->sock >= 0); htc = &fs->htc; fs->wrk = wrk; WS_Init(wrk->aws, fs->ws_name, malloc(WS_LEN), WS_LEN); while (1) { do { fd = accept(fs->sock, (void*)&addr_s, &len); } while (fd < 0 && errno == EAGAIN); if (fd < 0) { continue; } HTTP1_Init(htc, wrk->aws, fd, NULL, HTTP1_BUF, HTTP1_MAX_HDR); htc_status = HTTP1_Rx(htc); switch (htc_status) { case HTTP1_OVERFLOW: case HTTP1_ERROR_EOF: case HTTP1_ALL_WHITESPACE: case HTTP1_NEED_MORE: prepare_answer(htc, 400); prepare_body(htc); break; case HTTP1_COMPLETE: answer_appropriate(fs); break; } WS_Reset(wrk->aws, NULL); close(fd); } pthread_exit(0); NEEDLESS_RETURN(NULL); }
Conclusion
For an article I initially intended to keep short, it’s grown a bit more than I expected. If you’re reading this, I hope you’ve enjoyed the post. I’m not covering the rest of the code, since it’s just me playing with APIs I’m not familiar with, and not directly related to Varnish. I am aware of some security issues in the current code, and this is only a proof of concept, and a toy project. It also contains some linuxisms, so don’t expect to build it on any other platform. The code is not TODO-free. You can find the full project on GitHub.