Creating a Varnish 4 module

Don’t let the title fool you, Varnish 4 hasn’t been released yet. However, the master branch (which holds Varnish 4 developments) in its current state should be close enough to what the first release will look like. I’ve been bitten by API changes several times, but that’s the deal when writing code against work in progress^[1].

What’s new in Varnish 4 ?

I will simply copy/paste an answer from Varnish Software (blame my laziness on this one):

The 4.0 release is underway and should be released after the summer, with a prerelease during the summer, hopefully. The main new features are:

• New logging framework (see this blog post https://www.varnish-software.com/blog/varnishlog-query-language)
• Increased performance (multiple acceptor threads, amongst others)
• HTTP 2.0 compatible architecture. HTTP 2.0 is still far away, but the internals are being reworked in order to fit with what the specs are going to look like
• Directors in vmods. You can now write a director in a vmod. Should make it a lot easier to write a new director.

In addition there are numerous improvements that will allow us to add features quicker.

The last point about directors is the one I’m interested in in this post. The fact that directors are now declared programmatically offers even more flexibility. You can now make Varnish serve static contents by simply writing a director that fetches resources from the filesystem instead of over HTTP, or can you ? Anyway, this changeset doesn’t really give any clue on what’s new for VMOD developers.

What’s new for VMOD developers ?

The short answer is:

Most probably more than I could tell you.

So I’ll do my best to list and describe significant changes for VMODs, but I do not claim completeness. As my experience with VMODs is a bit limited to those I’ve built, so is my knowledge of Varnish internals. I’ll try to point changes in the code related to the changes listed above.

New logging framework

I don’t know whether it is actually related, but the logging API changed. With Varnish 3 you would use WSP and WSPR macros:

struct sess *sp;
[...]
WSP(sp, SLT_Debug, "%s", "printf-like logging");

Varnish 4 introduces two equivalent functions, respectively VSLb and VSLbt. It works exactly the same, besides being functions instead of macros, except that the sess pointer is replaced by a vsl_log pointer:

struct vsl_log *vsl;
[...]
VSLb(vsl, SLT_Debug, "%s", "printf-like logging");

With this, varnishtest introduces a new expectlog instruction and a new -errvcl argument that seems to replace the -badvcl one for Varnish servers, with the expected message explaining why the VCL won’t compile.

Architecture

The changes in the architecture to allow alternative protocols like SPDY (which is the starting point of the HTTP 2.0 draft) also mean a lot of changes in the code.

Function signatures

Varnish 3 passes a sess pointer to the VMOD functions, it is replaced by a vrt_ctx pointer in Varnish 4. This structure is described as a composite object, it holds other objects (req, bereq, …) needed throughout the VCL.

Includes

Some files have moved, including headers. This is not a big deal, but at least I’ve noticed a new bin/varnishd/cache/ directory containing the cache.h header I use a lot, and all corresponding cache_*.c.

Objects

This time I’m talking about objects as in Object-Oriented Programming. The kind of programming that talks to the enterprisey developer I am. With Varnish 4, you can now manipulate objects in your VCL. But don’t run away, it’s just a means to keep track of state.
With Varnish 3, you can make stateful functions in your VMODs, and you would use Private Pointers to keep track of state. With Varnish 4, you can also declare objects that will encapsulate both state and behavior (nothing fancy like inheritance or whatever sounds too much enterprisey). The key difference is that it creates a named variable in the VCL, where private pointers OTOH must be handled entirely in the VMOD space (in any case Varnish does the housekeeping).

Directors in VMODs

This change means a lot for both Varnish users and VMOD creators. Instead of declaring backends and directors declaratively, you need to declare them in the vcl_init and vcl_fini functions. You’ll lose a bit of readability, a small cost for the flexibility it gives : you can now have not only backends in a director, but also other directors.

VCL

From the man page:

A director is a logical group of backend servers clustered together for redundancy. The basic role of the director is to let Varnish choose a backend server amongst several so if one is down another can be used.

With Varnish 3 you create directors declaratively in similar fashion to backends:

backend node1 {
  .host = "192.168.1.1";
  .port = "http";
}
backend node2 {
  .host = "192.168.1.2";
  .port = "http";
}
director my_cluster round-robin {
  { .backend = node1; }
  { .backend = node2; }
}
sub vcl_recv {
  set req.backend = my_cluster;
}

With Varnish 4 it’s different, you would do it this way:

import directors;
backend node1 {
  .host = "192.168.1.1";
  .port = "http";
}
backend node2 {
  .host = "192.168.1.2";
  .port = "http";
}
sub vcl_init {
  new rr = directors.round_robin();
  rr.add_backend(node1);
  rr.add_backend(node2);
}
sub vcl_recv {
  set req.backend = rr.backend();
}

You may remember I said earlier that you can now put directors behind other directors. This allows new network topologies behind your Varnish like active/passive clusters with a combination of fallback and round robin directors.

import directors;
backend active_node1 {
  .host = "192.168.1.1";
  .port = "http";
}
backend active_node2 {
  .host = "192.168.1.2";
  .port = "http";
}
backend passive_node1 {
  .host = "192.168.2.1";
  .port = "http";
}
backend passive_node2 {
  .host = "192.168.2.2";
  .port = "http";
}
sub vcl_init {
  new active_rr = directors.round_robin();
  active_rr.add_backend(active_node1);
  active_rr.add_backend(active_node2);
  new passive_rr = directors.round_robin();
  passive_rr.add_backend(passive_node1);
  passive_rr.add_backend(passive_node2);
  new fb = directors.fallback();
  fb.add_backend(active_rr);
  fb.add_backend(passive_rr);
}
sub vcl_recv {
  set req.backend = fb.backend();
}

Build scripts

Since directors have been moved to a director VMOD, the std is not the sole VMOD in the Varnish source tree anymore (there’s also a new debug module).
The VMOD build scripts have been moved to lib/libvcl/, a small change to do in your makefile.

Objects

To declare objects, in the VCC, the syntax looks like:

Module blog
Object my_object( [...] ) {
	Method VOID .do_something( [...] );
	Method STRING .return_something( [...] );
}
# [...] means you can add parameter types

The generated header looks like:

/*
 * NB:  This file is machine generated, DO NOT EDIT!
 *
 * Edit vmod.vcc and run vmod.py instead
 */
struct vrt_ctx;
struct VCL_conf;
struct vmod_priv;
struct vmod_blog_my_object;
VCL_VOID vmod_my_object__init(const struct vrt_ctx *, struct vmod_blog_my_object **, const char *[, ...]);
VCL_VOID vmod_my_object__fini(struct vmod_blog_my_object **);
VCL_VOID vmod_my_object_do_something(const struct vrt_ctx *, struct vmod_blog_my_object *[, ...]);
VCL_STRING vmod_my_object_return_something(const struct vrt_ctx *, struct vmod_blog_my_object *[, ...]);

Well, this is C, so you don’t really have objects. Instead, there is a vmod_<module_name>_<object_name> structure you need to define, this is where you keep track of state. The structure is then transmitted to the object methods, that is to say functions following the vmod_<module_name>_<object_name>_<method_name> naming scheme.
Varnish expects two special functions (init and fini) which act as constructor and destructor. Free nitpick, the destructor call doesn’t seem to be implemented yet (more on that later).
In your VCL, you would use the new keyword new to instantiate a named object:

sub vcl_init {
	new my_instance = blog.my_object( [...] );
}
sub vcl_recv {
	my_instance.do_something( [...] );
}

This is stateful made easy !

A word on objects

The funny thing with Varnish is that the code base looks Object Oriented-ish. A lot of objects consists in a structure with a magic field that serves as a sort-of weak runtime type system. Let’s take the same example as in my previous post: the workspace structure.
First, there is the structure, holding the state:

struct ws {
        unsigned                magic;
#define WS_MAGIC                0x35fac554
        unsigned                overflow;       /* workspace overflowed */
        const char              *id;            /* identity */
        char                    *s;             /* (S)tart of buffer */
        char                    *f;             /* (F)ree/front pointer */
        char                    *r;             /* (R)eserved length */
        char                    *e;             /* (E)nd of buffer */
};

And then you get all the WS_* functions that serve as methods, they all take a struct ws* as the first argument (call that this or self or whatever your favorite OOP language chose to reference the current instance). The WS_Init function acts as some sort of constructor.
Of course it doesn’t provide the same level of visibility than say, Java. And I could easily mess with a ws structure and perform direct writes to its fields ignoring the methods. That is not the case for the workspace API, but private methods are emulated with static functions. If there were private methods for ws objects, the function names would probably be in lower case (eg. static ws_garbage_collect).
It’s also probably worth mentioning the CHECK_OBJ_NOTNULL macro that takes a pointer and a magic number, and check whether the pointer is not null and seems to reference the right structure (there’s no guarantee, but still a valuable check). And also the CAST_OBJ_NOTNULL is similar but allows to copy and cast the reference to another pointer. Two other macros come handy for such structures, ALLOC_OBJ and FREE_OBJ. Better know them when working with backends.

Working with backends

Varnish 4 introduces a new type on the VMOD side: VCL_BACKEND. The name is a bit misleading though, because it isn’t defined as a pointer to a backend structure, but to a director structure instead. This is because backends are abstracted to directors, which gives the flexibility to put a director behind another director as shown above.
As for backends themselves, they are wrapped into a vdi_simple structure, which is a single-backend director.

Creating a Varnish 4 module then ?

After this short introduction, it is finally time to build the VMOD, but what VMOD ?

The goal

The one thing I’d always wanted to do since I know Varnish is to use it to serve static contents. But this is Varnish Cache, and by design it is meant to only speak HTTP. I’ve even written a VCL to do that, using std.fileread, but it doesn’t work with binary files, and it didn’t work properly even with text files, and it was fun but pointless. So this time, I’ve tried to leverage the new programmatic directors feature to achieve this, but it failed. The VMOD is called libvmod-fsdirector.

What The Fail ?

A quick side note to explain how and why I’ve failed. In order to create a director, you need a director structure:

struct director {
        unsigned                magic;
#define DIRECTOR_MAGIC          0x3336351d
        const char              *name;
        char                    *vcl_name;
        vdi_getfd_f             *getfd;
        vdi_fini_f              *fini;
        vdi_healthy             *healthy;
        void                    *priv;
};

It looks quite self-describing in my opinion. You need a function to provide a file descriptor that your director knows how to pick. Another function is used to know whether the director is healthy or not (it depends on you definition of healthy). You are also given a private pointer to keep track of your state and you have to provide a function to clean it up. If you want to write a director like the built-in ones, you can’t reuse the function that take care of all the boilerplate.
So this looks fairly simple, right ? Yes but… There always is a “but”. The problem here is that I’m trying to build a backend through the director facility. I’m trying to create a server thread AND a varnish backend, all with the director API. If you haven’t spotted the problem, please re-read the previous statement.

First attempt – naive

Create an object with a method that returns a VCL_BACKEND, just like built-in directors. Everything is created once in the constructor. Since we’re not actually creating a director per-se (we are not putting existing backends or directors behind it) we need a dummy backend.

import fsdirector;
backend dummy {
        .host = "127.0.0.1";
        .port = "8080";
}
sub vcl_init {
        new fs = fsdirector.file_system();
}
sub vcl_recv {
        set req.backend = fs.backend();
}

This one failed because I’ve tried to programmatically create the backend, register it, and return it as a genuine native backend. It probably failed because registration needed a CLI object. Anyway, it was fun to try, and at least it made obvious to me that I wasn’t trying to build a director at all.

Second attempt – pragmatic

Declare a backend in the VCL, pass it to the VMOD so that a server thread can be created accordingly. No director involved. I’ve also tried to enhance the backend, but it failed. If the backend had a nul port, I would bind a dynamic port in the server thread and change the port in the vrt_backend structure but it didn’t work. So instead I can only declare a backend with the desired port and the server thread will listen to this very port.

import fsdirector;
backend static {
        .host = "127.0.0.1";
        .port = "8080";
}
sub vcl_init {
        new fs = fsdirector.file_system(static, "/var/www");
}

This second attempt which succeded (minus dynamic binding) is the actual goal this module is trying to reach. We just have to build it then.

The build

Since we’re going to make VMOD for Varnish 4, we need a few changes in the build system. I’m talking of course about libvmod-example‘s build system.

autoconf

In the configure.ac file, we need to ensure we have a Varnish *4* source tree. At least I have decided I wanted to make sure of that, this part is actually optional.
You can use this to
check the new VMOD facility:

AC_CHECK_FILE(
       [$VARNISHSRC/lib/libvcl/vmodtool.py],
       [],
       [AC_MSG_FAILURE(["$VARNISHSRC" is not a Varnish 4 source directory])]
)

automake

In the src/Makefile.am, we can then use this very script to generate the vcc_if.* files:

vcc_if.c vcc_if.h: vmod_fsdirector.vcc
	@PYTHON@ $(VARNISHSRC)/lib/libvcl/vmodtool.py vmod_fsdirector.vcc

I’ve also decided to tweak the includes to add varnishd in my include path in order to (lazily) include "cache/cache*.h" instead of "bin/varnishd/cache/cache*.h".

INCLUDES = -I$(VARNISHSRC)/include \
           -I$(VARNISHSRC)/bin/varnishd \
           -I$(VARNISHSRC)

And since it’s using libmagic which definitely *is* magic, let’s add the flag:

libvmod_fsdirector_la_LIBADD = -lmagic

vcc

As stated before, I didn’t manage to create a proper director, so instead of having something similar to what we find in vmod-directors, we simply declare an object without methods (only a constructor and the implicit destructor).

Module fsdirector
Object file_system(BACKEND, STRING) {
}

The implementation

Unlike the name suggests, fsdirector does not provide a new director, this approach failed. Probably my wrongdoing… So instead, this VMOD spawns a thread that will listen to requests and serve responses from the filesystem. You only need to declare a backend on localhost, and pass it to the constructor and the module creates a server thread according to the backend’s properties.

backend static {
        .host = "127.0.0.1";
        .port = "8080";
}
sub vcl_init {
        new fs = fsdirector.file_system(static, "/var/www");
}

With this sample, we create a server thread that listens on the port 8080 and serves files from /var/www.

The object

This is probably not the best example, since the object has no method, but the point is still valid since there implicitly is a destructor. Since we declare an object, we need to provide the structure that will hold the state between method calls. In this module it looks like this:

struct vmod_fsdirector_file_system {
        unsigned                 magic;
#define VMOD_FSDIRECTOR_MAGIC    0x94874A52
        const char               *root;
        struct vdi_simple        *vs;
        int                      sock;
        struct vss_addr          **vss_addr;
        char                     port[6];
        char                     sockaddr_size;
        struct sockaddr_in       sockaddr;
        pthread_t                tp;
        struct worker            *wrk;
        struct http_conn         htc;
        magic_t                  magic_cookie;
        char                     *thread_name;
        char                     *ws_name;
};

The first field is not required, but doesn’t cost much and can be used for defensive programming to make sure you are given the right structure. There are various fields, some of them using Varnish data structures, such as the vile vdi_simple.
 

The  vdi_simple gate

As I explained before, we are not supposed to manipulate backends directly, they are wrapped inside a vdi_simple structure so that everything can be handled as directors. This is a nice approach, except that it prevents from reading the backend’s data in the VMOD space. So I came up with a brittle workaround that consists in copying the structure in the VMOD’s source:

/*--------------------------------------------------------------------
 * Stolen from bin/varnishd/cache/cache_backend.c
 */
struct vdi_simple {
        unsigned                magic;
#define VDI_SIMPLE_MAGIC        0x476d25b7
        struct director         dir;
        struct backend          *backend;
        const struct vrt_backend *vrt;
};

Notice the backend and vrt_backend structure ? That’s a lot of code to read that piles up to understand which is responsible for what. As for the vdi_simple object, I can retrieve such a structure from the priv field of the director structure, the one I get from the VCL.

// include/vrt.h
typedef struct director *               VCL_BACKEND;
// bin/varnishd/cache/cache_backend.h
struct director {
        unsigned                magic;
#define DIRECTOR_MAGIC          0x3336351d
        const char              *name;
        char                    *vcl_name;
        vdi_getfd_f             *getfd;
        vdi_fini_f              *fini;
        vdi_healthy             *healthy;
        void                    *priv;
};

So how do we use all that in the constructor ? Let’s write the implementation of the generated signature of vmod_file_system__init:

VCL_VOID
vmod_file_system__init(const struct vrt_ctx *ctx,
    struct vmod_fsdirector_file_system **fsp,
    const char *vcl_name, VCL_BACKEND be, const char *root)
{
        struct vmod_fsdirector_file_system *fs;
        struct vdi_simple *vs;
        AN(ctx);
        AN(fsp);
        AN(vcl_name);
        AZ(*fsp);
        CHECK_OBJ_NOTNULL(be, DIRECTOR_MAGIC);
        CAST_OBJ_NOTNULL(vs, be->priv, VDI_SIMPLE_MAGIC);
        ALLOC_OBJ(fs, VMOD_FSDIRECTOR_MAGIC);
        AN(fs);
        *fsp = fs;
        fs->vs = vs;
        fs->thread_name = malloc(sizeof("fsthread-")    + strlen(vcl_name));
        fs->ws_name     = malloc(sizeof("fsworkspace-") + strlen(vcl_name));
        AN(fs->thread_name);
        AN(fs->ws_name);
        sprintf(fs->thread_name, "fsthread-%s", vcl_name);
        sprintf(fs->ws_name,  "fsworkspace-%s", vcl_name);
        AN(root);
        assert(root[0] == '\0' || root[0] == '/');
        fs->root = root;
        [...]
        server_start(fs);
}

First, we have a few checks on the parameters, nothing unusual. The fsp pointer is expected to be NULL since it’s the duty of the constructor to instantiate it.
Then, we check the given director holds a vdi_simple object, and thus a backend, not a true director. Again this is might suffer breaking changes in the future.
The next step allocates an fsdirector object, which implies a structure starting with a magic field. This not only saves me complexity by reusing Varnish’s object utilities but I can also reuse the CHECK_OBJ_NOTNULL macro with my own structure.
For the root argument, I went super lazy and only checked for either an empty string or something that looks like an absolute path, nothing more. And finally, I can start the server thread.
So of course, anything allocated by the constructor must be freed by the destructor:

VCL_VOID
vmod_file_system__fini(struct vmod_fsdirector_file_system **fsp)
{
        struct vmod_fsdirector_file_system *fs;
        void *res;
        // XXX It seems that the destructor is not called yet.
        //     A little reminder then...
        abort();
        fs = *fsp;
        *fsp = NULL;
        CHECK_OBJ_NOTNULL(fs, VMOD_FSDIRECTOR_MAGIC);
        AZ(pthread_cancel(fs->tp));
        AZ(pthread_join(fs->tp, &res));
        assert(res == PTHREAD_CANCELED);
        [...]
        free(fs->thread_name);
        free(fs->ws_name);
        free(fs->wrk->aws);
        FREE_OBJ(fs->wrk);
        FREE_OBJ(fs);
}

I haven’t tested this yet for obvious reasons. Anyway, this is just a matter of time until this is implemented. Now for the vdi_simple usage, here is the server_start function called in the constructor:

static void
server_start(struct vmod_fsdirector_file_system *fs)
{
        struct vdi_simple *vs;
        const struct vrt_backend *be;
        vs = fs->vs;
        be = vs->vrt;
        AN(VSS_resolve(be->ipv4_addr, be->port, &fs->vss_addr));
        fs->sock = VSS_listen(fs->vss_addr[0], be->max_connections);
        assert(fs->sock >= 0);
        WRK_BgThread(&fs->tp, fs->thread_name, server_bgthread, fs);
}

That’s just a lazy reuse of Varnish’s code to open a socket and start a worker thread (for me it means a thread with a workspace). I’ve written this module for fun, and to dig into Linux system calls and Varnish’s code. This time Varnish won, even though I initially wrote it with the socket and pthread APIs.

The server thread

The worker thread is just a wrapper to a pthread which receives a worker structure and does whatever a worker does… The previous statement should make it obvious that choosing a Varnish worker thread over a standard pthread was just driven by fun and curiosity…
The other big deal with this server thread is to actually listen to HTTP requests. So after a long thinking (of about half a second) I decided to look at what Varnish has to offer. And surprisingly, I managed to use an http_conn structure and some HTTP1_* functions. And without trying too hard, I had a simple HTTP listener:

static void *
server_bgthread(struct worker *wrk, void *priv)
{
        struct vmod_fsdirector_file_system *fs;
        struct sockaddr_storage addr_s;
        socklen_t len;
        struct http_conn *htc;
        int fd;
        enum htc_status_e htc_status;
        CAST_OBJ_NOTNULL(fs, priv, VMOD_FSDIRECTOR_MAGIC);
        assert(fs->sock >= 0);
        htc = &fs->htc;
        fs->wrk = wrk;
        WS_Init(wrk->aws, fs->ws_name, malloc(WS_LEN), WS_LEN);
        while (1) {
                do {
                        fd = accept(fs->sock, (void*)&addr_s, &len);
                } while (fd < 0 && errno == EAGAIN);
                if (fd < 0) {
                        continue;
                }
                HTTP1_Init(htc, wrk->aws, fd, NULL, HTTP1_BUF, HTTP1_MAX_HDR);
                htc_status = HTTP1_Rx(htc);
                switch (htc_status) {
                        case HTTP1_OVERFLOW:
                        case HTTP1_ERROR_EOF:
                        case HTTP1_ALL_WHITESPACE:
                        case HTTP1_NEED_MORE:
                                prepare_answer(htc, 400);
                                prepare_body(htc);
                                break;
                        case HTTP1_COMPLETE:
                                answer_appropriate(fs);
                                break;
                }
                WS_Reset(wrk->aws, NULL);
                close(fd);
        }
        pthread_exit(0);
        NEEDLESS_RETURN(NULL);
}

Conclusion

For an article I initially intended to keep short, it’s grown a bit more than I expected. If you’re reading this, I hope you’ve enjoyed the post. I’m not covering the rest of the code, since it’s just me playing with APIs I’m not familiar with, and not directly related to Varnish. I am aware of some security issues in the current code, and this is only a proof of concept, and a toy project. It also contains some linuxisms, so don’t expect to build it on any other platform. The code is not TODO-free. You can find the full project on GitHub.