Varnish
From Docunext Technology Wiki
Contents |
Summary
Varnish is a reverse proxy cache, and it uses the varnish configuration language (VCL) for configuration.
Features
- ESI - only in trunk as of July 2008? - Now available in Debian Sid, hopefully it will make it into Lenny soon.
- Low maintenance performance
- Rewriting capabilities
Admin Console
After using telnet to connect to the admin port (6082 default) of varnish, the stats command resulted with this for me:
37337 Client connections accepted
24438 Client requests received
987 Cache hits
1223 Cache hits for pass
11715 Cache misses
23277 Backend connections success
0 Backend connections failures
11672 Backend connections reuses
22301 Backend connections recycles
12 Backend connections unused
873 N struct srcaddr
1 N active struct srcaddr
19 N struct sess_mem
2 N struct sess
186 N struct object
414 N struct objecthead
208 N struct smf
35 N small free smf
7 N large free smf
15 N struct vbe_conn
2 N worker threads
489 N worker threads created
0 N worker threads not created
0 N worker threads limited
0 N queued work requests
489 N overflowed work requests
0 N dropped work requests
11549 N expired objects
1 N objects on deathrow
0 HTTP header overflows
0 Objects sent with sendfile
20557 Objects sent with write
37337 Total Sessions
24438 Total Requests
309 Total pipe
0 Total pass
22968 Total fetch
9116032 Total header bytes
164704168 Total body bytes
26144 Session Closed
178 Session Pipeline
0 Session Read Ahead
19470 Session herd
2152594 SHM records
265613 SHM writes
0 SHM MTX contention
20612 allocator requests
166 outstanding allocations
3084288 bytes allocated
101773312 bytes free
22968 Backend requests made
Purge
Example purge:
# telnet localhost 6082 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. url.purge ^/$ 200 10 PURGE ^/$
The varnishstat tool is also very nice.
So I figured out a nice thing today - I use cookies with a lot of my servers to battle spam, and varnish won't cache those by default. I want to cache those even if there is a cookie, so I use this VCL rule from the Varnish wiki:
if (req.request == "GET" && req.url ~ "\.(gif|jpg|swf|css|js)$") {
lookup;
}
This made a huge difference is cache hits for me, as seen here:
1307343 Client requests received
1057349 Cache hits
6589 Cache hits for pass
118692 Cache misses
Failover
I hope to get this working sometime, but so far I haven't been able to get it to work how I wanted it to. I think it might only work for missing content, not when a backend goes down and is unreachable.
VCL Failover Syntax
In varnish, failovers are done (as of Apr 2009 / Varnish 1.1.2) using "restarts". I read that this should work:
sub vcl_fetch {
if (obj.status != 200 && obj.status != 403 && obj.status != 404) {
restart;
}
}
But I get an error:
sudo /etc/init.d/varnish restart
Stopping HTTP accelerator: varnishd.
Starting HTTP accelerator: varnishd failed!
Expected action, 'if' or '}'
(/etc/varnish/default.vcl Line 221 Pos 9)
restart;
--------#######-
Looks like that's the syntax for Varnish 2.x. Unfortunately there were significant changes in the syntax from 1 to 2, but I don't mind that too much. More importantly I'm grateful for awesome open source software like Varnish!
Awesome, I finally figured it out! New section...
Working VCL Failover Example
The docs on the Varnish site aren't very clear, but there is a bug which explains it better. What I needed was to setup polling, aka probe.
- url - the url to test
- timeout - how long to wait (I'm using 30ms for a local host)
- interval - how often to test
- window - how many intervals to consider
- threshold - how many intervals need to be active for the backend to be healthy
Probing NGINX causes 499 http error code due to connection close...
src/http/ngx_http_request.h: /* * HTTP does not define the code for the case when a client closed * the connection while we are processing its request so we introduce * own code to log such situation when a client has closed the connection * before we even try to send the HTTP header to it */ #define NGX_HTTP_CLIENT_CLOSED_REQUEST 499
This was getting caused by NGINX trying to proxy the request. I set a specific location for polling and set a root for it in the nginx conf file.
Now for the working example: Varnish Failover
To-Do Fail-Over Setup
The setup is working well for my services, but I'm still need to document and formalize the setup, so that it is easier to debug. Right now I have three Varnish servers running on separate networks, each setup differently with different back-end arrangements. No fun!
Changing Requests and Responses
Hmmm, beresp is not working on my Varnish installation (2.0.5).
Errors
ESI
No ESI processing, binary object: 0x1f at pos 0.
This wasn't too bad - ESI isn't supported on compressed content. I turned off deflate and it worked! This is pretty awesome. Thanks to:
OK, I got ESI processing working! Hooray! FYI: I installed varnish from debian sid on a system running lenny, and it worked OK.
Now I have Apache2 (dynamic processing) -> varnish (caching) -> nginx (compressing).
VCL
Problem loading compiled VCL program: ./bin.XXoShgzO: undefined symbol: VRT_init_simple_backend
Not sure what the problem with this is yet.
Random Stops?
Lately Varnish has been randomly stopping, so I setup a cron job to automatically restart it. I will eventually use monit or at least check if its not running before it gets restarted. The logs don't tell much:
Mar 14 14:15:09 vpn-comet varnishd[1686]: Child (1688) not responding to ping, killing it. Mar 14 14:15:16 vpn-comet varnishd[1686]: Child (1688) died signal=3 Mar 14 14:15:16 vpn-comet varnishd[1686]: Child cleanup complete Mar 14 14:15:16 vpn-comet varnishd[1686]: child (4500) Started Mar 14 14:15:19 vpn-comet varnishd[1686]: Pushing vcls failed: CLI communication error Mar 14 14:15:19 vpn-comet varnishd[1686]: Child (4500) said Closed fds: 4 5 6 10 11 13 14 Mar 14 14:15:19 vpn-comet varnishd[1686]: Child (4500) said Child starts Mar 14 14:15:19 vpn-comet varnishd[1686]: Child (4500) said managed to mmap 1073741824 bytes of 1073741824 Mar 14 14:15:19 vpn-comet varnishd[1686]: Child (4500) said Ready Mar 14 15:44:25 vpn-comet varnishd[4704]: child (4711) Started Mar 14 15:44:25 vpn-comet varnishd[4704]: Child (4711) said Closed fds: 4 5 6 10 11 13 14 Mar 14 15:44:25 vpn-comet varnishd[4704]: Child (4711) said Child starts Mar 14 15:44:25 vpn-comet varnishd[4704]: Child (4711) said managed to mmap 1073741824 bytes of 1073741824 Mar 14 15:44:25 vpn-comet varnishd[4704]: Child (4711) said Ready
I'm now using monit to restart varnish, but its still failing about once per day.
Future Varnish Features
I'm glad that the Varnish authors are keeping the future open for it. There are a lot of ideas being reviewed and I even tossed one in the pile: CDB support. I'd love to be able to lookup key / value pairs within the VCL process. Alternatively, something like Tokyo Tyrant or Tokyo Cabinet would be great, too!
Related Pages
External Links
- http://wiki.itlinux.cl/doku.php?id=varnish:mini_howto
- My CDB Support Suggestion
- http://vincentfretin.ecreall.com/articles/varnish-guru-meditation-on-timeout