Varnish

varnish.js is a library to help with the management of varnish servers.

from: varnish-cache.org

Varnish is a web accelerator. You install it in front of your web application and it will speed it up significantly. 

Note: This library only targets Varnish 3.x and above. For clarity on the feature please refer to the documentation

README Contents

• Features
• Installation
• Usage
• Api
  • Server
  • Admin
  • VCL
  • Stat
  • Top
  • Log
• Resources
• License

npm install varnish

You don't need varnish on the same machine or even at all to use this package. In order to run the integration tests you will need varnish running locally. Check out the documentation and installation.

var varnish = require('varnish');

Discover

varnish.discover(function(err, servers){
	servers.forEach(function(server){
		console.log(server.pid);
	});
});

Create

var server = varnish.create();

server
	.id('test')
	.name('mine')
	.storage('malloc')
	.shmlog("200M")
	.workers('2,500,300')
	.pidfile('/var/run/varnish/example.pid')
	.secret('/hidden/file')
	.listen(':9000')
	.listen('proxy:8383')
	.management('localhost:4242')
	.vcl('/etc/varnish/default.vcl')
	.ttl(900)
	.param('max_restarts', 10);
	
server.start(function(err, server){
	console.log(server.pid);
});

Manage

var admin = server.admin();

admin
	.ping()
	.start()
	.stop()
	.backend()
	.use()
	.kill();

Analyse

var stats = server.stats({fields: ['client_conn', 'client_req', 'cache_hit']});

stats.on('data', function(json){
	json.timestamp
	json.cache_hit.value;
	json.cache_hit.persec;
	json.description
});

stats.kill();

Configure

var vcl = varnish.vcl();

vcl.recv(function(resp, req){
  var header = req.http("Accept-Encoding");
  this.if(req.url.like("\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$"))
    .comment("No point in compressing these")
    .unset(header)
  .elseif(header.like('gzip'))
    .set(header, "gzip")
  .elseif(header.like("deflate"))
    .set(header, "deflate")
  .else()
    .comment("unknown algorithm")
    .unset(header);
});

admin.inline('refname', vcl, function(err){
	if(err) return next(err);
	admin.use('refname', function(err){
		
	});
});
	

varnish.get()

Find a varnishd instance based on id or instance name (-n). A function can be passed as a filter. The first server matches wins.

varnish.get(pid, callback);
varnish.get(name, callback); // -n referenced
varnish.get(fn, callback); // filter function

varnish.discover()

varnish.discover();
varnish.discover({filter: fn});
varnish.discover({filter: [fn,fn]})

filters receive a process obj.

• pid: pid
• ppid: Parent pid
• user: user
• command: command running
• args: string of args passed to command

varnish.create()

A helper class for wrapping the generation of command line options to create a varnishd instance. For more clarity on the options to pass please refer to the varnishd documentation.

Configuration options cannot be changed once a server has been initialized and has a pid. Attempts to change will be ignored.

• pid: false
• path: '/usr/local/sbin/varnishd'
• running: false
• child: false

Server.listen(host)

Listen for client requests on the specified address and port. The address can be a host name (“localhost”), an IPv4 dotted-quad (“127.0.0.1”), or an IPv6 address enclosed in square brackets (“[::1]”). If address is not specified, varnishd will listen on all available IPv4 and IPv6 interfaces. If port is not specified, the default HTTP port as listed in /etc/services is used. Multiple listening addresses and ports can be specified.

server
  .listen(':9000')
	.listen('host.com:8888')
	.toString(); // 'varnishd -a :9000,host.com:8888'

Server.backend(host)

Use the specified host as backend server. If port is not specified, the default is 8080

server
	.backend('backend.com')
	.toString(); // 'varnishd -b backend.com'

Server.compile()

Print VCL code compiled to C language and exit. Specify the VCL file to compile with the -f option (Server.vcl).

server
	.compile
	.toString(); // 'varnishd -C'

Server.debug()

Enables debugging mode: The parent process runs in the foreground with a CLI connection on stdin/stdout, and the child process must be started explicitly with a CLI command. Terminating the parent process will also terminate the child.

server
	.debug()
	.toString(); // 'varnishd -d'

Server.foreground()

Run in the foreground.

server
	.foreground()
	.toString(); // 'varnishd -F'

Server.vcl(vclfile)

Use the specified VCL configuration file instead of the builtin default. See vcl for details on VCL syntax. When no configuration is supplied varnishd will not start the cache process.

server
	.vcl('/varnish/vcl/test.vcl')
	.toString(); // 'varnishd -f /varnish/vcl/test.vcl'

Server.group(group)

Specifies the name of an unprivileged group to which the child process should switch before it starts accepting connections. This is a shortcut for specifying the group run-time parameter.

server
	.group('admin')
	.toString(); // 'varnishd -g admin'

Server.hash()

Specifies the hash algorithm.

• simple_list: A simple doubly-linked list. Not recommended for production use.
• classic[,buckets]: A standard hash table. This is the default. The hash key is the CRC32 of the object's URL modulo the size of the hash table. Each table entry points to a list of elements which share the same hash key. The buckets parameter specifies the number of entries in the hash table. The default is 16383.
• critbit: A self-scaling tree structure. The default hash algorithm in 2.1. In comparison to a more traditional B tree the critbit tree is almost completely lockless.

server
	.hash('classic,16383')
	.toString(); // 'varnishd -h classic,16383'

Server.id()

Specify the identity of the varnish server. This can be accessed using server.identity from VCL

server
	.id('integration')
	.toString(); // 'varnishd -i integration'

Server.shmlog()

Specify size of shmlog file. Scaling suffixes like 'k', 'm' can be used up to (e)tabytes. Default is 80 Megabytes. Specifying less than 8 Megabytes is unwise.

server
	.shmlog('80M')
	.toString(); // 'varnishd -l 80M'

Server.name()

Specify a name for this instance. Amonst other things, this name is used to construct the name of the directory in which varnishd keeps temporary files and persistent state. If the specified name begins with a forward slash, it is interpreted as the absolute path to the directory which should be used for this purpose.

server
	.name('ref')
	.toString(); // 'varnishd -n ref'

Server.pidfile()

Write the process's PID to the specified file.

server
	.pidfile('/var/run/varnish.pid')
	.toString(); // 'varnishd -P /var/run/varnish.pid'

Server.param()

Set the parameter specified by param to the specified value. See Run-Time Parameters for a list of parameters. This option can be used multiple times to specify multiple parameters.

server
  .param('ban_lurker_sleep', 100)
  .param('cli_buffer', 64000)
  .toString(); // 'varnishd -p ban_lurker_sleep=100 -p cli_buffer=64000'

Server.secret()

Path to a file containing a secret used for authorizing access to the management port.

server
  .secret('/secret.file')
  .toString(); // 'varnishd -S /secret.file'

Server.storage()

Use the specified storage backend. See Storage Types for a list of supported storage types. This option can be used multiple times to specify multiple storage files. You can name the different backends. Varnish will then reference that backend with the given name in logs, statistics, etc.

server
  .storage('malloc,500M)
  .toString(); // 'varnishd -s malloc,500M'

Server.management()

Offer a management interface on the specified address and port. See Management Interface for a list of management commands.

server
  .management('localhost:5421')
  .toString(); // 'varnishd -T localhost:5421'

Server.reverse()

Connect to this port and offer the command line interface. Think of it as a reverse shell. When running with -M and there is no backend defined the child process (the cache) will not start initially.

server
  .reverse('localhost:1245')
  .toString(); // 'varnishd -M localhost:1245'

Server.ttl()

Specifies a hard minimum time to live for cached documents. This is a shortcut for specifying the default_ttl run-time parameter.

server
  .ttl(500)
  .toString(); // 'varnishd -t 500'

Server.user()

Specifies the name of an unprivileged user to which the child process should switch before it starts accepting connections. This is a shortcut for specifying the user run- time parameter.

If specifying both a user and a group, the user should be specified first.

server
  .user('nw')
  .toString(); // 'varnishd -u nw'

Server.version()

Display the version number and exit.

server
  .version()
  .toString(); // 'varnishd -V'

Server.workers()

Start at least min but no more than max worker threads with the specified idle timeout. This is a shortcut for specifying the thread_pool_min, thread_pool_max and thread_pool_timeout run-time parameters.

If only one number is specified, thread_pool_min and thread_pool_max are both set to this number, and thread_pool_timeout has no effect.

server
  .workers('2,500,300')
  .toString(); // 'varnishd -w 2,500,300'

Server.start()

Server.kill()

Server.admin()

Server.stat()

Server.log()

Admin client for varnishadm

var varnish = require('varnish')
  , admin = new varnish.Admin(host, port, options);

Options:

• name {String} name of the instance to connect to
• auth {String||Buffer} content of secret file for varnish
• file if auth not present will load file contents.
• auto connect {Boolean}

Using the Server class you can also create an Admin instance.

var admin = server.admin();

The nice thing about creating it from the server is all properties set that apply to the admin class will be passed through automatically if the server has already been initialized and is running.

Events

• connect: when a socket connection has been established with the management port.
• authenticated: when the authentication challenge from management port has been responded to correctly.
• close: when connection to the admin port has been closed.
• error: any network related error.

Errors

Command Errors

All errors that occur talking directly to the admin port will be handled in a callback and will not emit an error event.

error instanceof Error in addition they have the a code property.

• 100: 'Syntax Error'
• 101: 'Unknown Request'
• 102: 'Not Implemented'
• 104: 'Too Few Parameters'
• 105: 'Too Many Parameters'
• 106: 'Bad Parameter'
• 107: 'Authentication Required'
• 200: 'OK'
• 201: 'Truncated'
• 300: "Can't Perform Operation"
• 400: "Communication Error"
• 500: "Close"

not implemented by varnishadm

• 800: 'Response Parsing Error'

Admin.connect()

Use when autoconnect flag == false

Admin.send(command, arg1, arg2, Function)

Send command to varnishadm. This is talking to the socket directly.

admin.send('help', function(err, resp){
	
});

admin.send('vcl.load', 'dosattack', '/path/to/file/');
admin.send('vcl.use dosattack');

resp in all cases will be a string of the complete response from the admin.

Command Wrappers

The following methods wrap the send command and parse the output of the response for you. As a result all callbacks passed to these method will receive three parameters.

• error {Error} object or null
• resp parsed {Object} if error == null
• raw the string response from the socket.

Admin.ping(callback)

Ping backend

admin.ping(function(err, pong){
	if(!pong) // we have problems
})

Admin.status(callback)

Check if child state == 'running'

admin.status(function(err, running){
	if(!running) // we need to start it
})

Admin.start(callback)

Starts child process (sets state == 'running')

admin.start(function(err){
	// started if no error
});

Admin.stop(callback)

Stops child process (sets state == 'stopped')

admin.stop(function(err){
	// stopped if no error
});

Admin.list(callback)

Returns list of loaded vcl files

VCL {Object}

• name {String} name of vcl
• num {String} ???
• status {String} 'active|available|boot'

admin.list(function(err, list){

console.log(list);	
/*
 [ {
     "status": "available",
     "num": "0",
     "name": "old"
   },
   {
     "status": "active",
     "num": "7",
     "name": "super_cache"
   }
 ] */
});

Admin.load(reference, path, callback)

Load a vcl from a file

Admin.use(reference, callback)

Make a vcl config active.

Admin.inline(reference, inline, callback)

Make a vcl config active.

Admin.discard(reference, callback)

Discard a loaded vcl

Admin.settings(name, callback)

Get details on varnish configuration parameters

Param:

• value: {String} current value
• unit: {String} the unit type varnish expects
• default: {String} default value
• description: {String} description

Complete List of Parameters

{
  auto_restart: {
    value: 'on',
    unit: 'bool',
    default: 'on [bool]',
    desc: 'Restart child process automatically if it dies.'
  },
  cli_buffer: {
    value: '8192',
    unit: 'bytes',
    default: '8192 [bytes]',
    desc: 'Size of buffer for CLI input. You may need to increase this if you have big VCL files and use the vcl.inline CLI command. NB: Must be specified with -p to have effect.'
  },
  cli_timeout: {
    value: '10',
    unit: 'seconds',
    default: '10 [seconds]',
    desc: 'Timeout for the childs replies to CLI requests from the master.'
  }
}

Admin.set(parameter, new, callback)

Set a parameter

admin.set('cli_timeout', 20, function(err){
  
});

Admin.panic(callback)

Check for latest panic

Admin.clear(callback)

Clears the last panic

Admin.storage(callback)

Get list of current storage w/ type

admin.storage(function(err, devices){
	console.log(devices);
	/* {
    "storage.Transient": "malloc",
    "storage.s0": "malloc"
  } */
})

Admin.backend(callback)

List of all backend currently referenced in loaded VCLs

Backend:

• name {String}
• host {String}
• port {String}
• refs {String} number of vcls pointing at backend
• admin {String}
• status {String}
• passed {String}
• window {String}

app.backend(function(err, backends){
	console.log(backends);
	
	 /* [
    {
      "name": "prod",
      "host": "108.166.39.30",
      "port": "80",
      "refs": "1",
      "admin": "probe",
      "status": "Sick",
      "passed": "1",
      "window": "5"
    },
    {
      "name": "api",
      "host": "198.61.245.73",
      "port": "80",
      "refs": "6",
      "admin": "probe",
      "status": "Healthy",
      "passed": "5",
      "window": "5"
    }
  ] */
	
})

Admin.ban(url, callback)

Ban urls/rules from cache

A chain-able interface that allows vcl files to be written in javascript.

Stats client for varnishstat

var varnish = require('varnish')
  , stat = new varnish.Stat(options);

Options:

• fields {Array} array of varnish fields to collect stats on
• every sample rate in second units.
• name instance of varnish to listen for

Events

• data: stats object
  • timestamp: {Date} object normalized to current timezone.
  • fields: each field passed in options will be available. Each will have the following properties.
    • value: the current value of the field
    • persec: the normalized value over the last sec
    • description: description about the field
• exit: when connection to stats has been closed.
• error: any data received on stderr. This is not an Error object. It is a Buffer.

Stat.list(callback) static

Stat.list(function(err, list){
  console.log(list);
  /*
  {
    client_conn: "Client connections accepted"
  , client_drop: "Connection dropped, no sess/wrk"
  , client_req: "Client requests recieved"
  , cache_hit: "Cache hits"
  // truncated 
  }
  */
})

Complete List of Stat Fields

Stat.kill(callback)

(The MIT License)

Copyright (c) 2013 Nathan White [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.