Joe’s Blog!

I recently needed some simple scripts to backup files on various machines, stuff like configs and even some small CouchDB files. Not finding something already out there I put together tens3, two simple scripts to get and put files to Amazon S3. They provide the following:

• uses s3 to backup a directory of files (no subdirectories)
• uses fadvise to be easy on filesystem caches and disks
• purges files after X days
• streams files rather than loading them entirely into memory

They are very simple to use, just create a configuration file together:

amazon_access_key_id: "someid"
amazon_secret_access_key: "somekey"
backup_dir: "/some/path/"
purge_threshold: 3
bucket_name: "somebucket"


Backup a directory of files:

$ ./tens3_put tens3.conf

Restore a file from a backup:

$ ./tens3_get tens3.conf date somefile ./somefile

The date is the date that the file was backed up in a YYYYMMDD format.

Enjoy and let me know if you find any bugs or want new features.

Headed to Boston next week, planning to meet-up next Tuesday (7/28) 7pm at Cambridge Brewing Co. Drop by for a beer, food and maybe a little Erlang.

Inspired by Holger Just’s haproxy configuration tool I decided to write one my own that worked better for my setup and haproxy_join was born. It’s a simple Ruby script that allows you to break up a monolithic haproxy configuration file in to pieces.

haproxy_join expects files and directories to be in the following scheme:

HAPROXY_PATH/conf/global.cfg (file)
HAPROXY_PATH/conf/defaults.cfg (file)
HAPROXY_PATH/conf/frontend.cfg (file)
HAPROXY_PATH/conf/frontend.d (dir of frontend configs)
HAPROXY_PATH/conf/backend.d (dir backend configs)


The HAPROXY_PATH and resulting configuration file are specified when running the haproxy_join command. On most systems it would look like the following, all you need to do is break up your current config and put it in the above structure.

haproxy_join haproxy.cfg /etc/haproxy/


haproxy_join will also attempt to backup your configuration file before writing a new one.

This works great in tandem with a tool like Chef, allowing you to have Chef manage each small configuration file with a template and haproxy_join to concatenate them together each time they are changed. You can achieve this by using a Chef recipe based on the default opscode haproxy recipe and a slightly modified haproxy init script based on Holger Just’s haproxy init script. The recipe will notify haproxy Chef to restart haproxy if a configuration has changed and the init script will run haproxy_join before it restarts haproxy. I have posted an example of the cookbook recipe and the init script. Neither of these have been heavily used/tested so try them out before you put them into production.

Hope this helps anyone with large haproxy configurations. Let me know if you have any questions in the comments.