Joe’s Blog!

I recently setup a rsyslog server for my employer. Rsyslog is an enhanced syslog server that is multi-threaded. There is all sorts of filtering and rules that can be setup in the configuration. Cool stuff. In an effort to get as many systems using syslog as I can I needed to track down how to get syslog support built into nginx as a couple of our backend systems use it. The first thing I found was the out of date patch on this page over at the nginx wiki. Unfortunately this doesn’t work for me as I already run 0.7.61. A quick email to the nginx mailing list and I found out about another patch set for 0.7.x and 0.8.x by the same author as the old one. Basically it’s as easy as patching the nginx source with the aforementioned patch and then the standard ./configure, make and make install with one catch. You need to add –with-syslog to your configure. Unfortunately this didn’t seem to work for me I needed to also add the CFLAG to add syslog support in as well by running export CFLAGS=”$CFLAGS -DUSE_SYSLOG”. After that run your build and things should be ready to go. After the make if you want to double check that syslog support got built just run strings objs/nginx | grep openlog if you get “openlog” as a result you should be ready to rock.

Now for your rsyslog setup you may want to have nginx log to its own files rather than the default syslog log file. To do that just add the following to your configuration:

if $programname == 'nginx' and $syslogseverity <= '4' then /var/log/nginx_error.log
if $programname == 'nginx' and $syslogseverity == '5' then /var/log/nginx_access.log
if $programname == 'nginx' and $syslogseverity >= '6' then /var/log/nginx_debug_info.log


This will filter out all the logs for nginx by the severity of the log message. I have all errors go to one (<= 4), access logs (== 5) go to another and debug and info messages (>= 6) go to the last.

For more details on configuring rsyslog check out the wiki.

I’ve been debugging issues with HTTP, my backend servers and HAProxy. After a quick email to the HAProxy mailing list I found out about a configuration option stats socket PATH. This will create a socket you can send commands to and get more information out of HAProxy. To do this I just used some simle unix tools, the key is socat. From the man:

socat is a relay for bidirectional data transfer between two independent data channels. Each of these data channels may be a file, pipe, device (serial line etc. or a pseudo terminal), a socket (UNIX, IP4, IP6 – raw, UDP, TCP), an SSL socket, proxy CONNECT connection, a file descriptor (stdin etc.), the GNU line editor (readline), a program, or a combination of two of these. These modes include generation of “listening” sockets, named pipes, and pseudo terminals.

Here are a few examples of how to use the stats socket. First, you need to add stats socket PATH to your configuration and restart haproxy. You should then find a socket located at the path specified, I used /tmp/haproxy. Now you can send it commands to get more information and stats from HAProxy.

echo "show stat" | socat unix-connect:/tmp/haproxy stdio


This will give you stats on all of your backends and frontends, some of the same stuff you see on the stats page enabled by the stats uri configuration. As an added bonus it’s all in CSV.

echo "show errors" | socat unix-connect:/tmp/haproxy stdio


show errors will give you a capture of last error on each backend/frontend.

echo "show info" | socat unix-connect:/tmp/haproxy stdio


This will give you information about the running HAProxy process such as pid, uptime and etc.

echo "show sess" | socat unix-connect:/tmp/haproxy stdio


This will dump (possibly huge) info about all know sessions.

For more details check out the docs section 9 and stats socket in section 3.1.

Bonus socat fun.

socat is a more full featured cousin of netcat. Both can be used in similar ways, one thing I use them for occasionally is debugging REST and etc. This was a real help when working with an API that didn’t have a library, I could test things out without needing to make erroneous calls to the API. In the simplest case you can have either of them listen on a port and output all the details of the request. To do this with socat run:

socat tcp-listen:8000 stdio

This will listen for connections on port 8000. Doing the same thing with netcat is easy as well:

netcat -l -p 8000

For instance you can see the output from creating a document in CouchDB.

In one terminal:

$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'rest_client'
=> true
irb(main):003:0> RestClient.put("http://localhost:8000/somedb/somedoc", "{\"somekey\": \"somevalue\"}", :content_type => "application/json")


In another run your mock server:

$ socat tcp-listen:8000 stdio
PUT /somedb/somedoc HTTP/1.1
Accept: application/xml
Content-Type: application/json
Accept-Encoding: gzip, deflate
Content-Length: 24
Host: localhost:8000

{"somekey": "somevalue"}


Oh! By the way, if you install netcat from source, don’t compile with -DGAPING_SECURITY_HOLE unless you know what you are doing.

Headed to Boston next week, planning to meet-up next Tuesday (7/28) 7pm at Cambridge Brewing Co. Drop by for a beer, food and maybe a little Erlang.

Inspired by Holger Just’s haproxy configuration tool I decided to write one my own that worked better for my setup and haproxy_join was born. It’s a simple Ruby script that allows you to break up a monolithic haproxy configuration file in to pieces.

haproxy_join expects files and directories to be in the following scheme:

HAPROXY_PATH/conf/global.cfg (file)
HAPROXY_PATH/conf/defaults.cfg (file)
HAPROXY_PATH/conf/frontend.cfg (file)
HAPROXY_PATH/conf/frontend.d (dir of frontend configs)
HAPROXY_PATH/conf/backend.d (dir backend configs)


The HAPROXY_PATH and resulting configuration file are specified when running the haproxy_join command. On most systems it would look like the following, all you need to do is break up your current config and put it in the above structure.

haproxy_join haproxy.cfg /etc/haproxy/


haproxy_join will also attempt to backup your configuration file before writing a new one.

This works great in tandem with a tool like Chef, allowing you to have Chef manage each small configuration file with a template and haproxy_join to concatenate them together each time they are changed. You can achieve this by using a Chef recipe based on the default opscode haproxy recipe and a slightly modified haproxy init script based on Holger Just’s haproxy init script. The recipe will notify haproxy Chef to restart haproxy if a configuration has changed and the init script will run haproxy_join before it restarts haproxy. I have posted an example of the cookbook recipe and the init script. Neither of these have been heavily used/tested so try them out before you put them into production.

Hope this helps anyone with large haproxy configurations. Let me know if you have any questions in the comments.

I recently switched to using Claws Mail from Mozilla Thunderbird. This was mostly to try something new and because of the seeming stagnation of the Thunderbird project. So far so good, here are a couple things that I think are cool.

Lots of plugins and themes. Claws Mail has for all sorts of tasks and features. The ones that I think are key are GTKHTML, Notification (libnotify/notify-osd) and SpamAssassin. SpamAssassin in interesting because it actually uses a full SpamAssassin installation like you would see on a mail server. The plugin connects to the spamd daemon running on your system and you can teach it what is spam and ham. If you are running Ubuntu all/most of the plugins are available in the normal repositories.

Also very cool is its auto-generated mail filters. Basically you can right click any email and create a filter based on the headers. This worked great for all the mailing lists and ticketing systems I use.

Claws also has a network log which worked great for diagnosing issues with an IMAP or SMTP session.

There are a few quirks, the first that I noticed was that when Claws is checking the mail servers for new mail it locks out some of the menu items. This is annoying if an automatic check happens when you are trying to change a config item. Another annoyance is when using the up and down keys to go through messages you either have to click on the message or hit enter to view it. It would be nice if they loaded as soon as you arrowed to it. Lastly, the interface seems a little less polished than Thunderbird but that may be just because I am more accustomed to TBird after years of use.

If you are interested in trying Claws out and use Ubuntu I recommend adding the Claws PPA to your apt sources to get the latest version and claws-mail-extra-plugins.