Joe’s Blog!

I’ve been debugging issues with HTTP, my backend servers and HAProxy. After a quick email to the HAProxy mailing list I found out about a configuration option stats socket PATH. This will create a socket you can send commands to and get more information out of HAProxy. To do this I just used some simle unix tools, the key is socat. From the man:

socat is a relay for bidirectional data transfer between two independent data channels. Each of these data channels may be a file, pipe, device (serial line etc. or a pseudo terminal), a socket (UNIX, IP4, IP6 – raw, UDP, TCP), an SSL socket, proxy CONNECT connection, a file descriptor (stdin etc.), the GNU line editor (readline), a program, or a combination of two of these. These modes include generation of “listening” sockets, named pipes, and pseudo terminals.

Here are a few examples of how to use the stats socket. First, you need to add stats socket PATH to your configuration and restart haproxy. You should then find a socket located at the path specified, I used /tmp/haproxy. Now you can send it commands to get more information and stats from HAProxy.

echo "show stat" | socat unix-connect:/tmp/haproxy stdio


This will give you stats on all of your backends and frontends, some of the same stuff you see on the stats page enabled by the stats uri configuration. As an added bonus it’s all in CSV.

echo "show errors" | socat unix-connect:/tmp/haproxy stdio


show errors will give you a capture of last error on each backend/frontend.

echo "show info" | socat unix-connect:/tmp/haproxy stdio


This will give you information about the running HAProxy process such as pid, uptime and etc.

echo "show sess" | socat unix-connect:/tmp/haproxy stdio


This will dump (possibly huge) info about all know sessions.

For more details check out the docs section 9 and stats socket in section 3.1.

Bonus socat fun.

socat is a more full featured cousin of netcat. Both can be used in similar ways, one thing I use them for occasionally is debugging REST and etc. This was a real help when working with an API that didn’t have a library, I could test things out without needing to make erroneous calls to the API. In the simplest case you can have either of them listen on a port and output all the details of the request. To do this with socat run:

socat tcp-listen:8000 stdio

This will listen for connections on port 8000. Doing the same thing with netcat is easy as well:

netcat -l -p 8000

For instance you can see the output from creating a document in CouchDB.

In one terminal:

$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'rest_client'
=> true
irb(main):003:0> RestClient.put("http://localhost:8000/somedb/somedoc", "{\"somekey\": \"somevalue\"}", :content_type => "application/json")


In another run your mock server:

$ socat tcp-listen:8000 stdio
PUT /somedb/somedoc HTTP/1.1
Accept: application/xml
Content-Type: application/json
Accept-Encoding: gzip, deflate
Content-Length: 24
Host: localhost:8000

{"somekey": "somevalue"}


Oh! By the way, if you install netcat from source, don’t compile with -DGAPING_SECURITY_HOLE unless you know what you are doing.

I recently needed some simple scripts to backup files on various machines, stuff like configs and even some small CouchDB files. Not finding something already out there I put together tens3, two simple scripts to get and put files to Amazon S3. They provide the following:

• uses s3 to backup a directory of files (no subdirectories)
• uses fadvise to be easy on filesystem caches and disks
• purges files after X days
• streams files rather than loading them entirely into memory

They are very simple to use, just create a configuration file together:

amazon_access_key_id: "someid"
amazon_secret_access_key: "somekey"
backup_dir: "/some/path/"
purge_threshold: 3
bucket_name: "somebucket"


Backup a directory of files:

$ ./tens3_put tens3.conf

Restore a file from a backup:

$ ./tens3_get tens3.conf date somefile ./somefile

The date is the date that the file was backed up in a YYYYMMDD format.

Enjoy and let me know if you find any bugs or want new features.

Headed to Boston next week, planning to meet-up next Tuesday (7/28) 7pm at Cambridge Brewing Co. Drop by for a beer, food and maybe a little Erlang.

Working at Cloudant I use CouchDB on a daily basis. This evening for fun I decided to write some Ruby to take update notifications and push them into RabbitMQ. There are other examples of using the update notifications and Ruby in Couch such as the view updater out on the Couch wiki. It turned out super simple. There are a few AMQP libraries for Ruby, in this example I am going to use carrot.  It’s based on the  amqp library without all the eventmachine stuff. So here it goes:

couch_amqp.rb :

#!/usr/bin/ruby

require ‘rubygems’
require ‘carrot’

def main
queue = “couchdb”
run = true
couchq = Carrot.queue(:queue => queue)

while run do

notifications = gets

if notifications == nil
run = false
else
couchq.publish(notifications)
end

end
end

main

As you can tell we connect to a queue called “couchdb” on by default this is on localhost. Next we have a loop that continually runs and grabs updates from stdin. I then publish each notification to the queue and that’s that. To get the messages out of the queue I used irb and carrot.

[user@host ~]$ irb
irb(main):001:0> require ‘rubygems’
=> true
irb(main):002:0> require ‘carrot’
=> true
irb(main):003:0> couchq = Carrot.queue(:queue => “couchdb”)
=> #<Carrot::AMQP::Queue:0x7f8d2284b640 <snip>
irb(main):004:0> couchq.pop
=> “{\”type\”:\”updated\”,\”db\”:\”test1\”}\n”

So yeah, pretty simple stuff. Go ahead relax!

[EDIT 06/05/2009 2326 PST : Don't forget to add the entry to your local.ini]

[update_notification]

couch_amqp=/PATH/TO/couch_amqp.rb

I recently switched an application I am working on from nginx and mongrel to nginx and passenger. The setup is easy as can be but I noticed an issue on one of my long running operations. I have a controller that spawns some threads, performs operations and returns back a result to the page. Sometimes this takes a minute and using mongrel you would just increase the proxy_read_timeout in nginx. With passenger it times out after 60 seconds. There is not a configuration parameter for adjusting this. I found that adjusting upstream.read_timeout and upstream.send_timeout in /usr/lib/ruby/gems/1.8/gems/passenger-2.2.1/ext/nginx/Configuration.c (below) to a higher value and reinstalling the passenger module solves the problem.

/usr/lib/ruby/gems/1.8/gems/passenger-2.2.1/ext/nginx/Configuration.c

::snip::
ngx_conf_merge_msec_value(conf->upstream.send_timeout,
prev->upstream.send_timeout, 60000);

ngx_conf_merge_msec_value(conf->upstream.read_timeout,
prev->upstream.read_timeout, 60000);
::snip::

Thanks to the passenger team and the quick response on the mailing list.