The _changes feed is a running list of all the documents that have changed in a database listed in order by sequence number. This is similar to update notifications but gives more information such as the document IDs and is HTTP based (with multiple feed styles) rather than stdout. Additionally you can create design document filters which can be specified as a query parameter to give you only the parts of the feed you want. All in all _changes is a pretty powerful feature.

Now for the fun stuff, the code. There are a few dependencies I used to do this, specifically focused on making it fast. As such I used EventMachine based libraries for AMQP and HTTP requests. The first bit of code takes the _changes feed for the “test” database, parses the feed, uses the document ID to request that document and publish it to the queue. One key item to note is that this code requires the latest yajl-ruby from github to run properly. Additionally, this works nicely with feed=continuous so it grabs the documents as they are changed without a need for polling.

Note that there is a variable for since, this allows you to start from a specific sequence number so you can skip over old changes.

The next bit of code works from the other side of the queue. It subscribes to the queue, parses the JSON, performs some operations on it and puts the results back into another CouchDB database called “results”.

What could it be used for? My first thought is some sort of parallel computation, boot up a few dozen EC2 nodes and start dumping data into CouchDB. Have all those nodes pop messages off the queue, process them and dump the results back into Couch. Legitimately one could chain these together to process the results again. The queue ends up being a simple job management system with the EC2 nodes popping new messages as they finish processing them. With a little bit of work, features and the right use case I think could be a pretty powerful system.

Check out the code, my other projects and follow me on twitter @williamsjoe.

[edit: made a slight improvement to changes_sub.rb on 20100107]

Best albums:

Wavves – Wavves
Telefon Tel Aviv – Immolate Yourself
Riceboy Sleeps – Riceboy Sleeps
Dirty Projectors – Bitte Orca
Black Moth Super Rainbow – Eating Us
Animal Collective – Merriweather Post Pavilion
Japandriods – Post-Nothing
People Under the Stairs – Carried Away
Fuck Buttons – Tarot Sport
Russian Circles – Geneva

Best EPs:

Extra Life/Nat Baldwin – A Split
LITE – Turns Red
Abe Vigoda – Reviver
STATS – Marooned

From wikipedia:

Biodynamic agriculture is a method of organic farming with homeopathic composts that treats farms as unified and individual organisms, emphasizing balancing the holistic development and interrelationship of the soil, plants, animals as a self-nourishing system without external inputs insofar as this is possible given the loss of nutrients due to the export of food.

To me this totally has an analog in datacenters, server farms (pun intended) and machine rooms. To paraphrase the above wikipedia quote:

An electrodynamic datacenter is one that is treated as a unified and individual organism. That is each datacenter is an autonomous entity and needs to be thought about as an organism where all the components (CRACs, servers, network, power, etc) are balanced and interrelated without external inputs insofar as this is possible given the loss of capacity (bandwidth, compute, storage, etc) due to export of data, compute or another resource.

Putting it like that seems pretty reasonable and would seem to lean toward making datacenters as efficient as possible. The goal being reducing external inputs (power, bandwidth and etc) while still getting the desired amount of output. Practices such as running datacenters hot, data locality optimization or shutting down part (or all) of a datacenter while not needed would be common place. This would require tight monitoring, analysis, controls and automation on inputs and outputs. This also means developing a quantitative relationship between consumption/utilization and production, ie how much input is required for X amount of output. Certainly an interesting problem to solve and system to build although I imagine some level of this has been implemented by the Googles of the world. While datacenters will likely never be self-sustaining in the end this may be a reasonable way to think about datacenter controls and management especially as we all try to go green for monetary and environmental reasons.

Yeah … it’s that good.

• Multiple posts at Fuad AlTabba, with an erlang implementation.
• Ruby rbtree library (uses C).
• An implementation in Ruby.
• Trees in Erlang.
• Red-black trees in two hours, with a link to Chris Okaski’s Red-Black Trees in a Functional Setting with implementation in Haskell.

Now for your rsyslog setup you may want to have nginx log to its own files rather than the default syslog log file. To do that just add the following to your configuration:

if $programname == 'nginx' and $syslogseverity <= '4' then /var/log/nginx_error.log
if $programname == 'nginx' and $syslogseverity == '5' then /var/log/nginx_access.log
if $programname == 'nginx' and $syslogseverity >= '6' then /var/log/nginx_debug_info.log


This will filter out all the logs for nginx by the severity of the log message. I have all errors go to one (<= 4), access logs (== 5) go to another and debug and info messages (>= 6) go to the last.

For more details on configuring rsyslog check out the wiki.

socat is a relay for bidirectional data transfer between two independent data channels. Each of these data channels may be a file, pipe, device (serial line etc. or a pseudo terminal), a socket (UNIX, IP4, IP6 – raw, UDP, TCP), an SSL socket, proxy CONNECT connection, a file descriptor (stdin etc.), the GNU line editor (readline), a program, or a combination of two of these. These modes include generation of “listening” sockets, named pipes, and pseudo terminals.

Here are a few examples of how to use the stats socket. First, you need to add stats socket PATH to your configuration and restart haproxy. You should then find a socket located at the path specified, I used /tmp/haproxy. Now you can send it commands to get more information and stats from HAProxy.

echo "show stat" | socat unix-connect:/tmp/haproxy stdio


This will give you stats on all of your backends and frontends, some of the same stuff you see on the stats page enabled by the stats uri configuration. As an added bonus it’s all in CSV.

echo "show errors" | socat unix-connect:/tmp/haproxy stdio


show errors will give you a capture of last error on each backend/frontend.

echo "show info" | socat unix-connect:/tmp/haproxy stdio


This will give you information about the running HAProxy process such as pid, uptime and etc.

echo "show sess" | socat unix-connect:/tmp/haproxy stdio


This will dump (possibly huge) info about all know sessions.

For more details check out the docs section 9 and stats socket in section 3.1.

Bonus socat fun.

socat is a more full featured cousin of netcat. Both can be used in similar ways, one thing I use them for occasionally is debugging REST and etc. This was a real help when working with an API that didn’t have a library, I could test things out without needing to make erroneous calls to the API. In the simplest case you can have either of them listen on a port and output all the details of the request. To do this with socat run:

socat tcp-listen:8000 stdio

This will listen for connections on port 8000. Doing the same thing with netcat is easy as well:

netcat -l -p 8000

For instance you can see the output from creating a document in CouchDB.

In one terminal:

$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'rest_client'
=> true
irb(main):003:0> RestClient.put("http://localhost:8000/somedb/somedoc", "{\"somekey\": \"somevalue\"}", :content_type => "application/json")


In another run your mock server:

$ socat tcp-listen:8000 stdio
PUT /somedb/somedoc HTTP/1.1
Accept: application/xml
Content-Type: application/json
Accept-Encoding: gzip, deflate
Content-Length: 24
Host: localhost:8000

{"somekey": "somevalue"}


Oh! By the way, if you install netcat from source, don’t compile with -DGAPING_SECURITY_HOLE unless you know what you are doing.

• uses s3 to backup a directory of files (no subdirectories)
• uses fadvise to be easy on filesystem caches and disks
• purges files after X days
• streams files rather than loading them entirely into memory

They are very simple to use, just create a configuration file together:

amazon_access_key_id: "someid"
amazon_secret_access_key: "somekey"
backup_dir: "/some/path/"
purge_threshold: 3
bucket_name: "somebucket"


Backup a directory of files:

$ ./tens3_put tens3.conf

Restore a file from a backup:

$ ./tens3_get tens3.conf date somefile ./somefile

The date is the date that the file was backed up in a YYYYMMDD format.

Enjoy and let me know if you find any bugs or want new features.