The _changes feed is a running list of all the documents that have changed in a database listed in order by sequence number. This is similar to update notifications but gives more information such as the document IDs and is HTTP based (with multiple feed styles) rather than stdout. Additionally you can create design document filters which can be specified as a query parameter to give you only the parts of the feed you want. All in all _changes is a pretty powerful feature.

Now for the fun stuff, the code. There are a few dependencies I used to do this, specifically focused on making it fast. As such I used EventMachine based libraries for AMQP and HTTP requests. The first bit of code takes the _changes feed for the “test” database, parses the feed, uses the document ID to request that document and publish it to the queue. One key item to note is that this code requires the latest yajl-ruby from github to run properly. Additionally, this works nicely with feed=continuous so it grabs the documents as they are changed without a need for polling.

Note that there is a variable for since, this allows you to start from a specific sequence number so you can skip over old changes.

The next bit of code works from the other side of the queue. It subscribes to the queue, parses the JSON, performs some operations on it and puts the results back into another CouchDB database called “results”.

What could it be used for? My first thought is some sort of parallel computation, boot up a few dozen EC2 nodes and start dumping data into CouchDB. Have all those nodes pop messages off the queue, process them and dump the results back into Couch. Legitimately one could chain these together to process the results again. The queue ends up being a simple job management system with the EC2 nodes popping new messages as they finish processing them. With a little bit of work, features and the right use case I think could be a pretty powerful system.

Check out the code, my other projects and follow me on twitter @williamsjoe.

[edit: made a slight improvement to changes_sub.rb on 20100107]

• Multiple posts at Fuad AlTabba, with an erlang implementation.
• Ruby rbtree library (uses C).
• An implementation in Ruby.
• Trees in Erlang.
• Red-black trees in two hours, with a link to Chris Okaski’s Red-Black Trees in a Functional Setting with implementation in Haskell.

socat is a relay for bidirectional data transfer between two independent data channels. Each of these data channels may be a file, pipe, device (serial line etc. or a pseudo terminal), a socket (UNIX, IP4, IP6 – raw, UDP, TCP), an SSL socket, proxy CONNECT connection, a file descriptor (stdin etc.), the GNU line editor (readline), a program, or a combination of two of these. These modes include generation of “listening” sockets, named pipes, and pseudo terminals.

Here are a few examples of how to use the stats socket. First, you need to add stats socket PATH to your configuration and restart haproxy. You should then find a socket located at the path specified, I used /tmp/haproxy. Now you can send it commands to get more information and stats from HAProxy.

echo "show stat" | socat unix-connect:/tmp/haproxy stdio


This will give you stats on all of your backends and frontends, some of the same stuff you see on the stats page enabled by the stats uri configuration. As an added bonus it’s all in CSV.

echo "show errors" | socat unix-connect:/tmp/haproxy stdio


show errors will give you a capture of last error on each backend/frontend.

echo "show info" | socat unix-connect:/tmp/haproxy stdio


This will give you information about the running HAProxy process such as pid, uptime and etc.

echo "show sess" | socat unix-connect:/tmp/haproxy stdio


This will dump (possibly huge) info about all know sessions.

For more details check out the docs section 9 and stats socket in section 3.1.

Bonus socat fun.

socat is a more full featured cousin of netcat. Both can be used in similar ways, one thing I use them for occasionally is debugging REST and etc. This was a real help when working with an API that didn’t have a library, I could test things out without needing to make erroneous calls to the API. In the simplest case you can have either of them listen on a port and output all the details of the request. To do this with socat run:

socat tcp-listen:8000 stdio

This will listen for connections on port 8000. Doing the same thing with netcat is easy as well:

netcat -l -p 8000

For instance you can see the output from creating a document in CouchDB.

In one terminal:

$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'rest_client'
=> true
irb(main):003:0> RestClient.put("http://localhost:8000/somedb/somedoc", "{\"somekey\": \"somevalue\"}", :content_type => "application/json")


In another run your mock server:

$ socat tcp-listen:8000 stdio
PUT /somedb/somedoc HTTP/1.1
Accept: application/xml
Content-Type: application/json
Accept-Encoding: gzip, deflate
Content-Length: 24
Host: localhost:8000

{"somekey": "somevalue"}


Oh! By the way, if you install netcat from source, don’t compile with -DGAPING_SECURITY_HOLE unless you know what you are doing.

couch_amqp.rb :

#!/usr/bin/ruby

require ‘rubygems’
require ‘carrot’

def main
queue = “couchdb”
run = true
couchq = Carrot.queue(:queue => queue)

while run do

notifications = gets

if notifications == nil
run = false
else
couchq.publish(notifications)
end

end
end

main

As you can tell we connect to a queue called “couchdb” on by default this is on localhost. Next we have a loop that continually runs and grabs updates from stdin. I then publish each notification to the queue and that’s that. To get the messages out of the queue I used irb and carrot.

[user@host ~]$ irb
irb(main):001:0> require ‘rubygems’
=> true
irb(main):002:0> require ‘carrot’
=> true
irb(main):003:0> couchq = Carrot.queue(:queue => “couchdb”)
=> #<Carrot::AMQP::Queue:0×7f8d2284b640 <snip>
irb(main):004:0> couchq.pop
=> “{\”type\”:\”updated\”,\”db\”:\”test1\”}\n”

So yeah, pretty simple stuff. Go ahead relax!

[EDIT 06/05/2009 2326 PST : Don't forget to add the entry to your local.ini]

[update_notification]

couch_amqp=/PATH/TO/couch_amqp.rb

stats() ->
gen_server2:call(?SERVER, {stats}).

versus

stats(Pid) ->
gen_server2:call(Pid, {stats}).

This allows more than one gen_server process to be started, the down side being you have to pass this Pid variable around. The other change is a new module called queue_merle, this is a sort of the process pool later that interfaces with merle. Obviously this is a very rough cut but seems to do the trick. The start function starts five merle processes and adds them to the queue, rotate rotates the queue taking all the head of the queue and inserting it into the bottom. I have impletemeted getkey and set as well. They accept a queue, key and/or value. The downside to this implementation is similar to that of using merle without defining ?SERVER, you have to know what, in this case, queue you are using and you need to make sure it is the most current otherwise you will end up getting more calls to one process than another. Here is an example of usage.

1> Queue = queue_merle:start().
{[<0.39.0>,<0.38.0>,<0.37.0>,<0.36.0>],[<0.33.0>]}
2> {Queue1, Result1} = queue_merle:set(Queue, a, “asdf”).
{{[<0.33.0>,<0.39.0>,<0.38.0>],[<0.36.0>,<0.37.0>]},ok}
3> {Queue2, Result2} = queue_merle:set(Queue1, b, “1234″).
{{[<0.36.0>,<0.33.0>,<0.39.0>,<0.38.0>],[<0.37.0>]},ok}
4> {Queue3, Result3} = queue_merle:getkey(Queue2, a).
{{[<0.37.0>,<0.36.0>,<0.33.0>],[<0.38.0>,<0.39.0>]},”asdf”}
5> {Queue4, Result4} = queue_merle:getkey(Queue3, b).
{{[<0.38.0>,<0.37.0>,<0.36.0>,<0.33.0>],[<0.39.0>]},”1234″}

As you can see the queue is rotating each time the functions are run but due to not allowing for multiple assignment one has to grab the new version of the queue each time and use it for the next operation. I imagine there is probably a cleaner way to do this, if I come up with one I like it will probably get added to mainline merle. Fun stuff.

From a comment in their source file:

More efficient handling of selective receives in callbacks gen_server2 processes drain their message queue into an internal buffer before invoking any callback module functions. Messages are dequeued from the buffer for processing. Thus the effective message queue of a gen_server2 process is the concatenation of the internal buffer and the real message queue. As a result of the draining, any selective receive invoked inside a callback is less likely to have to scan a large message queue.

This means if you send a ton of messages at once it can handle this more effectively. In the case of merle this means more gets/puts/deletes/etc in a shorter amount of time. Some of the downsides are stated on the mailing list. I believe for the workload that merle does (lots of small messages in short time spans) this is a great addition. For other use cases it may not be, you know when you should test.

I have run some tests using gen_server and gen_server2 doing a large number of ’set’ operations to memcached. The test consisted of running merle:set(a, “1″) a specific number of times (25k, 50k and 100k) with both gen_server and gen_server2. Since the mailbox gets backed up the Erlang processes are started before the operations complete on the memcached side. I didn’t have a good way to watch the memcached logs for when the operations completed and log timestamps so I used a simple stopwatch app to physically do the timing. Obviously this isn’t scientific but as you will see the differences are large enough its not a big deal.

(click here for a larger view)

As you can see gen_server2 performs much better (almost linearly?), shaving large amounts of time off. Also note that on the gen_server 100k tests I stopped the testing once it reached 5 minutes, so I am unsure how much longer those would have went on. Below is the raw data, I also preformed subsequent tests and found that my initial findings seemed to be accurate.

The latest source for merle using gen_server2 has been committed to github, give it shot and let me know if you find any bugs.

The first thing that needs to be done, if you do not already have a working chef/chef-server/ohai installation, is to get one up and running. If you already have a working install you should be able to skip down to the Erlang install portion and start from there.

Installing all this is pretty basic, check out the instructions. You should end up with a working installation with chef-server using stompserver (which we will be replacing) and couchdb. Here’s what I did:

sudo apt-get install ruby ruby1.8-dev rubygems libopenssl-ruby1.8 build-essential wget
wget http://rubyforge.org/frs/download.php/45904/rubygems-update-1.3.1.gem
sudo gem install rubygems-update-1.3.1.gem
sudo /var/lib/gems/1.8/bin/update_rubygems
sudo rm /usr/bin/gem
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem
sudo gem sources -a http://gems.opscode.com
sudo gem install chef ohai chef-server
apt-get install couchdb

I then added in my configuration for chef-server:

vi /etc/chef/server.rb

#
## Chef Server Config File
##
#
log_level :info
log_location STDOUT
ssl_verify_mode :verify_none
registration_url “http://localhost:4000″
openid_url “http://localhost:4000″
template_url “http://localhost:4000″
remotefile_url “http://localhost:4000″
search_url “http://localhost:4000″
cookbook_path [ "/var/localhost/site-cookbooks", "/var/localhost/cookbooks" ]
merb_log_path “/var/log/localhost-server-merb.log”
queue_user “guest”
queue_password “guest”

Chef::Log::Formatter.show_time = false

Since I am using a single machine for my chef server I started two chef-server instances, one with the defaults chef-server and another chef-server -p 4001. Two instances are needed for OpenID to work properly.

The next objective is to install Erlang.

sudo apt-get install erlang

Now for the RabbitMQ and STOMP plugin install. Per Update 2 (below) using the latest plugin and version 1.5.1 of RabbitMQ doesn’t work. The latest revisions from the repository for both is needed.

apt-get install mercurial
cd /opt/
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-stomp
apt-get install python-simplejson
make -C rabbitmq-server
make -C rabbitmq-stomp

I used the init script for rabbitmq-server found in debian package for 1.5.1 with a slight change, pointing to my installation in /opt. I put a copy of it here.

DAEMON=/opt/rabbitmq-server/scripts/rabbitmq-multi

You will want to edit the config for rabbitmq to start up with STOMP activated and then restart the service.

vi /etc/default/rabbitmq

SERVER_START_ARGS=’
-pa /opt/rabbitmq-stomp/ebin
-rabbit
stomp_listeners [{"0.0.0.0",61613}]
extra_startup_steps [{"STOMP-listeners",rabbit_stomp,kickstart,[]}]’

Finally, start everything up, minus stompserver.

/etc/init.d/couchdb start
/etc/init.d/rabbitmq-server restart
chef-server
chef-server -p 4001
chef-indexer

After all that you should have a working chef-server using RabbitMQ/STOMP rather than stompserver. The rest is up to you.

Update 2:
It looks like the issue (below) was caused by using incompatible versions of STOMP plugin and RabbitMQ. Using the latest server and plugin from their repository fixes the issue. I have adjusted the above instructions to state this change.

Update:
It seems that I am having issues with the chef-indexer connecting properly to rabbitmq-stomp. From what I can tell this seems to be an issue with the RabbitMQ STOMP plugin since the stomp ruby client seems to work fine against stompserver. I suppose the RabbitMQ folks do label it “experimental”. Perhaps in a future revision it will be corrected. Anywho, here are some examples of what I am seeing.

root@ubuntu-810:~# chef-indexer
/usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:233:in `write’: Broken pipe (Errno::EPIPE)
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:233:in `puts’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:233:in `_transmit’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:232:in `synchronize’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:232:in `_transmit’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:220:in `transmit’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:117:in `subscribe’
from /usr/lib/ruby/gems/1.8/gems/chef-0.5.2/lib/chef/queue.rb:79:in `subscribe’
from /usr/lib/ruby/gems/1.8/gems/chef-server-0.5.2/bin/chef-indexer:80
from /usr/bin/chef-indexer:19:in `load’
from /usr/bin/chef-indexer:19

root@ubuntu-810:~# irb
irb(main):001:0> require ‘rubygems’
=> true
irb(main):002:0> require ’stomp’
=> true
irb(main):003:0> conn = Stomp::Connection.open(‘guest’, ‘guest’, ‘localhost’)
=> #, @login=”guest”, @reconnectDelay=5, @transmit_semaphore=#, @failure=nil, @port=61613, @connect=#>,n <<"/">>]},n {rabbit_stomp,do_login,5},n {rabbit_stomp,process_received_bytes,2},n {rabbit_stomp,init,1},n {proc_lib,init_p,5}]}n”, @headers={“content-type”=>”text/plain”, “content-length”=>”290″, “message”=>”Processing error”}>, @reliable=false, @clientId=nil, @subscriptions={}, @host=”localhost”, @socket_semaphore=#, @socket=#, @passcode=”guest”, @closed=false>

root@ubuntu-810:~# tail /var/log/rabbitmq/rabbit.log
=INFO REPORT==== 3-Feb-2009::02:11:42 ===
starting STOMP connection <0.3746.0> from 127.0.0.1:59035

=ERROR REPORT==== 3-Feb-2009::02:11:42 ===
STOMP error frame sent:
Message: “Processing error”
Detail: “{undef,[{rabbit_channel,start_link,n [rabbit_stomp,<0.3746.0>,<0.3746.0>,<<"guest">>,n <<"/">>]},n {rabbit_stomp,do_login,5},n {rabbit_stomp,process_received_bytes,2},n {rabbit_stomp,init,1},n {proc_lib,init_p,5}]}n”

=INFO REPORT==== 3-Feb-2009::02:11:42 ===
ending STOMP connection <0.3746.0> from 127.0.0.1:59035

3> fermal:album_info(“animal%20collective”, “feels”).
[[album,
{name,"Feels"},
{artist,"Animal Collective"},
{id,"2050717"},
{mbid,"8a9ee208-3c9e-4fec-8f60-9a8b1db39960"},
{url,"http://www.last.fm/music/Animal+Collective/Feels"},
{releasedate," 18 Oct 2005, 00:00"},
{listeners,"184794"},
{playcount,"3539880"},
{toptags,"\n "}]]