Joe’s Blog!

I have been playing around with merle and have switched it from using the normal gen_server to using LShift’s modified gen_server2. It has a couple of changes to make things faster, the key is:

From a comment in their source file:

More efficient handling of selective receives in callbacks gen_server2 processes drain their message queue into an internal buffer before invoking any callback module functions. Messages are dequeued from the buffer for processing. Thus the effective message queue of a gen_server2 process is the concatenation of the internal buffer and the real message queue. As a result of the draining, any selective receive invoked inside a callback is less likely to have to scan a large message queue.

This means if you send a ton of messages at once it can handle this more effectively. In the case of merle this means more gets/puts/deletes/etc in a shorter amount of time. Some of the downsides are stated on the mailing list. I believe for the workload that merle does (lots of small messages in short time spans) this is a great addition. For other use cases it may not be, you know when you should test.

I have run some tests using gen_server and gen_server2 doing a large number of ’set’ operations to memcached. The test consisted of running merle:set(a, “1″) a specific number of times (25k, 50k and 100k) with both gen_server and gen_server2. Since the mailbox gets backed up the Erlang processes are started before the operations complete on the memcached side. I didn’t have a good way to watch the memcached logs for when the operations completed and log timestamps so I used a simple stopwatch app to physically do the timing. Obviously this isn’t scientific but as you will see the differences are large enough its not a big deal.

(click here for a larger view)

As you can see gen_server2 performs much better (almost linearly?), shaving large amounts of time off. Also note that on the gen_server 100k tests I stopped the testing once it reached 5 minutes, so I am unsure how much longer those would have went on. Below is the raw data, I also preformed subsequent tests and found that my initial findings seemed to be accurate.

The latest source for merle using gen_server2 has been committed to github, give it shot and let me know if you find any bugs.

Decided to freshen things up here on the blog. Hope the new theme does the trick.

I decided to play around with RabbitMQ and chef (Opscode recently released new version) this afternoon. My main objective was to see if I could replace stompserver with something a bit more robust. In this case RabbitMQ and it’s STOMP plugin, the following is a basic outline of what I did. Please note that this was done on a Ubuntu 8.10 machine, if you are using another non-apt distro you will likely need to adjust the package installation commands and etc.

The first thing that needs to be done, if you do not already have a working chef/chef-server/ohai installation, is to get one up and running. If you already have a working install you should be able to skip down to the Erlang install portion and start from there.

Installing all this is pretty basic, check out the instructions. You should end up with a working installation with chef-server using stompserver (which we will be replacing) and couchdb. Here’s what I did:

sudo apt-get install ruby ruby1.8-dev rubygems libopenssl-ruby1.8 build-essential wget
wget http://rubyforge.org/frs/download.php/45904/rubygems-update-1.3.1.gem
sudo gem install rubygems-update-1.3.1.gem
sudo /var/lib/gems/1.8/bin/update_rubygems
sudo rm /usr/bin/gem
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem
sudo gem sources -a http://gems.opscode.com
sudo gem install chef ohai chef-server
apt-get install couchdb

I then added in my configuration for chef-server:

vi /etc/chef/server.rb

#
## Chef Server Config File
##
#
log_level :info
log_location STDOUT
ssl_verify_mode :verify_none
registration_url “http://localhost:4000″
openid_url “http://localhost:4000″
template_url “http://localhost:4000″
remotefile_url “http://localhost:4000″
search_url “http://localhost:4000″
cookbook_path [ "/var/localhost/site-cookbooks", "/var/localhost/cookbooks" ]
merb_log_path “/var/log/localhost-server-merb.log”
queue_user “guest”
queue_password “guest”

Chef::Log::Formatter.show_time = false

Since I am using a single machine for my chef server I started two chef-server instances, one with the defaults chef-server and another chef-server -p 4001. Two instances are needed for OpenID to work properly.

The next objective is to install Erlang.

sudo apt-get install erlang

Now for the RabbitMQ and STOMP plugin install. Per Update 2 (below) using the latest plugin and version 1.5.1 of RabbitMQ doesn’t work. The latest revisions from the repository for both is needed.

apt-get install mercurial
cd /opt/
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-stomp
apt-get install python-simplejson
make -C rabbitmq-server
make -C rabbitmq-stomp

I used the init script for rabbitmq-server found in debian package for 1.5.1 with a slight change, pointing to my installation in /opt. I put a copy of it here.

DAEMON=/opt/rabbitmq-server/scripts/rabbitmq-multi

You will want to edit the config for rabbitmq to start up with STOMP activated and then restart the service.

vi /etc/default/rabbitmq

SERVER_START_ARGS=’
-pa /opt/rabbitmq-stomp/ebin
-rabbit
stomp_listeners [{"0.0.0.0",61613}]
extra_startup_steps [{"STOMP-listeners",rabbit_stomp,kickstart,[]}]’

Finally, start everything up, minus stompserver.

/etc/init.d/couchdb start
/etc/init.d/rabbitmq-server restart
chef-server
chef-server -p 4001
chef-indexer

After all that you should have a working chef-server using RabbitMQ/STOMP rather than stompserver. The rest is up to you.

Update 2:
It looks like the issue (below) was caused by using incompatible versions of STOMP plugin and RabbitMQ. Using the latest server and plugin from their repository fixes the issue. I have adjusted the above instructions to state this change.

Update:
It seems that I am having issues with the chef-indexer connecting properly to rabbitmq-stomp. From what I can tell this seems to be an issue with the RabbitMQ STOMP plugin since the stomp ruby client seems to work fine against stompserver. I suppose the RabbitMQ folks do label it “experimental”. Perhaps in a future revision it will be corrected. Anywho, here are some examples of what I am seeing.

root@ubuntu-810:~# chef-indexer
/usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:233:in `write’: Broken pipe (Errno::EPIPE)
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:233:in `puts’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:233:in `_transmit’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:232:in `synchronize’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:232:in `_transmit’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:220:in `transmit’
from /usr/lib/ruby/gems/1.8/gems/stomp-1.0.6/lib/stomp.rb:117:in `subscribe’
from /usr/lib/ruby/gems/1.8/gems/chef-0.5.2/lib/chef/queue.rb:79:in `subscribe’
from /usr/lib/ruby/gems/1.8/gems/chef-server-0.5.2/bin/chef-indexer:80
from /usr/bin/chef-indexer:19:in `load’
from /usr/bin/chef-indexer:19

root@ubuntu-810:~# irb
irb(main):001:0> require ‘rubygems’
=> true
irb(main):002:0> require ’stomp’
=> true
irb(main):003:0> conn = Stomp::Connection.open(‘guest’, ‘guest’, ‘localhost’)
=> #, @login=”guest”, @reconnectDelay=5, @transmit_semaphore=#, @failure=nil, @port=61613, @connect=#>,n <<"/">>]},n {rabbit_stomp,do_login,5},n {rabbit_stomp,process_received_bytes,2},n {rabbit_stomp,init,1},n {proc_lib,init_p,5}]}n”, @headers={“content-type”=>”text/plain”, “content-length”=>”290″, “message”=>”Processing error”}>, @reliable=false, @clientId=nil, @subscriptions={}, @host=”localhost”, @socket_semaphore=#, @socket=#, @passcode=”guest”, @closed=false>

root@ubuntu-810:~# tail /var/log/rabbitmq/rabbit.log
=INFO REPORT==== 3-Feb-2009::02:11:42 ===
starting STOMP connection <0.3746.0> from 127.0.0.1:59035

=ERROR REPORT==== 3-Feb-2009::02:11:42 ===
STOMP error frame sent:
Message: “Processing error”
Detail: “{undef,[{rabbit_channel,start_link,n [rabbit_stomp,<0.3746.0>,<0.3746.0>,<<"guest">>,n <<"/">>]},n {rabbit_stomp,do_login,5},n {rabbit_stomp,process_received_bytes,2},n {rabbit_stomp,init,1},n {proc_lib,init_p,5}]}n”

=INFO REPORT==== 3-Feb-2009::02:11:42 ===
ending STOMP connection <0.3746.0> from 127.0.0.1:59035