We finished deploying the monitor in a RackspaceCloud instance running Ubuntu 9.

I cannot call the deployment smooth – mostly because BackgroundDB simply wouldn’t start. Saying this:


/var/lib/gems/1.8/gems/packet-0.1.15/lib/packet/packet_master.rb:117:in `exec': No such file or directory - packet_worker_runner 10:9:monitor_runner_worker:4:/opt/rails/rras-monitor/lib/workers:/opt/rails/rras-monitor/script/load_worker_env (Errno::ENOENT)
from /var/lib/gems/1.8/gems/packet-0.1.15/lib/packet/packet_master.rb:117:in `fork_and_load'
from /var/lib/gems/1.8/gems/packet-0.1.15/lib/packet/packet_master.rb:68:in `load_workers'
from /var/lib/gems/1.8/gems/packet-0.1.15/lib/packet/packet_master.rb:63:in `each'
from /var/lib/gems/1.8/gems/packet-0.1.15/lib/packet/packet_master.rb:63:in `load_workers'
from /var/lib/gems/1.8/gems/packet-0.1.15/lib/packet/packet_master.rb:20:in `run'
from /opt/rails/rras-monitor/vendor/plugins/backgroundrb/server/lib/master_proxy.rb:14:in `initialize'
from /opt/rails/rras-monitor/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_start_stop.rb:109:in `new'
from /opt/rails/rras-monitor/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_start_stop.rb:109:in `start_bdrb'
from /opt/rails/rras-monitor/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_start_stop.rb:16:in `start'
from ./script/backgroundrb:24

Nasty error took a lot of time to figure out. For some reason gem tool works slightly different on Ubuntu and fails to install scripts in /usr/bin or any accessible locations in PATH. So setting up mongrel_rails and backgroundb became a nightmare.

Finally comparing “packet” gem files to the working version of RRAS Monitor on another server I figured out that “/usr/bin/packet_worker_runner” in our case is not a symlink but in fact  a generated script. After that we simply copied that script from another box and corrected the paths.

Attaching the final version of the script for anybody how comes across the problem:

#!/usr/bin/ruby
#
# This file was generated by RubyGems.
#
# The application 'packet' is installed as part of a gem, and
# this file is here to facilitate running it.
#

require ‘rubygems’

version = “>= 0″

if ARGV.first =~ /^_(.*)_$/ and Gem::Version.correct? $1 then
version = $1
ARGV.shift
end

gem ‘packet’, version
load ‘packet_worker_runner’
Now that we have cloud deployment in place I can work to improve connectivity between Monitor instances.

Stay in touch!

• fixes to the charts – now they look better and aligned correctly
• created email notification system that sends out emails to one or many people if a service breaks or is fixed
• few CSS corrections for the layout

Please don’t forget to configure your email settings in config/initializers/mail.rb before using e-mail feature.

Enjoy!

Screenshot

Working with one of the projects I’ve had very unpleasant experience trying to figure out which one of a little over 10 services is not working.

I didn’t want to go out an look for solution since recently I’ve done similar research and wasn’t amazed by any solutions out there. Some of them cost money too. So I went ahead and wrote my own monitor system in Rails. 1 day of work I got it working – very extensible and also fits my choice of technologies for best fit with RRAS.

515 lines of code and it went in “production” (I deployed and set it up on the server).

I definitely see some room for improvement there, but it is up in running in the way it is right now with 2 working monitors (URLMonitor and ProcessMonitor). Few words about them:

• URL Monitor allow to do simple URL check and match results with a regexp

• Process Monitor take a PID file path a input and perform some system checks to records CPU and RAM used as well as it reports errors if the PID file is missing or the process isn’t running. I can use it to monitor literraly any Linux service including mysql and mongrel.

So at this point it is pretty basic but we also have some JMX enabled Java servers on our service list , so I am going to keep working to get some stats reported  from those . SQL monitor and script monitors are in the way to.

Hey one thing I forgot to mention I do have graphs implemented so it displays metrics visually. Unfortunately I can’t show it here because at this point I don’t have that much data for display.

That’s all I had to say. Welcome to use and contribute to the project. It is hosted at github http://github.com/rras/rras-monitor/tree/master .  You can follow the updates to the server on our company twitter @prowss .

Bye

Few major goals I am going to be aiming for.

1) Easy mesh-ups

Building Ruby on Rails web sites I noticed that something it is difficult to mesh projects up – say take a forum (Beast or RForum) , glue it to your own application and then for example add Radiant or another CMS.  It is very frustrating since the software is there  for you but you can’t just say “I want Beast and Radiant to work together with my application”. So one of the main goals is going to provide that easiness.

2) Deployment

I’d love to streamline the deployments. Upload zipped application, run tests, deploy – just clicking mouse. Additionally sometimes you hit a bug after deployment and you want to rollback – that makes your fingers hurt typing – database, backup files. I am going to simplify rollback routine too.

3) Performance testing / management

Rails is not so much fun to watch in production. Logs are pretty boring so what I’d like to do is have few Mongrels running in more verbose mode so we can see some stats and monitor it.

4) Clustering

Want to do some cool things with clustering – some smart load balancing – rules to be able to use your models and determine what cluster segment it is going to go . Many people disagree but I would love to have 10% of my users (I can decide who) to use new user interface and report issues before complete rollout. It would be fun to implement.

Implementation

Ultimately I’d like to implement nifty and yet robust application server for Ruby on Rails applications. The server is going to  be built on the top of Mongrel. I am not planning to integrate Mongrel very deep just much as I need to provide some cool extras on the top of standard Mongrel distribution.  There will be very tight OS integration to streamline security and integration with other products. Extensibility … the server will be extensible by plugins so you can add additional monitoring and deployment tools. Each of a few plugins will be running in a separate Mongrel instance to avoid interruption for the main server in case of a crash or a bug.

Lastly the project location and licensing. Right now the project is hosted at GitHub http://github.com/rras/rras/tree/master . It is open source and licenses under AGPL (Affero GPL). The most evil open source license I could imagine. Right now I can’t make a call how big the project is going to be, how much interest will it get neither what I want to get from it . So I am just covering my a*s before I can decide. One thing I will definitely try to avoid is reselling the code as an commercial application server not paying me back for my work – so MIT or LGPL is not very likely to happen unless somebody makes big enough donation.