FrJo.info

Here I am again with a problem to solve. I’ve become problem solver to my self-created problems. Duh. Now what I need to solve is regarding a new monitoring software recently ported to Mac os X. It’s been one month now that BixData ported their suite, BixAgent; BixServer; BixDesktop, to my platform of predilection.

BixData’s suite let you monitor servers, services, processes, S.M.A.R.T. status, etc. as easy as if there was only one machine to look for. BixServer, the engine behind BixData product, is powered by a Mysql server and let you able to manage everything within a nice interface. BixDesktop is that graphical interface that you might use to configure machines, watched services, servers, processes, notification setup or what else I might forget. At last but not least, BixAgent may be installed on remote machine you want to monitor processes, ram usage, disk status etc.

The product is very young so it turns out unstable and you quickly realize that maybe this is a good product but, you don’t have that much time to wait for it to get aged. Being young doesn’t mean that it’s not good. Myself for example, I’m 22 years old and I don’t like to think I’m worthless. So I gave few shots at BixData and worked with those guys to make it a better product. I guest I was looking to over book my already full schedule with it. keep busy.

Something that is missing and that I was very unsure about is the way to start those modules. Basically, at that point in the software development, both, agent and server, need to be started by hand. So far, that doesn’t make it a problem, you just add it to a simple line to your startup script and you’re good to go. Not quite. There is no daemon switch to set so it runs in the background. So far not so good.

On the BixData forum, I found a guy who wrote a startup script for his FreeBSD machine. He basically use nohup command to start the software, and nohup keep it running with “no” “h”ang”up”. Sadly, nohup doesn’t exist in Mac os X. Since I’m using BixData software mainly on Mac os X machines, I needed to find a better way to handle those programs. I pushed myself to solve this issue because at that point in time, I was starting the agents in a “screen” and detaching myself from it. By doing so, I keep the process running even after I closed the SSH connections. One of the problem that I faced at that point was that once the agent loose connection with the server, the process exit with a “Broken pipe” error. Therefore, the entire purpose of that monitoring system is gone. My “Broken Pipe” situation is due to dynamic IPs from my ISP. The DNS domain-name follows but it takes few minutes and then the link is broken. Knowing some stuff on how launchd works on Mac os X, I’m working on that to solve my start/re-spawn problem. If you don’t know about launchd, you must google it.

A word about Launchd
Launchd is an Open Source project mainly maintained and funded by Apple Computer inc. They registered that Open Source project under the Apache 2 Licensing. Launchd is replacing in all Tigerized flavor of Mac os X processes such as xinit, init.d, StartupItems and Crons. Launchd is a powerful tool for you to have in your toolbox when you want to manage processes. Launchd and launchctl are tools that let you do stuff you are use to do with those in it, cron or startup items stuff through a simple xml text file.

Let’s built that xml file
I need for bixAgent to execute “./bixagent” from within the bixagent folder within bixdata folder that you extract after downloading it. BixAgent and BixServer are both very picky from where you start it’s respective daemon. If you’re out of it’s bixagent/bixserver folder, you will have errors and it will never start. Look at the troubleshooting section for more info on that behavior. Also, I need the process to be started at boot, never stop, be executed by user bixdata and write to log files and error log files. That’s about it.

Get to it
I read a lot about “under the hood” stuff on AFP548 web site and from a link to an other, I ended up with an SW Lingon that let me create an xml.plist file with a GUI interface. It’s always a good way to start when you’re in complete darkness regarding a technology. I did a basic config and readed couple of howtos and after hours of trial errors, I went back to THE old friend, vim, and created a new xml.plist file from scratch. I figured out that Lingon write a syntax error when selecting one of the options that I needed. Never again will I waste time by using a GUI tool. You can download my plist here. In addition, I created a plist file for bixserver. You can download the bixserver plist here.

Carefull when editing
Xml plist files are very strict syntax regarding. Be careful when editing the plist file with a GUI editor, they usually add up ghosts in your file. Use either TextEdit or Property List Editor that comes in the Developper Tools. Don’t use BBEdit, just because it’s a simple text editor and they make you pay for it. Why not using what is free when you can, while you can.

Make it yours
In this section, we’ll be morphing my plist to reflect your environment. The first real “key” is Program. It is the absolute path to bixagent/bixserver folder within bixdata folder. Be very careful here. The only text that you need to edit is in blue. Usually, you will see my “bixAgent” as bixdata if you just uncompressed the archive.

Here, the “bixServer” is usualy bixdata in the standard archive.

Next you need to edit StandardErrorPath and StandardOutPath to reflect your log facility. You may remove the following section if you don’t want to log it’s activities. For the server, you may or may not change the log file.

Next, you have the user executing section. That section tells launchd on who’s authority to run the program. Again, that section might be remove if you don’t care at all about security. Bad choice if you do, you should know at this point what you are doing. The blue section is the only field you need to change here. In my setup, the agent and the server runs as the same user.

The last but most important variable is the WorkingDirectory. That working directory tells where to change directory to before executing arguments. Previously, I said that bixAgent/server is very picky from where you start the software, this is where it takes place. The blue part must reflect your installation path. Note here that this is ALMOST the same path as in the Program section without the trailing “/bixagent”.

Same thing here for the server.

Lock and load
Save that plist file in your /Library/LaunchDaemon folder. If you’re not a Terminal person, just reboot and everything should go smoothly. If you want to go the CLI way, open up a Terminal window and get root, move to LaunchDaemon folder and load the plist.

darwin:~ francois$ sudo su
Password:
darwin:~ root# cd /Library/LaunchDaemons/
darwin:/Library/LaunchDaemons root# ls
info.frjo.BixAgent.plist
darwin:/Library/LaunchDaemons root# launchctl
launchd% load info.frjo.BixAgent.plist

Now check if the plist has been loaded and it doesn’t crash. In that case, “list” in launchctl should display your plist file as following.

launchd% list
…….
com.apple.tftpd
info.frjo.BixAgent
launchd%

If you see it. Congratulation, you followed the documentation above carefully. By confirming that your plist has been loaded like above, you made sure that your system will always load it at startup. Next, you can reboot or start it manually.

launchd% start info.frjo.BixAgent
launchd%

You should see in your process list something that is started by the user you chosed to run it as with the process name at bixagent.

darwin:/Library/LaunchDaemons root# ps aux|grep bixagent
root 7483 1.9 0.0 27372 424 p1 S+ 11:26PM 0:00.01 grep bixagent
bixdata 3848 0.0 0.1 34136 1064 ?? Ss 2:06PM 1:46.67 bixagent
darwin:/Library/LaunchDaemons root#

At that point, I like to try out the configuration and simulate the process crash. Perform the following command as root to kill, stop, the bixagent process.

darwin:/Library/LaunchDaemons root# killall bixagent

The action you just performed stoped the agent. Launchd should have restarted it. To check out if it did, perform the “ps aux|grep bixagent” command like above and you should have something simmilar. The “3848″ in my example is the process id. After killing it, this number MUST be diffrent. If it’s the same, then you didn’t kill it properly.
You are done. Every time your daemon crash or get a “Broken pipe” it will restart. Every time you reboot, it will start at load. Nice ein!!!

Troubleshooting
A well known section here

Case1: In your system.log(use tail CLI command or Console GUI app) if you see something like the following, the have your path messedup. The Program or the WorkingDirectory path are wrong.

Aug 20 20:53:44 darwin launchd: info.bixagent: execv(”/Network/Servers/darwin.lan/Users/bixdata/bixAgent/bixagent/bixagent”, …): No such file or directory
Aug 20 16:53:44 darwin launchd: info.bixagent: execve(): No such file or directory

Case2: Again, in your system.log, you might see the following. In that case, you have a non standard xml file. You provably used a text editor that added up some non xml standard values.

Aug 20 16:59:30 darwin launchd: info.bixagent: exited abnormally: Segmentation fault
Aug 20 16:59:30 darwin launchd: info.bixagent: respawning too quickly! throttling

Need more help on that, post comments and I’ll read them.

References
Known good article about Launchd
BixData center

Download it.
You may get bixagent from here at bixdata site or the package I made to deploy easily the agent. With the package, there is no need to compile or configure. It installs bixagent with ssl enabled and credentials already set.