PortSensor Development Blog

When you first launched PortSensor you may have wondered, "How do I monitor server load?". It's a good question, and the answer isn't very obvious from staring at the bewildering simplicity of a fresh copy of PortSensor.

PortSensor has an enormous amount of power and flexibility hidden behind a few simple tools -- primarily the biggest product-defining feature we have: SSH Sensors. A couple of our inspirations deserve a quick nod for pioneering this concept in modern computing: Unix/Linux and the GNU Project (among many other related projects). These are ingenious examples of combining individually simple tools to do unimaginably useful and creative things.

As such, you're going to get the most mileage out of PortSensor and this article when you're monitoring a Unix-based server (Solaris, Linux, BSD, MacOSX, etc). That doesn't mean if you're a fellow 'Chief Slave to the Servers' at a Windows shop you couldn't hack a couple things together and play along (doing so would make an interesting future blog entry itself), but for our example we're going to use the magical Unix-based shell.

Experimenting with SSH Commands

First, just open an SSH session as you normally would with one of your servers -- assuming “normally” means you end up with a shell prompt (as opposed to blinking binary LEDs on the front panel of your Altair, or whatever).

Let's try a couple quick things at the console. Don't worry if your distro doesn't support something throughout this tutorial, the real goal is giving you the fundamental ability to create useful new sensors on your own in your chosen environments. I'll try to keep things generic -- all my examples are using the BASH shell on a Red Hat Enterprise Linux server.

Let's take a look at our disk space by partition/filesystem.



Now if we were PortSensor, we would want a summarized version of that output to display next to our sensor. We just want the important part -- how much disk space we've used on our root partition, so we can fire a notification when we're running low on space. There are a number of ways to parse the output, but let's focus on grep and awk since they will be invaluable new tools to you if you haven't encountered them before.

Let's print just the root partition. You're going to want to substitute the proper value you're seeing in your output. In our case it's /dev/sda5.



Now let's use awk to reorganize the data. At this point it doesn't matter what awk is, you can punch it into Wikipedia later.



The $5 instructs awk to print only the 5th column. The resulting output is perfect for our sensor status report, so let's create an SSH Sensor in PortSensor.

Turning Our Example Into an SSH Sensor

Fire up PortSensor and create the appropriate Server Group and Server if they don't exist yet. If you need a walk-through, refer to the Getting Started guide. Once you have a server created you need to modify it to fill in the SSH Details section. You can authenticate using a private key (e.g., RSA) or with a simple password. For now just use whichever method you used to log in for the example. Your password/passphrase will be encrypted in your XML file, though if you aren't using private keys you should always blank out the data before sending your file to somebody through an unsecured channel (like e-mail). Be sensible, don't use root. In the future we'll build this extra security functionality into an 'export' feature.



Right-click on your server and select 'Add Sensor' from the menu. Create an SSH Sensor as follows:




Now double-click your new sensor to refresh it. You should see the following:


You're probably wondering why we see two boxes after our output above. In the Windows version of PortSensor, the table renders non-printable characters (in this case, carriage return and line feed) as squares. The Linux version is capable of printing multiple-row table cells, though that may get annoying (especially when they're blank).

Here's one way to fix it. Append the following command to the end of your Run Command:

| tr -d "\r\n"

tr is another GNU goodie. It stands for Translate, and the -d enables delete mode for CR (\r) and LF (\n) characters.

If you need multiple lines to print on a single line without running together, you could use something like:

| tr "\r\n" ' '     <-  That's a quoted space in the second argument.

Your sensor output should now look like:


TIP: You may occasionally decide to keep carriage returns in your output since they display properly in a tooltip when you hover over a table cell. This means you could make a sensor draw the entire table of the df -h command when you hover over the cell.

That's all there is to it!

You may have noticed we had the option to enter a regular expression to validate our sensor output. That's really a topic for another article, but if you wanted something quick and dirty to give an alarm at 90% you could use a regular expression like:

^([0-9]%|[0-8][0-9]%)$

That says, "If the output is a single digit 0-9%, or a double-digit 00-89%, then we're okay. If it's anything else (such as 95%) then fail."

More SSH Sensor Examples
Here are a couple other interesting command combinations you can use for SSH Sensors:

Top 5 Running Processes by CPU %:

ps -eo "%C%% %c" --no-headers | sort -r -n | head -n 5

Server Load Averages:

cat /proc/loadavg | awk '{print $1,$2,$3}' | tr -d "\r\n"

Qmail Loads:

/var/qmail/bin/qmail-qstat

(note: your SSH user will need to be a member of the qmail group)


Feel free to share your own command combinations in the comments.

Enjoy!
-Jeff@WGM