PortSensor Development Blog

Hey there!

I'm happy to say PortSensor has now fully scratched the itch we felt while using our previous server monitoring tools over here at WebGroup Media.

I'll be honest, there was a dark period in the history of our company where the majority of server issues were being brought up by customers before we even noticed them.

It wasn't negligence, we just always have more R&D projects going than we have people. While that alone is a terrible excuse, the situation was exacerbated by the fact our previous tools didn't realize the real problem -- we had better things to do than constantly check a browser window or inbox. With these other monitoring tools, at best, we were finding out about issues around the same time as our most active hosting customers noticed their sites performing poorly.

That's why we approached PortSensor differently. It's a desktop tool that shows a green, yellow or red LED in your systray. As the status changes a non-intrusive pop-up lets you know what's going on. You can set up your own notifications without an administrator's help.

If you're away from your desk, PortSensor works like you'd expect -- it can fire off an alarm or send you an email. You can also nest notifications so a monitor like disk space emails your desk at 80% utilization or your cell phone at 90%.

So that brings me to our real question. PortSensor is great, why aren't more people using it?

After some discussion, inside and outside WGM, our team felt like the current free version (1 server with 5 sensors) probably isn't conducive to getting a feel for everything PortSensor can do. It's definitely not enough to become acquainted with how great SSH sensors are for monitoring metrics like disk space, load and processes on Unix-based machines. A lot of people hadn't even tried SSH Sensors or Custom Notifications.

We thought about raising the sensor/server limits, but decided to do one better. PortSensor is now free for full functionality and unlimited sensors across unlimited servers. The sole exception is currently Custom Notifications which let you set various thresholds on a sensor with a responding action. That's now the pivotal feature between 'free forever' and 'help pay our rent'.

We'll happily generate a trial license for anybody who wants to explore the Custom Notifications functionality. And that's exactly what we'll do as a bonus for everybody who downloads PortSensor and provides some info on the little survey we have up on the site.

And hey, if you think we've pegged the wrong culprit and you'd like to add some comments, the forums (http://www.portsensor.com/forums) and my inbox (jeff AT webgroupmedia.com) are always open.

The downloads are updated on the website!

-Jeff Standen, Project Lead

Hot off of the heels of our PortSensor 2 release we've added even more features and enhancements. For those of you already using PortSensor you can update from within PortSensor itself by going to "Help"=> "Software Updates" => "Check for Updates". To try a fresh install you can find the download at http://www.portsensor.com/download.php . Here's an overview of what's new:

* Rule Operators

Previously, PortSensor has been capable of powerful validation thanks to the ability to create customized notification rules. We found that quite often this capability can be overkill, and regular expressions are not always the easiest way to solve simple validations like number comparisons. Now, when creating a rule you can specify that the response from the server must be less than, greater than, or equal to a value you specify (and when you need the power of regular expressions they are always an available option).

* Combined the status and output columns

To save some more space and cut down the clutter, we've combined the status column with the output column. We did this because the vast majority of the time your sensors (hopefully) have a status of OK, and you can tell that is the case from the colored icon as well as the color of the output column. If something goes wrong with one of your sensors, an error message will display prepended to the sensor output in the output column. This is definitely an improvement in situations where the error message was so long you had resize the status column just to be able to read it!

* Display of rule triggered in the output

When a sensor's status changes to critical or warning because of a notification rule, the name of the rule and its attributes (e.g. "WARNING if > .50") will display prepended to the output column. This makes it easier to know why a sensor has failed if the cause was due to a rule, and is certainly more informative than the generic "Validation Failed" message it displayed before.

* Ability to hide disabled sensors.

Oftentimes, sensors we've disabled just get in the way of the sensors we really want check on, so we've added a button to hide them next to the button that toggles the ability to show only critical status sensors.

* More useful data in notification emails.

We've spruced up the content in the notification emails so now it contains the server name, any error message or rule triggered, and the actual output returned by the sensor. This is in addition to the sensor's name and status. The result gives a much better idea of what is going on when a sensor has failed through whatever device you check email from without always having to run and look at PortSensor itself.

* Improved manual refresh

The refresh button has been improved so it will now always run the "maximum sensors to run at once" (from the preference setting) number of sensors. Before, it would skip sensors that it had run recently and whose next scheduled run time had not yet been reached. With this improvement you can always refresh all of your sensors systematically with confidence for when you can't wait until their next scheduled run times.

* Sensor wait countdown timer column.

To give you a better idea of when and how often your sensors are set to run we've added a column which displays a countdown for each sensor to give you an idea of how long before that sensor will run again. Some might find it a little distracting so we've hidden it by default. To reveal it and see a little more behind the scenes of PortSensor simply drag the rightmost side of the last column divider. This lets you tell at a glance if any of your sensors aren't set to refresh often enough.

* Notification rules and actions now display below their sensors in Overview

Now, notification events, rules, and actions can be seen in PortSensor's main view. Any sensors that you have enabled custom notifications for will display with an expandable tree node, so it's easy to tell which sensors you've defined notifications for, and which are using the defaults. It also makes it easier to see what rules will influence the status of your sensors without having to right-click and open the edit dialog every time.

* Shortened last check date column to display only the time to save screen space.

* Several other minor improvements and bug fixes.

Mike Fogg, Lead Developer
PortSensor Project - http://www.portsensor.com/
WebGroup Media, LLC. - http://www.webgroupmedia.com/

What's New in PortSensor 2.0

At a glance:
* Multiple Notifications per Event
* Nested Alerts
* Sensor Run/Schedule Optimization
* SSH Delegate
* MacOSX (x86) Support

The major focus of PortSensor 2.0 was to add more flexibility to the alarm system. You can now nest alarm rules and configure multiple actions to respond with at any point. That means you can now do a lot of interesting things, especially with SSH sensors, such as escalating alerts on a server disk space sensor every 10%.

The alarm rules system does add some complexity, but every sensor has the option to use 'defaults' which will behave exactly like PortSensor 1.0 -- simply playing a sound if anything is wrong.

Another major focus of 2.0 was the optimization of many concurrent sensors. We set up a much better scheduling system which ensures that the sensors running in any given batch are the sensors that have been waiting the longest to run. In previous builds it was possible to always start from the top of the list, with a small number of runs per batch and short sensor wait times, and not always get full sensor coverage.

An additional aspect of optimization had to do with SSH sensors. In prior builds PortSensor would open an SSH connection per sensor. In 2.0 we use a single SSH connection per server which is responsible for running any attached SSH sensors. This removes a lot of SSH overhead while checking sensors, as well as fixing a potential issue where several monitoring users could tie up the available SSH connections on a machine.

You'll also notice a new graphic at the top of the application window. We felt there were enough boring server monitoring tools -- PortSensor now has some personality!

You can grab the new builds from:
http://www.portsensor.com/download.php

We'd love to hear your feedback, both about the changes and what you think we should be working on next. Drop by the forums or send a message to support@portsensor.com with your thoughts.

Special 2.0 Price: In the interest of growing the PortSensor Community we've also dropped the price of PortSensor 2.0 to $99 for unlimited servers and sensors. (One year of support and upgrades is included)

Thanks for your interest!
--
Jeff Standen, Project Manager
Mike Fogg, Lead Developer
PortSensor Project - http://www.portsensor.com/
WebGroup Media, LLC. - http://www.webgroupmedia.com/

When you first launched PortSensor you may have wondered, "How do I monitor server load?". It's a good question, and the answer isn't very obvious from staring at the bewildering simplicity of a fresh copy of PortSensor.

PortSensor has an enormous amount of power and flexibility hidden behind a few simple tools -- primarily the biggest product-defining feature we have: SSH Sensors. A couple of our inspirations deserve a quick nod for pioneering this concept in modern computing: Unix/Linux and the GNU Project (among many other related projects). These are ingenious examples of combining individually simple tools to do unimaginably useful and creative things.

As such, you're going to get the most mileage out of PortSensor and this article when you're monitoring a Unix-based server (Solaris, Linux, BSD, MacOSX, etc). That doesn't mean if you're a fellow 'Chief Slave to the Servers' at a Windows shop you couldn't hack a couple things together and play along (doing so would make an interesting future blog entry itself), but for our example we're going to use the magical Unix-based shell.

Experimenting with SSH Commands

First, just open an SSH session as you normally would with one of your servers -- assuming “normally” means you end up with a shell prompt (as opposed to blinking binary LEDs on the front panel of your Altair, or whatever).

Let's try a couple quick things at the console. Don't worry if your distro doesn't support something throughout this tutorial, the real goal is giving you the fundamental ability to create useful new sensors on your own in your chosen environments. I'll try to keep things generic -- all my examples are using the BASH shell on a Red Hat Enterprise Linux server.

Let's take a look at our disk space by partition/filesystem.



Now if we were PortSensor, we would want a summarized version of that output to display next to our sensor. We just want the important part -- how much disk space we've used on our root partition, so we can fire a notification when we're running low on space. There are a number of ways to parse the output, but let's focus on grep and awk since they will be invaluable new tools to you if you haven't encountered them before.

Let's print just the root partition. You're going to want to substitute the proper value you're seeing in your output. In our case it's /dev/sda5.



Now let's use awk to reorganize the data. At this point it doesn't matter what awk is, you can punch it into Wikipedia later.



The $5 instructs awk to print only the 5th column. The resulting output is perfect for our sensor status report, so let's create an SSH Sensor in PortSensor.

Turning Our Example Into an SSH Sensor

Fire up PortSensor and create the appropriate Server Group and Server if they don't exist yet. If you need a walk-through, refer to the Getting Started guide. Once you have a server created you need to modify it to fill in the SSH Details section. You can authenticate using a private key (e.g., RSA) or with a simple password. For now just use whichever method you used to log in for the example. Your password/passphrase will be encrypted in your XML file, though if you aren't using private keys you should always blank out the data before sending your file to somebody through an unsecured channel (like e-mail). Be sensible, don't use root. In the future we'll build this extra security functionality into an 'export' feature.



Right-click on your server and select 'Add Sensor' from the menu. Create an SSH Sensor as follows:




Now double-click your new sensor to refresh it. You should see the following:


You're probably wondering why we see two boxes after our output above. In the Windows version of PortSensor, the table renders non-printable characters (in this case, carriage return and line feed) as squares. The Linux version is capable of printing multiple-row table cells, though that may get annoying (especially when they're blank).

Here's one way to fix it. Append the following command to the end of your Run Command:

| tr -d "\r\n"

tr is another GNU goodie. It stands for Translate, and the -d enables delete mode for CR (\r) and LF (\n) characters.

If you need multiple lines to print on a single line without running together, you could use something like:

| tr "\r\n" ' '     <-  That's a quoted space in the second argument.

Your sensor output should now look like:


TIP: You may occasionally decide to keep carriage returns in your output since they display properly in a tooltip when you hover over a table cell. This means you could make a sensor draw the entire table of the df -h command when you hover over the cell.

That's all there is to it!

You may have noticed we had the option to enter a regular expression to validate our sensor output. That's really a topic for another article, but if you wanted something quick and dirty to give an alarm at 90% you could use a regular expression like:

^([0-9]%|[0-8][0-9]%)$

That says, "If the output is a single digit 0-9%, or a double-digit 00-89%, then we're okay. If it's anything else (such as 95%) then fail."

More SSH Sensor Examples
Here are a couple other interesting command combinations you can use for SSH Sensors:

Top 5 Running Processes by CPU %:

ps -eo "%C%% %c" --no-headers | sort -r -n | head -n 5

Server Load Averages:

cat /proc/loadavg | awk '{print $1,$2,$3}' | tr -d "\r\n"

Qmail Loads:

/var/qmail/bin/qmail-qstat

(note: your SSH user will need to be a member of the qmail group)


Feel free to share your own command combinations in the comments.

Enjoy!
-Jeff@WGM

Someday soon we plan to implement a 'Discovery Mode' ability where PortSensor can automatically detect servers and configure detected services. That will save a lot of hassle when configuring the a monitoring file for the first time.

Until that day comes, however, it's useful to know some shortcuts in creating a bunch of similar sensors across several servers.

For this example let's assume:

• We have three groups of servers based on their OS (Windows, Linux and Solaris).
• Each group has a single server.
• We want to create sensors on each server to monitor Ping, SMTP and POP3.
  

Here's what we need to do:

1. First, we need to create the Ping, POP3 and SMTP sensors on the first server in the list.
   
   
   
   Figure 1: Creating an SMTP Sensor
   
   
   
2. Once you have all three sensors set up, right click on each sensor and select Duplicate from the menu.
   
   
   Figure 2: Duplicating a Sensor
   
   
   
3. Now simply drag each copied sensor to the next server. Repeat this 'Duplicate and Drag' process until all your servers have copies of your desired sensors.
   
   
   Figure 3: Dragging a Sensor to a New Server

That's all there is to it! This process is especially a time-saver when working with more complex sensors, such as SSH Sensors, which have several properties you need to configure.

Happy monitoring!