Skip to content

Entries tagged "dmonitor".

I'm fireproof, you're not

7th May 2009

I've mostly been avoiding the computer this evening, but I did spend the last hour working on attempt #2 at distributed monitoring.

The more I plot, plan & ponder the more appealing the notion becomes.

Too many ideas to discuss, but in brief:

My previous idea of running tests every few minutes on each node scaled badly when the number of host+service pairs to be tested grew.

This lead to the realisation that as long as some node tests each host+service pair you're OK. Every node need not check each host on every run - this was something I knew, and had discussed, but I assumed that would be a nice optimisation later rather than something which is almost mandatory.

My previous idea of testing for failures on other nodes after seeing a local failure was similarly flawed. It introduces too many delays:

  • Node 1 starts all tests - notices a failure. Records it
    • Fetches current results from all neighbour nodes.
    • Sees they are OK - the remote server only just crashed. Oops.
  • Node 2 starts all tests - notices a failure. Records it.
    • Fetches current results from all neighbour nodes.
    • Sees they are OK - the remote server only just crashed. Oops.

In short you have a synchronisation problem which coupled with the delay of making a large number of tests soon grows. Given a testing period of five minutes, ten testing nodes, and 150 monitored hosts+services, you're looking at delays of 8-15 minutes. On average. (Largely depends on where in the cycle the failing host is, and how many nodes must see a failure prior to alerting.)

So round two has each node picking tests at "random" (making sure no host+service was tested more than 5 minutes ago) and at the point a failure is detected the neighbour nodes are immediately instructed to test and report their results (via XML::RPC).

The new code is simpler, more reliable, and scales better. Plus it doesn't need Apache/CGI.

Anyway bored now. Hellish day. MySQL blows goats.

ObFilm: Hellboy

Tags: dmonitor, mysql. | 2 comments.

 

A lot of people drink mineral water

4th May 2009

dmonitor now has a webpage.

I've been running it for a night now, watching alerts come and go via manual firewall rules so I'm pretty confident it works reliably and in a way that avoids transient failures. The only obvious failure case is if each monitoring node loses the links to each of the others. (Solution there is to have a sufficiently large number of them! Hence the reason the configuration is file/directory based. rsync for the win!)

Still I will leave it there for now. The only things missing are better instructions, and more service checking plugins..

Now to go paint some more ... adding new wall decorations has inspired me a little!

ObFilm: Heathers

Tags: dmonitor, flat, hacks, space invaders. | No comments

 

We can't just let you walk away!

3rd May 2009

My Desktop

I use a number of computers in my daily life, but the machine I use the most often is my "desktop box". This is one of a pair of machines sat side by side on my desk.

One machine is desktop (Sid) and one is the backup host (Lenny). The backup machine is used by random visitors to my flat, and otherwise just runs backups for my remote machines (www.steve.org.uk, www.debian-administration.org, etc) every 4/6/24 hours.

I apply updates to both boxes regularly but my desktop machine tends to have lots of browsers open, and terminals. I rarely restart it, or logout. So the recent updates to X, hal, udev, mostly pass me by - I can go months without logging out and restarting the system.

On Saturday the desktop machine died with a OOM condition when I wrote some bad recursive code for indexing a collection of mailboxes. Oops.

When it came back I was greeted with a new login window, and all the fonts look great. Now in the past the fonts looked OK, but now? They look great.

I cannot pin down what has changed precisely, but everything looks so smooth and sexy.

So, a long entry, the summary is "I restarted my machine after a few months of being logged in, and now it looks better".

Distributed Monitoring?

Random conversation with Alex about monitoring yesterday made me curious to see if anybody has put together a useful distributed monitoring system?

Assume you have a network with Nagios, or similar, monitoring it. If your link between the monitoring box and the hosts being checked is flaky, unreliable, or has blips you will see false positives. We've all been there and seen that.

So, what is the solution? There are two "obvious" ones:

  • Move the monitoring as close to the services as possible.
  • Monitor from multiple points.

Moving the monitoring closer to the services does reduce the risk of false positives, but introduces its own problems. (i.e. You could be monitoring your cluster, and it could be saying "MySQL up", "Web up", but your ISP could have disconnected you - and you're not connected to the outside world. Oops. The solution there is to test external connectivity too, but that re-introduces the flakyness problem if your link is lossy.)

Distributed monitoring brings up its own issues, but seems like a sane way to go.

I wrote a simple prototype which has the ability to run as a standalone tester, or a CGI script under Apache. The intention is that you run it upon >3 nodes. If the monitoring detects a service is unavailable it queries the other monitoring nodes to see if they also report a failure - if they do it alerts, if not it assumes the failure is due to a "local" issue.

There is a lot of scope for doing it properly, which seems to be Alex's plan, having the nodes run in a mesh and communicate amongst each other "Hey I'm node #1 - I cannot see service X on host Y - Is down for you too?" - but the simple version of having the script just do a wget on the CGI-version on the other nodes is probably "good enough".

I really don't track the state of the art in this field, just struggle to battle nagios into submission. Do there exist systems like this already?

(sample code is here, sample remote-status checks are here and here. Each node will alert if >=2 nodes see a failure. Otherwise silence is golden.)

ObFilm: X-Men Origins: Wolverine

Tags: desktop, dmonitor, fonts, sid. | 3 comments.