Skip to main content

This log was made for rolling, rolling, rolling....

So I promised a blog post about monitoring and alerting in IT systems, so here it is. As with most liberal arts snobs I have a 'personal philosophy' about how to do things the best way. This philosophy is broken down in to three different components, based upon two critera; timeliness and context. Timliness in terms of how quickly an event needs to be acted upon, and the context in which the event is applicable. Therefore, a holy trinity of monitoring allows IT professionals to get the best information possible for any given situation. After all... the more you know.....

First off is the most immediate in timeliness, that of immediate alerting. When a system is about to come crashing down, seconds are of the essence. It's in this context that immediate alarming and alerting allows 1st level responders to get in to a system at the first sign of trouble. With any luck, they can repair and cirvumvent any problems before the problem is noticed by the client. This type of alerting involves alarms about what's going on in the system right now, at this moment. Often it involves monitoring of hardware and networks, and databases. There are lots of different tools that exist in the application sphere to accomplish this type of monitoring, and many also include hooks to ticketing systems and gateways to send texts and pages to appropriate support personnel. This is the type of system that you would see an operator sitting in front of, watching for any sign of trouble.

The second type of alerting and monitoring is what I call mid-term monitoring. This is the type of alerting that helps in a system health context, but is not immediate. When considering this type of alerting questions such as, "How did my system do over the past 3 days?" and "Have there been any spikes at odd times over the past week?" The context is still system health, but the perspective is much broader. The best mid-term monitoring solutions encompass more than just mid-term monitoring, but also integrate log interrogation. These tools help not only keep an eye on system performance over a period of time, but allow for debugging of system logs in ways that are fast and effecient. One of the best examples of a tool in this field is called Splunk, probably the finest log monitoring and indexing software out there right now.

The final level of monitoring is far less immediate, and it revolves around capacity management. In capacity management, we are most concerned with how our resources are being utilized over a long period of time. Statistics need time to gather and be aggregated to be useful for capacity management. A single spike on a single day may signal trouble to the first two levels of monitoring and alerting, but for capacity management you want to see that spike continue before you are concerned. Where as the immediate monitoring moves at the speed of a hare, capacity management is slow and careful like a tortiose.

Combined, these three types of monitoring allow systems to be as visilble as possible to the IT staff maintaining them. Coming up with a good strategy for dealing with each of these levels is key to building a successful enterprise IT system.

Happy logging!

Comments

Popular posts from this blog

The beat goes on

Yesterday Apple revealed their long awaited entry into the streaming music field. They were able to do this quickly because of the acquisition of Beats last year, and the systems and intellectual property that came with that purchase. Considering that the music reveal was pretty much the only big news out of a pretty benign developer keynote, I'll take a few moments to talk about what I think about it. Apple was perhaps the defining company in the music revolution of the past 20 years. With the introduction of the iPod that revolutionized portable music, to the creation of the iTunes store and the eventual death of DRM, Apple has been at the forefront of digital music. This leadership comes with high expectations to continue to lead, and so many people have long questioned Apple not getting into the streaming music business quicker. For the past few years new companies have come forth to lead the change in the streaming music evolution. From Pandora and its ability to create un

The Great Experiment

Recently, a tech journalist that I've followed for many years, and who is an Apple fanboy, posted a series talking about why he switched from an iPhone to an Android phone . It's a good read, and worth the time to see why he made the decision he did. Since I have a Verizon Galaxy Nexus sitting on my desk as a Wi-Fi device, I thought, "What the heck, let's give this a go for a week." So for the past week I've shelved my trusty iPhone 5 and have delved deep into the world of stock Android 4.1. So in the spirit of "copying is the sincerest form of flattery" here's my write-up of my experiences with Google's mobile OS. First, I need to make one caveat. After using the Nexus for a week I have to say that I do NOT like this device. It constantly loses 4G signal, and the battery life almost makes it unusable. I could barely make it to lunch before I was at 20-30% battery. So in the spirit of fairness, if I truly wanted to switch full time to Andro

CES 2013

This past week was a big week for the tech industry, with the holding of the Consumer Electronics Show 2013. Recent years have been a bit 'meh', but this year really had some interesting tech show up. In particular the theme seemed to be changes coming to our living room TV's. Much of what we saw this year revolved around ways to get entertainment to our TV's with set top boxes that tie into other services, or all new TV technology like 4k (Ultra High Defenition). Personally, I'm less excited about UHD, since I just bought a new TV, and am quite happy with it. Plus, I don't think we have the internet bandwidth for UHD content yet. The really cool advances are less technological for me, but structural. One of the things I love about internet delivered entertainment, is the ability to control what you watch a LOT more than the old days of flipping cable channels. I love the idea of discovering a new show, downloading an entire season and devouring it as quickl