System Management

I received a call over the weekend from a client. A mission critical server had failed and will not boot giving a dreaded RAID array failed message. This server has all of the clients data and they cannot afford to lose any of it.

After arriving, I was able to give the client some peace of mind in that the RAID array that failed was only the operating system RAID 1 set. They had another RAID 5 array with all their data. The first major issue that I saw was they had a hot spare but it was configured to be a hot spare for the RAID 5 array only. Keep in mind all the drives are the same size, type, and speed so there is no need to have 2 hot spares for both RAID arrays. So, the hot spare did not save the failed RAID 1 array.

In looking at the status of the hard drives that failed (both in the RAID 1, which seemed very unlikely) I noticed one of the drives had failed 1 year ago. You read it right, the drive failed 1 year ago (off by a month). I first thought how in the world can you not know a drive has failed for 1 year? If you are wondering, they do have a system administrator.

Another issue I noticed is that the system administrator explained to me that server was not being backed up! I know what you are thinking but I'm not going to say it. He knew the server should be backed up, but hadn't gotten around to it.

After being in the midst of this perfect storm I looked at the failed hard drives. One of the drives did not have any SMART errors so I decided to force it online. After the OS booted I then set the hot spare as a global hot spare which then the RAID 1 took it right away and started replicating the data.

What this got me thinking about is about failed hard drives and notification. Most servers have some sort of RAID setup. But, what if you don't go in the server room very often. How do you know a hard drive is failed? You need system monitoring with alerts and notifications, and what options do you have? The first that comes to mind is System Center Operations Manager.

With System Center Operations Manger you can install management packs to manage and monitor different systems. When something goes wrong, the SCOM will alert you. There are of course other management applications that you can use to monitor your systems like HP System Insight Manger, Dell Open Manage, ipMonitor, and others. Each environment needs its own system monitoring solution. You might be able to get away with one, or you might need to incorporage multiple management systems. Either way, this can be as important as you backup and should be treated as such.