Yet another voice in the cacophony.
Monday, October 21 2019 @ 04:09 PM PDT

 What's New  
No new stories

COMMENTS last 48 hrs
No new comments

There are no upcoming events

 User Functions  


Don't have an account yet? Sign up as a New User

 Other Stuff  
< ? bostonites # >
The Syndicate

 They say they come in threes   
 Author:  matt
 Dated:  Tuesday, August 31 2004 @ 11:45 AM PDT
 Viewed:  993 times  
LifeWhen I woke up yesterday morning, I felt like somebody was standing on my balls. Not literally, of course, or I'd be in a hospital right now, and probably better off. It felt (not surprisingly, in retrospect) like that special kind of jet lag you only get when you've been having a little too much fun in Vegas, and then take the red-eye back to the east coast.

I had, in fact, spent pretty much all weekend working and simultaneously adjusting my sleep schedule so that I'd actually be awake on Monday morning. I even managed to lose a day along the way... Friday, I think. Even after putting in that hellish weekend, I was still about a day behind on delivering a software release. Thus began my day.

By about 8PM, I had become sufficiently frustrated that I was cursing (loudly) at my computer, using all sorts of colorful imagery in a vain attempt to convince the aforementioned bucket of bits to Do What I Mean. A few more rounds and a couple of cigarettes later, I had reached a heretofore unimaginable pinnacle of crapulence.

In an effort to capture precisely the meanness and rage I was feeling, I said to my coworker, "If there were a three legged dog in the room, I'd be kicking it."

Given my emotional state, it's perhaps not that surprising that I very nearly lost my tenuous grasp on reality when a new and even darker page opened on the day in the form of an e-mail from the CEO. You see, it wouldn't open.

I did what any rational person would do. I exclaimed loudly and came very close to competing in the 23 inch CRT Shot Put finals before I settled down a bit, and tried to connect to our web-based email system, which also wouldn't open. You see, our primary server was down.

I banged my head on the screen a few times, before posting a request to our hosting provider for a reboot of said machine. I grew more and more impatient as minutes turned to dozens, and vague responses from the ISP finally, an hour later, coalesced into what should have been obvious from the get-go: we had cooked a hard drive.

I say obvious because not two days earlier this same drive had taken down the system for an hour or so because it had reached capacity. "Surely", you must be thinking, "disk space is planned for!", and it is. We make sure we have three times the disk capacity that we expect our customers to use. But, you see, we had a few extra logfiles.

The other day, as I removed 18 Gigabytes of extraneous logfile, not once did I think that this was a tremor precursing a larger quake. Hindsight being what it is, the pieces all fall into place. Whether or not the drive space issue contributed to the eventual failure of the device is academic. What's pertinent, however, is the fact that having that much data on the drive seems to have been causing some issues with our network backup software, such that our most recent full backup is five days old.

Which leads me to today. The guys at hosting facility were able to get the operating system to load from the old drive (after a bit of coaxing), so the machine is running critical services again (as ironic and dangerous as that sounds, and is), and I'm going through it, by hand, pulling important data off while the normal backup system attempts to struggle through its daily(ish) ritual. This, of course, is simply preperatory work for the hard drive replacement that we'll be doing at some point in the next 72 hours.

All of which is a long way of saying that I'll be cleaning up a server for pretty much the next week or so, but that's okay because I couldn't go on my annual Labor Day vacation this year anyway, because the boss is out of town so I have to be on call.

Oh, and at this point I've been awake for about 26 hours, and it doesn't look like I'm going to get to sleep until sometime this evening. The real bitch of it is that my department is so understaffed and without budget that I can't do my job effectively, and things like this happen. I know how to prevent every single one of the contributing factors in this debacle, but I can't get us set up on a machine with RAID, I can't get a nice tape machine to handle backups, I can't bring in more programmers to handle the software load, I can't set up anything remotely resembling a high availability cluster, I can't... you get the idea.

It does kinda make me wonder... what are the other two going to be?

 What's Related  
  • More by matt
  • More from Life

  •  Story Options  
  • Mail Story to a Friend
  • Printable Story Format