I'm back - ugh

Well this has been fun.

My webserver had long been hosted in a customer's computer room, where it had plenty of physical security, good bandwidth, controlled environment, clean power, and two years of uptime. What's not to like?

Two months ago the huge UPS that powers the whole room failed, and the crash took out a hard drive in my webserver. This webserver is an old but rugged Compaq Proliant with RAID5 for the main content, and a spare standalone drive for a nightly online backup - the backup drive is what failed (and it failed hard; disassembly showed oxide scraped off the platters). Bummer, but no big deal, it was just a backup drive.

I replaced it and backed up (and verified) the whole system and then started an OS upgrade to replace the aging Red Hat 6.2, and a few hours later had a shiny new operating system on a RAID 5 partition, all ready to accept the saved data.

Or not.

The data on the backup drive was bogus, and none of my recovery tools would even pretend to find the partition or the filesystem. I decided that the RAID controller probably ate it, taking down nearly my whole website; so much for having a backup.

Thankfully, I had nearly all my static content checked into a Perforce SCM depot on my home network, so the better part of unixwiz.net lives, but everything else - tools, weblog, email, other sites, etc. - are gone.

So I got myself a new rackmount box - a Compaq Proliant 1850R - and have parked it at a data center, and am slowly rebuilding everything.

And setting up an offsite rsync backup.

Comments

[FrE]

Hello! Nice Website... I glad I could access the site... with all what you said!
I am currently reading 14 things in the TechTips section....

Thank you for the work....



FrE

[david]

Can never have enough backups..come to think of it, I should probably be checking all my data into a subversion/CVS repository and syncing it around.

This box feels much faster at previewing than your older one as well. It was probably a sign that it was time to get a new one..

[Danne]

Glad to see you back up. I just started to subscribe to your RSS feed, until it disappeared. Nice boxen too!

[Wendy]

Nice to see you back up.

[Stewart Vardaman]

I was wondering what was up. I just happened to pop back in while reviewing my bookmarks.

rsync is a wonderful thing. My employers's web servers are primarily in Biloxi, MS. This week we moved a few boxes out of harm's way, using rsync to keep things tidy in case we needed to make a rapid switchover Thursday AM.