Incoherent Babble

Ok, here’s the deal… At work we have a primary file server that has just under 1TB of storage in a RAID 5 array (5 x 250gb SATA - parity). The data is fairly evently split in half - one half user data, one half claims archival (remember, we’re in insurance…).

Presently we’re backing up to tape, performing full backups on a 3:2 schedule: Monday, Wednesday and Friday nights user data is backed up, Tuesday and Thursday nights the less-often-modified claims are backed up. Each tape holds 160gb (give or take - PNGs don’t compress all that much, so it’s far from the theoretical 320gb for them - data only partially evens it out since a good bit of it is TARs of database dumps and the like). At this point we’re doing about 230gb of each - two tapes a night.

Unfortunately, there are two key problems with this plan:

1. SDLT tapes are expensive, and finding quality “used” tapes is tough - our last batch of $1,000 tapes averaged a 30% error rate.
2. Restorations are long and complex. Find the tape, read the catalog of the tape, restore the 20kb file. All has to be done by me.

So I’ve suggested the idea to my boss that we should basically duplicate all that storage for quick easy restorations (not to mention a live failover, should our box go boom). Still, the idea of another $6,000 file server isn’t going to fly (and we all know that by now, don’t we?). Which brings me to all of you for opinions.

The way I see it, there are a few options:

We’ve got some old IBM NetVista PCs laying under a desk across from me (1.8ghz isn’t good for much in the way of business desktops these days). Toss in a couple of cheap drives (250gb drives are in the $110 range these days) and we’ve got some very basic (non-RAID) storage for backup purposes. This would probably have the advantage of being the cheapest solution available, if a little boring (oh, it gets better…).

Another option would be one of these actual NAS boxes. I know Buffallo makes a couple of versions (we’d need multiple ones to meet our capacity needs), and Linksys has a DIY NAS product we could throw a 500gb drive in (or 2 if we kept it cool, according to some reviews I’ve ready around).

The last option as I see it is to invest in a *real* backup solution. I know HP in particular makes some SAN / backup combos which should serve both purposes. Too bad we didn’t get one of those from that liquidated company a while back. I’m sure this would be (by far) the most expensive solution, if not the most business-appropriate.

Still, it seems that I have to be missing something. There have got to be some better (possibly more creative) options out there. Ideally I’d love to dump tapes all together, but there really aren’t any better solutions out there (that I see).

One of the problems I’d like everyone to consider whilst pondering my problem is this: We require off-site backup copies. Once a week a set of all our data goes off-site. Once a month, 3 weeks worth of those backups comes back (leaving 1 off-site copy a month permanently - or at least for several years). With tapes, we’re not talking about terribly significant cost here. We’ve already got $4,000 invested in tape media for our rotations, what’s a couple tapes a month in comparison? However, in order to realize my dream of dumping tapes entirely, we’d need to find some way of duplicating this kind of system (or come up with a better, similarly reliable one), if possible. I know a lot of large (read: multi-billion-dollar) companies have huge pipes between their offices and a data center with live fail-over to backups there. For our little (read: $20) company with its weenie T1 connection, this is doubtful to be an option (even if we sent a server somewhere with an initial load of all our data, we’re probably talking a couple gigs of daily changes to pump up over the ‘net every night).

So, to sum it all up, here are the important parts:

1. Backups need to be made nightly.
2. Off-site backups need to be made at least weekly.
3. Restoration should be as painless and quick as possible (assuming a single file is needed, not a hard disaster-recovery restore).
4. At least 1gb of storage will be required for any NAS / similar solutions, with the potential to grow quickly (our current server will hold 24 drives, it’s using 5).
5. A system that allows a user to recover a backup file themselves would be a plus, but not necessary by any means (and shadow-copies only exist in Windows 2003, so don’t ask).

If you’ve got any bright ideas, please let me know. Found something that’s worked really well at work? Found something that didn’t work at all or burst into flames under the strain? I’m looking for all kinds of opinions.

And for those that are still reading, you have no idea how tough it was for me to resist the urge to name this post “Back that Ass Up!”… Sometimes I just crack myself up…