Ok, here’s the deal… At work we have a primary file server that has just under 1TB of storage in a RAID 5 array (5 x 250gb SATA - parity). The data is fairly evently split in half - one half user data, one half claims archival (remember, we’re in insurance…).
Presently we’re backing up to tape, performing full backups on a 3:2 schedule: Monday, Wednesday and Friday nights user data is backed up, Tuesday and Thursday nights the less-often-modified claims are backed up. Each tape holds 160gb (give or take - PNGs don’t compress all that much, so it’s far from the theoretical 320gb for them - data only partially evens it out since a good bit of it is TARs of database dumps and the like). At this point we’re doing about 230gb of each - two tapes a night.
Unfortunately, there are two key problems with this plan:
- SDLT tapes are expensive, and finding quality “used” tapes is tough - our last batch of $1,000 tapes averaged a 30% error rate.
- Restorations are long and complex. Find the tape, read the catalog of the tape, restore the 20kb file. All has to be done by me.
So I’ve suggested the idea to my boss that we should basically duplicate all that storage for quick easy restorations (not to mention a live failover, should our box go boom). Still, the idea of another $6,000 file server isn’t going to fly (and we all know that by now, don’t we?). Which brings me to all of you for opinions.
The way I see it, there are a few options:
We’ve got some old IBM NetVista PCs laying under a desk across from me (1.8ghz isn’t good for much in the way of business desktops these days). Toss in a couple of cheap drives (250gb drives are in the $110 range these days) and we’ve got some very basic (non-RAID) storage for backup purposes. This would probably have the advantage of being the cheapest solution available, if a little boring (oh, it gets better…).
Another option would be one of these actual NAS boxes. I know Buffallo makes a couple of versions (we’d need multiple ones to meet our capacity needs), and Linksys has a DIY NAS product we could throw a 500gb drive in (or 2 if we kept it cool, according to some reviews I’ve ready around).
The last option as I see it is to invest in a *real* backup solution. I know HP in particular makes some SAN / backup combos which should serve both purposes. Too bad we didn’t get one of those from that liquidated company a while back. I’m sure this would be (by far) the most expensive solution, if not the most business-appropriate.
Still, it seems that I have to be missing something. There have got to be some better (possibly more creative) options out there. Ideally I’d love to dump tapes all together, but there really aren’t any better solutions out there (that I see).
One of the problems I’d like everyone to consider whilst pondering my problem is this: We require off-site backup copies. Once a week a set of all our data goes off-site. Once a month, 3 weeks worth of those backups comes back (leaving 1 off-site copy a month permanently - or at least for several years). With tapes, we’re not talking about terribly significant cost here. We’ve already got $4,000 invested in tape media for our rotations, what’s a couple tapes a month in comparison? However, in order to realize my dream of dumping tapes entirely, we’d need to find some way of duplicating this kind of system (or come up with a better, similarly reliable one), if possible. I know a lot of large (read: multi-billion-dollar) companies have huge pipes between their offices and a data center with live fail-over to backups there. For our little (read: $20) company with its weenie T1 connection, this is doubtful to be an option (even if we sent a server somewhere with an initial load of all our data, we’re probably talking a couple gigs of daily changes to pump up over the ‘net every night).
So, to sum it all up, here are the important parts:
- Backups need to be made nightly.
- Off-site backups need to be made at least weekly.
- Restoration should be as painless and quick as possible (assuming a single file is needed, not a hard disaster-recovery restore).
- At least 1gb of storage will be required for any NAS / similar solutions, with the potential to grow quickly (our current server will hold 24 drives, it’s using 5).
- A system that allows a user to recover a backup file themselves would be a plus, but not necessary by any means (and shadow-copies only exist in Windows 2003, so don’t ask).
If you’ve got any bright ideas, please let me know. Found something that’s worked really well at work? Found something that didn’t work at all or burst into flames under the strain? I’m looking for all kinds of opinions.
And for those that are still reading, you have no idea how tough it was for me to resist the urge to name this post “Back that Ass Up!”… Sometimes I just crack myself up…
You say the $6,000 file server isn’t going to fly, but has anyone done the sums for how much time (read money) it will take to do a resore from scratch if you lose the live system? Not to mention the lower service level that clients will be experiencing due to the staff not having all the info available.
Why are you using 2 tapes a night? Aren’t you doing a GFS (Grandfather, father, son) schedule, and only backing up changes? This also makes it quicker to find that 20kb file.
As for offsite, it you are only backing up the changes, it won’t take too long to shove that over the internet to somewhere each night will it?
Oh yes - forgot to mention “Allison Technologies” can most probably do you a good deal on cheap used tapes…
(Think about it).
I’ve done some rough estimates in my head for downtime. In between flashes of “oh my god, it’d be horrible” that re-surface during my nightmares on a regular basis, I came to the realization that it just wouldn’t happen. I know it sounds crazy (and it probably is), but there’s no way I see that they would agree to spend another $6,000 and have the server sit there “doing nothing” all the time. It’s another one of those mindsets where you’ll have to get burned by it before you take steps to prevent it. No matter how many times I say “you realize how long it would take to restore that if it were lost”, they bring back up the “well if the building burns down, that’s going to be one of the least of our worries” answer. Pointing out that the building doesn’t have to burn down for our file server to go poof doesn’t help, and I’ve long since given up trying. It’s not good, but it’s life in a small company, and I’ve moved along.
Performing full backups every night was a conscious decision on our part to help keep things simple. We’ve got untrained operations people (read: computer morons) swapping tapes and doing other menial day-to-day tasks of that sort. If we were to do a full backup today, then do only differentials from now on, we would eventually get to the point where the tape would fill up and sprout and error and send these idiots into an utter panic when something doesn’t “look right”. I would then instantly have to drop whatever I was doing and assist them and reassure them it was really alright, that we knew this would happen eventually. At which point, they would then proceed to insist in the future that I don’t know what I’m doing and that any future issues are obviously related to this one. In the end, we established it was much easier for everyone involved if we just did full backups every night, particularly to avoid confusion as to which copies should go off-site (with full backups every night, it doesn’t matter - pick one out of the stack).
I suppose online backups would be at least feasible at this point, but with only 150kb/s upload speeds, it just wouldn’t work. Now that I actually run some numbers in my head, it’s even more out of the question than it was before:
Oracle backups - ~4gb x 2
Exchange backup - 14gb (last I checked)
Updates server backup - 10gb (again, last I checked)
There’s more, but we can round off to 30gb for these purposes. At 160kb/s upload, we’re talking about 2 days. Again we’d run into problems doing incrementals of all that stuff (particularly things like Exchange).
So once again we’re back to a local on-site hardware solution… Any other ideas?
Oh, and Tomas, Jeffry and Pah said they were out of stock… Blasted Chief. I think he’s secretly trying to corner the market so he can launch his own brand of MonkeyTapes.
For anyone else wondering what the hell I was just talking about, go read The Case of the Tepid Tipster, one of everyone’s favorite blogs.
Backup to the old PC with a removable hard drive (encrypted backup) and give that to the boss to take home each night.
See how long he keeps it up, especially as he then has to remember to bring the disks in to keep the rotation going… You might just get that new file server.
After that I’m fresh out of ideas at the moment.
Yeah, I think you’ve come to pretty much the same conclusions as I have, and I’ve been mulling this over for about 3 months now. The sad reality of the situation is: there’s no *ideal* solution that would give me everything that I want.
On the plus side, if we had the live backup system going, we could focus the tapes more on off-site backups and monkey with that process some to more accomodate that need (rather than trying to juggle full backups while keeping the latest copy locally, but also having a pretty recent copy off-site and so forth).
At the very least, it would make the “Oh shit, I didn’t mean to click yes to delete that!” scenarios a lot easier to recover from… And the same for the random drag-and-drop “Akk, where’d it go?!” incidents (my god are those common around here… how stupid can you be?).
I hadn’t really explored the idea of the removable drives in the backup PC. We’d have to change our system of keeping one copy off site every month (which really, seems very paranoid to me - 12 backups a year off-site?), but if the idea was to simply keep *a* copy of our recent data (once a week) off-site until it was replaced the next week, we’d really only need a few extra drives to maintain the rotation. A couple of enclosures and a TrueCrypt install later, we’ve got our encrypted removable drives.
Hmm, I shall explore the options further. If anyone else has bright ideas (epiphanies?) please don’t keep it to yourselves…
Hello Mr. Meller(son) {remind me of the Matrix: Anderson}
I am not sure if this idea is viable, but cant you contact your ISP to remove uplink caps to the offsite server. That way, you’ll be able to upload the backups at the maximum speed possible by your operator and since the backup server is in the same town, there wont be any excess international bandwidth charges.
If I get any more revelations, I will let them appear over here
That might indeed be a viable solution, if we had an ISP that gave a shit. Bellsouth is not known to be exactly accomodating (as I’m sure other people who have dealt with them can attest). They would be more than happy to make us upgrade to a T3 line (the most the copper line coming into the building could accomodate - 43mbps?), but somehow I doubt they would give us anything half-way (or even if they could in this particular instance).
Now, if we had that fiber line coming into the building that I’d wanted us to put in, you know, just in case we needed to eventually upgrade to a gigabit link… Then it’s just a matter of caps coming in on the line anyway and in theory they should be able to do that. Of course at that point we’d be spending so much money with them each month they’d probably be more flexible with our requests (demands?). We got a new toy (some cool KVM switches) that we’re in the process of re-wiring into our racks, so I haven’t put much more thought into the backup solution yet, but I think the cheap drives in the crap extra machine is going to be the only viable alternative.
With some luck, it could get expanded into a bunch of removable drives that could be swapped in and out on a regular basis. But we’ll cross that $1,000 bridge when we come to it.
Thanks for the sounding board, guys. Let me know if you get anything else.