Geek question: best solution for storage and backup of data?

[johnnyb]

Hi All,

I currently have about 5GB of important stuff (data, photos,etc) on our home computer. I have been using the Mozy backup service for a year or so now, and it seems to be working fine (although I'll really find out if I have to do a restore).

I'm about to transfer all of our digital videos to my computer for editing, as well as long term storage (I don't know what the lifespan of digital tapes is). I want to keep them in the dv-avi format on my computer, which as far as I can tell is the least compressed option and this will give me the most flexibility for editing and changing to different formats in the future. The problem is that this format requires about 200MB per minute, so my 20 one hour tapes converts to about 240GB. I expect this to increase as we film more stuff.

I only have about 40GB spare at the moment, so I'm looking at buying an external hard drive, probably with 500GB capacity (like a WD My Book). What I'm trying to work out is the best backup strategy for this amount of data. The two options I see are:

1. Use Mozy for the video files - as Mozy has a flat rate it won't cost me any more to do this, but the time required to upload all this to the Mozy server will be weeks if not months (fortunately my ISP does not charge for upload usage, I think). And if I ever have to do a restore it will take a similarly long time.

2. Spend some more money on the external hard drive and get one with RAID 1 capability. This mostly protects against hard drive failure, but doesn't protect against disaster like fire.

Does anyone have any suggestions or recommendations on which way I should go, or any alternative suggestions? Thanks.

John.

 

[Psyk]

Really depends on how important your data is.

Many people just burn copies to DVD and keep multiple copies of it.

However if the data is that important (i.e. what would it cost you if you lost it), you could look at a tape unit via USB and run backups to it. The larger data capacity of tape means you can hold multiple versions of your data should you need to restore back to an unedited/unaltered version should you need to.

So an external disk will give you manageable backups which are easy to restore from.

Tapes will give you longer term archival and you can send a copy somewhere else for safe keeping if you're worried about the 'fire' taking your place out.

 

[BKR]

When we were backing up about 30Gb of photos,etc I just bought a second hard drive, plugged it into the PC and set it as the slave, then did a copy from 1st hard drive to the second. Its very fast and when finished you can unplug and store the HD so no viruses, failures. To burn DVDs or use Usb ports will be miles slower then a straight HD to HD copy.
You can get a 500GB for about $200-250 at places like The Disc Shop or wherever.
http://www.thediscshop.com.au/index...ategory_id=32&option=com_virtuemart&Itemid=94

Chris

 

[redsquash]

Buying a hard drive for storage only is a waste of technology in m view and very expensive for $/gb.
What happens if it fails.

Copy to DVD while watching a movie on TV to avoid boredom as

I do or
invest in Blue ray technology. Quite expensive at the moment

 

[ggumpshots]

My understanding is that if you delete a file on your computer Mozy will delete the backup file after 30 days and it wont be recoverable.
Is that correct?

 

[GordElner]

I can't see that 30 day delete being part of any backup system. The file obliviously would not be backed up after you delete it, but the very concept of backup would be violated by a practice such as the one you mention.

 

[johnnyb]

Thanks for the suggestions everyone. I've not considered the DVD backup solution as I didn't want to deal with all those discs - I'm lazy, and I was hoping modern technology would come to my rescue. But I should probably consider it again. I'll also have a look at a tape backup solution.

redsquash, I'll need an external hard drive to get the videos off the original tapes, so I'm not using it just for storage. My real question is how do I back up this hard drive, ie your question about what happens if it fails is exactly what I'm trying to answer.

Regarding the 30 day thing with mozy, yes, my understanding is that they delete files after some time if you delete them locally. Apparently it's a bit of a controversial topic, but my view is that backup is different from archiving. For backup I want a copy of my current stuff to restore from. What you're talking about GordElner is what I would consider archiving, which may or may not be combined with a backup mechanism. Mozy does a pure backup (by my definition). The 30 day thing is just a bit of security in case you accidentally delete a file and don't realise it for a few days.

John.

 

[Psyk]

Good points johnnyb. Backup and Archival although similar are typically treated differently because of the different types of data. For example, my Canon EOS takes very large JPG's (I don't use RAW) but then I compress them down for lesser quality which I can share on a web page or e-mail to family. Yet I still keep the original's... So I have the original plus multiple copies which I use... and I keep all for convenience.

Thanks for the suggestions everyone. I've not considered the DVD backup solution as I didn't want to deal with all those discs - I'm lazy, and I was hoping modern technology would come to my rescue. But I should probably consider it again. I'll also have a look at a tape backup solution.

If you're talking modern technology, then I can tell you enterprise businesses and companies use combined solution of automated backups using disk and tape and many are adopting a hybrid system of disks that look like tapes for their environment.

Disks because they're easy to restore from. Also, if a system loses it's data then they can revert to other disk and pretty much continue operation as per normal.

Tapes are excellent for $ per gig, so they can store them offsite very cheaply. Enterprise class tape drives are hugely faster than the systems that can sustain the data flow to them, and I suspect that you'll see the same with a USB tape drive vs a DVD/Bluray burner.

They hybrid systems gives them the benefit of disks, but the convenience of tape i.e. it drops straight into their current business data protection processes and existing backup software with very little process and change.

redsquash, I'll need an external hard drive to get the videos off the original tapes, so I'm not using it just for storage. My real question is how do I back up this hard drive, ie your question about what happens if it fails is exactly what I'm trying to answer.

Probably the most practical method would be tape because you're dealing with very large amounts of data. Tape backup units can be aroun $700+ dollars (e.g. a USB DAT tape drive), but the advantage is the convenience and speed of data backups, the resiliency of tape and the cost per gig to store it. I would trust terabytes of tapes sitting in my desk drawer or shelf vs. terabytes of USB disk drives over time.

The only downside is restore times for large amounts of data, which is why you would probably use USB drives for more current/active data, then archive off to tape when done (or when accessed less).

Regarding the 30 day thing with mozy, yes, my understanding is that they delete files after some time if you delete them locally. Apparently it's a bit of a controversial topic, but my view is that backup is different from archiving. For backup I want a copy of my current stuff to restore from. What you're talking about GordElner is what I would consider archiving, which may or may not be combined with a backup mechanism. Mozy does a pure backup (by my definition). The 30 day thing is just a bit of security in case you accidentally delete a file and don't realise it for a few days.
John.

Sounds like remote-mirroring (or replication to me), and they're useful too as it tries to maintain currency of data across multiple sites - which may or may not be a good thing (i.e. errors, accidental deletions are also replicated/mirrored).

One solution is not better than any other and ultimately depends on your use cases.

What you'll want too look at also is the backup application, something robust and easy to use. In 10 years time when you want to access those photo's or videos you'll need to ensure that the data written to tapes, replicated to other site, or written to USB drives can be read by whatever application put it there.

Typically for single-pc environments the built-in operating system one such as Windows backups can help ensure compatibility, but it's limited to types of devices that it may or may not support and flexibility of backups etc.

 

[GreatPig]

my 20 one hour tapes converts to about 240GB

Forget ordinary DVDs for that sort of volume. Even with DVD-9, you're talking something like 27 DVDs with hours of burning time.

I'm not real familiar with tape backup, although that's what they use at work for terabytes of data, but I'd probably go for just using extra hard disks myself. 500GB USB drives are pretty cheap now, and one terabyte ones not all that expensive either. Just keep two copies of the data on two separate drives. Tapes would be better for high reliability backup though, as it's much cheaper to keep multiple copies and to use rotational backups.

I store quite a few drive images and virtual machines, as well as music and photographs, and use USB disks for that - originally 500GB but I recently changed to 1TB when that wasn't enough. I only use one such disk, because if it fails it can all be recovered from the original machines or other backups (I also keep the music and photos on a server at home and another at work). The only time I would be stuffed is if the USB drive and original machine (for images and virtual machines) both failed at the same time. For a couple of important work machines, they're also backed up on the office tapes, and for the other machines, important data is backed up in multiple places so the worst case would be having to reinstall a machine from scratch and then recover the data from another backup. In the end, how important you consider the data to be determines how many backup copies you think you should have.

One thing to be wary of with an extra internal hard disk, other than the constant insertion and removal if you don't want to keep it always in there, is that if a machine crashes, it can take out both drives at the same time. I just recently had that happen to a machine at work. It had two hard disks in it (with separate data - not backups of each other) and it died. By "died", I mean it kept running but became unresponsive, and when I rebooted, kept reporting an error reading the boot drive. Subsequent testing of the drives in another machine revealed that the primary partitions on both drives were toast, although the drives themselves were still working and restoring the partitions from an image solved the problem (at least for now).

GP