[Guest]

Upload bandwidth is a terrible joke here in Australia, regardless of who your ISP is.

Last time I looked it isn't going to be all that much better under the NBN; well it'll be a lot better, but still not world standard.

[thxbb12]

During my honey moon trip in NZ, our camping car was totally wiped out by thieves, including all of the memory cards which contained 2 month worth of our traveling photos.  No amount of money can replace that.  Extremely frustrating.

 

Nowadays, I use the following setup to minimize the risk of losing any data:

• My PC has 3 TB mirrored (2 x 3TB configured in RAID 1).
  
• I have an external storage SAN using the same config (2 x 3TB in RAID 1; thinking of upgrading to 4 x 3TB in RAID 6 for even better reliability).
  
• I have CrashPlan as online backup service (unlimited space for < $100/year).
  

All this means is that I have 5 copies of my entire hard disk.  I also have a subset of important files on Dropbox (100 GB) to minimize the loss on sensitive data even more  (with a 30 day history capability - great when deleting or updating files by accident).

 

I run external storage backups on a regular basis (every other day or when a lot of stuff changes at once).

The limited upload bandwidth for the online service is not much of a problem as my PC is always on 24/7.

 

With this setup, my risk of actually losing data is close to zero.

[thxbb12]

As mentioned before - just get a RAID drive (Minimum RAID 5, better RAID 1).

A single drive failure will not be an issue anymore then.

RAID 5 can recover a single drive failure although it requires a certain intelligence in the drive controller.

RAID 1 is mirroring - which is simple (simple is always beautiful).

Admittedly I also have just RAID 5 though. :-)

 

RAID 6 addresses the lack of reliability of RAID 5 by allowing up to 2 HD failures out of 4 disks.

 

There are sophisticated solutions such as ROBO (I have one) but honestly I think this is nonsense. Go for a solution that suits your needs for 3 years and then buy a new one (and then migrate all your data).

"Upgradable" drives such as the DROBO just get outdated over the years.

 

The problem with DROBO is that they use their own RAID-like format which is closed. Although it seems all nice and great from their website presentations, if the hardware fails and it's out of production or not repairable, then you might be out of luck: nothing will be able to read your data back since the format is unkown. This is not the case with traditional RAID solutions.

[Guest]

While I use raid 1 (ilnux software raid is quite good); there was a recent research article which strongly suggest that 1 fault tolerance is very poor with regards for overall security. In your specific case you have 2 copies on raid 5 which is better. For folks that don't know ZFS advantage over ext4, fat32 or NTFS is that it provides checksum at the block level (i recommend you use sha-256). for the storage system I've worked on we support arbitrary d+p but generally use 2 copies @ 8+2 (raid5 is 2+1 but the parity is striped across all three disks). To be honest in my home systems I've only had 2 disk failures over the past 20 years and both disks were the same brand and died with in 2 years (worse they were corrupting data before failure - they were samsung f3 1tb drives). As other mention you really should keep an off site copy of the most important data (or have fireproof safe though I question that it can keep the drives/dvd cool enough in the event of serious fire). I don't keep a backup copies of pictures but I do keep a coupld of dvd @ relative house. I used to use tape (around 1998-2001 but gave up as drive density outpaced them. Hum. Wished zfs was better integrated into linux.

Got ZFS Raid Z (that's how level 5 is called with that file system) on one machine and it's automatically mirrored on regular basis over lan to another machine which also has ZFS (but not RAIDed). Both systems have unbuffered ECC memory, partitions are = 4 TB. Never ever had a problem with any of my files.

[thxbb12]

While I use raid 1 (ilnux software raid is quite good); there was a recent research article which strongly suggest that 1 fault tolerance is very poor with regards for overall security. In your specific case you have 2 copies on raid 5 which is better. For folks that don't know ZFS advantage over ext4, fat32 or NTFS is that it provides checksum at the block level (i recommend you use sha-256). for the storage system I've worked on we support arbitrary d+p but generally use 2 copies @ 8+2 (raid5 is 2+1 but the parity is striped across all three disks). To be honest in my home systems I've only had 2 disk failures over the past 20 years and both disks were the same brand and died with in 2 years (worse they were corrupting data before failure - they were samsung f3 1tb drives). As other mention you really should keep an off site copy of the most important data (or have fireproof safe though I question that it can keep the drives/dvd cool enough in the event of serious fire). I don't keep a backup copies of pictures but I do keep a coupld of dvd @ relative house. I used to use tape (around 1998-2001 but gave up as drive density outpaced them. Hum. Wished zfs was better integrated into linux.

 

zfs is indeed a great filesystem (fs), much better than ext4 or ntfs in pretty much every regard. The reason it's not part of the stock Linux kernel is because it is incompatible with Sun's Common Development and Distribution License (CDDL) under which ZFS is distributed.

A very promising fs is btrfs which will definitely replace my ext4 partitions once it's finally ready for production use 

[borisbg]

So no one is using NAS? I was thinking to buy one and use it as back up module.

[borisbg]

Got ZFS Raid Z (that's how level 5 is called with that file system) on one machine and it's automatically mirrored on regular basis over lan to another machine which also has ZFS (but not RAIDed). Both systems have unbuffered ECC memory, partitions are = 4 TB. Never ever had a problem with any of my files.

Hi Doh, can you elaborate on your set up a litle more? I have some knowledge  on the subject, but you guys totally threw me off with all of the acronyms, and especially the postings after yours.

[Klaus]

RAID 6 addresses the lack of reliability of RAID 5 by allowing up to 2 HD failures out of 4 disks.

 

 

The problem with DROBO is that they use their own RAID-like format which is closed. Although it seems all nice and great from their website presentations, if the hardware fails and it's out of production or not repairable, then you might be out of luck: nothing will be able to read your data back since the format is unkown. This is not the case with traditional RAID solutions.

 

Traditional RAID solutions require identical disks (same manufacturer, same size) and that may also be challenging after 2 years ...

[Klaus]

So no one is using NAS? I was thinking to buy one and use it as back up module.

 

NAS is just about attaching it to the network, not about reliability.

Yes, I have a NAS drive. One of my PCs is freezing forever when accessing it ;-)

[Guest]

Kind of off topic but one area where ZFS falls behind is performance (at least on linux). So no it is not uniformly better. Also I'm not sure if ZFS native raid is better than linux software raid (i.e, if you are better running zfs on top of linux software raid or using zfs native raid. The two aspect of ZFS that intrigue me are snapshot (a netapp feature) and block checksum.

-

snapshot is sort of like an automagic copy of a file for a few days - so if you delete a file accidently you can recover it. I'm not faimliiar with zfs implementation but on netapp there is a hidden folder that mirrors your file system. The implementation is quite efficient but it does consume some space. it is sort of like a trashbin but it preserves your directory structure and the retention/deletion is automagic (as well as more efficient).

-

Some terminology. In general raid is D+P (where D is data disk and P is parity disk). You can suffer the loss of up to P disks and still recover all data but the 'work' to recover that data is D. In some scheme the parity disk is striped across all disks; so raid 5 which is 2+1 but a bit of parity data is shared across all disk (this means that the loss of a disk results in more or less equal work to recover and if you loss 2 disks there would still be some original data available. There are some newer scheme in which the P disks are not uniform; they take a bit more space (think of D+P where D is 8 and P is 2 but you need 3 parity disks) but they reduce the work require to recover data. Generally the for values where P == 1 the parity is simple xor and for P > 1 the function is more complex.

-

With regards to Klaus comment; that might be true for hardware raid but for linux software raid (raid is implemented in the software) you can use partition as well as disks for raid volume so replacing disks is fairly easy. There are some newer methods of raid such as LVM and ZFS which carry things to an exterme (they have logical volume management which allows you to grow partitions et all by adding disks at a later time). I've never been a fan of LVM and have not used ZFS. (Ok LVM is not really a file-system it is a volume manager and ZFS has basically merged volume management and filesystem).

-

ZFS is a sun filesystem that was 'open sourced' (before oracle acquisition - now it is close source). There is a port effort to [free,net,...]BSD and linux. The BSD ports are farther along than linux. I'm not sure of the current status of the linux port but there is now a native port with version 0.6.2. For me the two big features of ZFS are block checksum and snapshot (because I'm a bit too eager sometimes with rm -rf .

 

zfs is indeed a great filesystem (fs), much better than ext4 or ntfs in pretty much every regard. The reason it's not part of the stock Linux kernel is because it is incompatible with Sun's Common Development and Distribution License (CDDL) under which ZFS is distributed.

A very promising fs is btrfs which will definitely replace my ext4 partitions once it's finally ready for production use