[popo]

Seeing some friends win or at least shortlisted in various photo competitions I thought I'd stop being lazy and start entering some myself.

To start with I picked a place I went to earlier this year, who runs a competition for photos taken either on their land or of them if elsewhere. So far so good? I was browsing the photos in DxO when I came across a batch of files indicated as "loading error". What does that mean? Sure enough, every program I had wouldn't recognise them. In part I never noticed this earlier, as my normal browser by default hides files it doesn't recognise, masking this possible problem.

No problem, I started up my backup server and looked at the same files on there. Normal. So why was the copy on my main desktop not openable?

I downloaded a hex editor so I can view the contents, and immediately saw the problem. The corrupt files were all zeros.

So in a moment I'll manually copy the good files back over the bad ones, but it does scare me that my new but almost full 6TB drive may have other bad files on it. This drive was bought only a couple months ago as a live copy bulk storage of all my historic data so far, moved on here from my previous live copy on 4 separate drives. That is separate again from my backup copy on different 4 drives.

Possibilities are the files were somehow corrupted before/during/after moving from old live drive to new live drive. I don't think I have any way to tell.

I suppose I could do a full scan of the 6TB drive to ensure there's no other errors with it. SMART data looks ok.

Now I have doubt on the integrity of the whole drive, I have few possibilities:
1, somehow scan for raw/jpeg files that are corrupt. If any exist, I may have a problem
2, do a binary compare of the enter drive against the backup. This will take forever!
3, assume the backup is correct, and overwrite everything on the live copy.

Thinking more, I think option 2 is the safe way to go. Does anyone know of software that can do that?

[JJ_SO]

6TB on one single HD? My first thought: gigantic! My second: And what if bit rot happens to all those files? And my third thought: That's about 4× the amount of data I have. Do you collect everything?

 

As for the software: Mac or Win? On Mac I use Data Rescue to scan a drive or get back randomly erased files.

 

Comparing software? Are you sure the host system is able to write clean data to the drive?

[popo]

Yes, 6TB on a single disk. You can even get 8TB if you want, but that meant buying a Seagate. No thanks.

That replaces 2x3TB + 2x2TB, which weren't full of course. It is not just photos, but pretty much my whole digital life.

Bit rot is a potential problem, but there are no easy solutions other than possibly ZFS mirroring and doubling up even more on hard disks... on an unrelated note, I'm wondering if it is time for me to move to server grade hardware. At least ECC ram, which is present on both my file servers but I hadn't bothered with on desktops.

I'm a Windows user. Note the files weren't missing or lost, but the content was all zeros. e.g. the file indicated a size of say 22MB, all of it digital zero. This is not the same as having deleted a file.

Anyway, I have randomly looked through some more file folders, and haven't seen anything unusual in any of the others so far. By why would 36 consecutive raw files be zeroed in this way?

[JJ_SO]

Before you start looking for another backup-strategy, I think it's better to examine why only 36 pictures (so far) got that loading error. I would consider a bad SD card first. If you discover it, trash it. Otherwise it's only a question of time before you loose important pics.

 

Is always the same card in your camera or do you alter them or change it between different bodies? Find out the source of those zeros would be the most important thing to me.

[Klaus]

If my connection was fast enough I would use Amazon Drive ...

https://www.amazon.com/clouddrive/pricin...navpricing

 

60$ per year.

[Brightcolours]

Problem with cloud services: They can just stop to exist. Especially when from Google.

[popo]

Before you start looking for another backup-strategy, I think it's better to examine why only 36 pictures (so far) got that loading error. I would consider a bad SD card first. If you discover it, trash it. Otherwise it's only a question of time before you loose important pics.

As mentioned earlier, the copy on my backup server was fine and I've done a manual restore of them. It was only the files on my live copy that were corrupt in this weird way. So the problem is either on a hard disk, or on file move, or some other filesystem corruption.

Since the files were originally copied to the hard disk, they have been moved to a different hard disk in my new system. I don't think there is any realistic way for me to tell if it got corrupted on the original hard disk, during the transfer to new hard disk, or since that transfer.

As a precautionary measure, I'm in the process of doing a full surface scan of the 6TB disk. Started it last night. Been running 8 hours with an estimated 2 hours to go...

Maybe when Skylake Xeons come out I'll switch to that with ECC ram, and perhaps create a mirrored ZFS local file store connected over 10G ethernet. Why can't I have lower cost interests?

I guess I am pushing things a bit being an early adopter of Skylake. When I built my current system, there was a bios update. Ok, applied no problem. It has been perhaps 6 weeks since then, and the manufacturer has released 4 more bios updates, two of which simply say "improve system stability" in the description. Scary... I put that on last night too. I've had occasional weird non-boots previously but if it boots it remains up fine (usually left on with CPU intensive processes in background).

[Klaus]

Problem with cloud services: They can just stop to exist. Especially when from Google.

 

We are talking about storage here. Amazon cloud disk is probably based on S3 which is probably the most used storage system on mother earth. I would have zero concerns storing my images here. They are providing backup and full life-cycle-management. At home you got aging disks and the risk that the disks get stolen, destroyed or simply die. If you got the bandwidth it's a no-brainer really.

 

I am not aware that Google has deprecated any core cloud service. Something like Google Wave or Reader wasn't really core.

[JJ_SO]

I prefer having my files at home. I hate being depending of any service which can fail, can become unavailable and especially hosts my files in the US. Besides, data traffic in this size costs a lot of  unneccessary energy, it's just an environmental thing as well. Not much for a single photog, but if it gets more used, I'm only declare it as "other people's problem".

 

No way for me.

[popo]

Hmm... cloud storage prices have come down a lot since the last time I looked. Unlimited for $60/y sounds good to me, but as Klaus says, bandwidth will be the limiting factor. My ISP provides a highly asymmetric network, which shows in their provided connections. I get 50M down, 3M up, and the upload is throttled in peak times. Allowing for overheads, I think my uploads work out roughly equivalent to 1 Gbyte/hour. As long as your data production rate averages below that, it could work... the only pain point is doing the initial upload in the first place! I think I saw previously Amazon had another option where you can mail in a hard disk to manually load.

 

The only paranoia thing is I'd probably want to encrypt everything before uploading which may need manual intervention. This backup copy would only really be accessed as disaster recovery, not routine access.