22 Replies Latest reply on Jun 23, 2015 1:23 PM by matija1492

    Odd RAW corruption problem on master+backup hard drive

    matija1492

      Hello,

       

      I’m a professional photographer and I have a confusing problem with corrupt photos. Even though this technically probably isn’t Lightroom problem, I could not think of a better place to ask and part of the question is related to Lightroom file handling. The problem coincided with upgrade to Lightroom CC 2015-1, but I can’t tell if it’s in any way connected.

       

      In short, a month after downloading and processing a large amount of RAW CR2 photos, many of them have corrupted. I only learned this yesterday when I wanted to export them.

       

      Corruption looks like this: http://www.justphotos.ca/errors/MEM404067.jpg

      The photo is not mine, but the problem is similar. Part of the photo looks normal, and part is destroyed. Ratio varies.

       

      I performed following diagnostics:

      • CheckDisk from Windows Explorer, including a surface scan - no problem detected
      • inspecting SMART information from the hard drive (via Crystal Disk Info) - the drive checked out as “good”, with no parameter obviously alarming (although I’m no expert)
      • checked RAM with memtest86 - no errors
      • AVG virus and rootkit scan - nothing found
      • Explorer and Lightroom can normally manipulate the files (copy, read, even develop), it’s just that they’re corrupt

       

      Most important question first: does anyone know what can cause this type of corruption and avoid detection by all of the above mentioned tests? What to do about it?

      Presumably, if there was an issue with hard disk SATA cable or motherboard

       

      — — — — — — — — — — — —

       

      Here’s a bit more background where things get weirder and where someone with LR knowledge might come in handy.

       

      The disk in question is a WD 4TB with tens of thousands of photos. I checked a lot of folders randomly and so far found corruption in only two, chronologically one after the other (which may mean they are adjacent on the hard drive, but it also may not if it was significantly fragmented). There may be others, viewing RAWs after deleti

       

      Photos were taken a month ago. I copied them manually to a master drive (internal) and to a backup drive (USB external, I only connect it for backups). I viewed all photos in Lightroom and post processed a few hundreds. They were all fine. After that they justsat there for a month with not activity and no viewing (ie. I can’t know exactly when they went corrupt).

       

      After postprocessing, I used a software for synchronizing folders from master to backup drive. It copies new files, existing files which are newer or which have a differing file size.

      Next question is crucial, so please help me shed some light on it. Does Lightroom IN ANY WAY modify CR2 files themselves? I know it creates XMP sidecars. But will it under any circumstances ever modify the master RAW file?

       

      This is important because I know for a fact that original flawless RAW files were manually copied to backup disk. The synchronizing software would not copy corrupt files from the master drive unless they were changed somehow.

      As far as I know, regular disk corruption is invisible to the OS or filesystem - when it occurs, neither file size nor date of modification are modified - only the original physical data on the hard drive plate.

      It may be of academic importance right now, but I need to know to possibly modify my backup solutions for the future.

       

      Thanks for any suggestions!

        • 1. Re: Odd RAW corruption problem on master+backup hard drive
          dj_paige Level 9

          This is caused by a hardware malfunction.

           

          It could be the camera card, card reader or USB cable/port, hard disk or memory. If you have tried the diagnostics mentioned above, then most likely it is a bad camera card, or card reader/USB Cable/USB port

          • 2. Re: Odd RAW corruption problem on master+backup hard drive
            matija1492 Level 1

            Thanks for your answer, but I tried to make it clear above: the photos were transferred to hard disk and were fine for at least a few weeks. So it absolutely can't be camera, card, card reader or USB cable.

             

            After copying them from the cards, I went through each and every photo and post-processed some of them. They were in 100% perfect condition. Additionally, after processing them, I temporarily copied them to a different computer where they also were in 100% perfect condition.

             

            Whatever happened, happened after a few weeks and somehow affected both master and backup drives.

            • 3. Re: Odd RAW corruption problem on master+backup hard drive
              dj_paige Level 9

              Since I have been through these debates in the past, let me tell you that regardless of your situation that you described, this is always a hardware issue. ALWAYS. As in every single time. That's all I can tell you. It's up to you to figure out what the actual root cause is.

              • 4. Re: Odd RAW corruption problem on master+backup hard drive
                matija1492 Level 1

                I realize that, but since it's not usual "disk errors and corruption" issue I wanted advice from people who may have experienced something similar in the past. I know it's up to me to figure out the cause, and this is exactly what I'm doing by asking for ideas and advice.

                 

                Also, question pertaining to Lightroom and possible CR2 modification is very important to trace back the spread of corruption from master to backup.

                • 5. Re: Odd RAW corruption problem on master+backup hard drive
                  trshaner Adobe Community Professional & MVP

                  matija1492 wrote:

                  Next question is crucial, so please help me shed some light on it. Does Lightroom IN ANY WAY modify CR2 files themselves? I know it creates XMP sidecars. But will it under any circumstances ever modify the master RAW file?

                  No. You can check this by looking at the files in question with Windows Explorer. Compare the 'Date' column to the 'Date Modifed' column. They should be the same. If selected in the Catalog Settings LR uses and writes to XMP sidecar files.

                  As suggested this is most likely caused by a faulty memory card, card reader, or cable. The fact that the corrupted files appear in the same close date range may help to determine the cause. Did you do something different when transferring these files, such as use a different cable or card reader?

                   

                  To prevent this in in future make sure you build 'Standard' or '1:1' Previews on import and then review all of the image files inside LR for corruption. Until previews are built LR uses the small thumbnail image embedded in the raw file, which in most cases is unaffected by file corruption. The file will look OK initially until LR builds an actual preview from the raw data.

                  • 6. Re: Odd RAW corruption problem on master+backup hard drive
                    ssprengel Adobe Community Professional & MVP

                    There is one circumstance where LR can modify the raw file contents, but it is the metadata not the raw image data, itself, like your corruption seems to indicate.  In my experience this writing occurs during import or perhaps if you change the capture time.  Some cameras don't add a timezone to their internal dates and LR fixes this so that you can sort things by date-time later and have multiple camera bodies interleave their photos properly.

                     

                    The reason this setting was added was LR always used to do this and with one LR release LR actually corrupted all the raw files due to a bug, so LR hurried out a fix and added this setting so people could turn off the auto-fix-timestamps operation that LR had been doing.

                     

                    Since your corruption occurred some weeks into your editing, it is likely the hard disk that is failing or if it is an external drive, something wrong with the connection to that.

                    • 7. Re: Odd RAW corruption problem on master+backup hard drive
                      matija1492 Level 1

                      Thanks for this information.

                       

                      The issue here is twofold. One is how the files got corrupted on a hard drive where surface scan nor SMART parameters indicate no issues exist. But the second issue is even more pressing, and that is how the corruption spread to the backup drive?

                       

                      If the lightroom indeed modified RAW files themselves at some point after I was done processing them, it might explain why the synchronization software copied files again, this time corrupt files over valid ones. (if they weren't changed, they wouldn't get copied again)

                      In this scenario it doesn't matter whether LR modified embedded metadata or even just changed the modified date without doing anything to the file itself -- in both cases the sync software would copy the files again.

                      • 8. Re: Odd RAW corruption problem on master+backup hard drive
                        trshaner Adobe Community Professional & MVP

                        Two questions:

                         

                        1) Are you sure you edited the corrupted image files? Is it possible these files were corrupted on import, but were not edited and did not have previews built?

                         

                        2) Did you check the system ''Date'  and 'Date Modified' to see if they are different?

                         

                        • inspecting SMART information from the hard drive (via Crystal Disk Info) - the drive checked out as “good”, with no parameter obviously alarming (although I’m no expert)

                         

                        Can you post a screenshot of the Crystal Disk info for the system drive the image files are stored on?

                        • 9. Re: Odd RAW corruption problem on master+backup hard drive
                          ssprengel Adobe Community Professional & MVP

                          Both Dates and File Size can be used by a backup program to determine if a file has changed.  Maybe is was the directory information that was corrupted and the file size or dates changed, there.

                           

                          Obviously after-import corruption/deletion is one reason to make a backup copy during the import process that doesn't get touched, again, so changes are not replicated if something goes wrong as it seems to have in this case.

                          • 10. Re: Odd RAW corruption problem on master+backup hard drive
                            matija1492 Level 1

                            1) It is 100% certain that files were not corrupt on import. I process each and every wedding in a systematic way as follows:

                            1. copy to master and backup hard drives
                            2. build 1:1 previews for all photos
                            3. view and rate each image
                            4. process rated images

                             

                            So I looked at these images in 5-6 separate occasions, from built thumbnails, to culling (2 passes) to editing. Both edited and non edited files got corrupt.

                             

                            2) The "created" and "date modified" fields are identical. They are the times when the image was taken (camera time), not when it was copied to hard drive.

                            modifiedcreated.JPG

                             

                            Here's crystal disk info:

                            diskinfo.JPG

                            • 11. Re: Odd RAW corruption problem on master+backup hard drive
                              matija1492 Level 1

                              Regarding your previous post, LR option to write data into proprietary files is turned off. Date created/modified info would also point to this.

                               

                              As far as backup goes, in principle I still can't find a logical fault with my method of syncing modified or newer files since nothing modifies CR2 -- copying XMPs manually each time I change something would take ages. But I guess in practice something did go awry. I just hope I find out what.

                              Regarding corrupted directory information, I would think that checkdisk would find at least one fault with the filesystem, which it didn't.

                               

                              It's driving me crazy and I haven't even gotten to salvaging what can be salvaged (it's a wedding that hasn't been exported or delivered yet).

                              • 12. Re: Odd RAW corruption problem on master+backup hard drive
                                ssprengel Adobe Community Professional & MVP

                                I suppose it's too late to recover originals off the memory cards, right?

                                • 13. Re: Odd RAW corruption problem on master+backup hard drive
                                  ssprengel Adobe Community Professional & MVP

                                  The other thing you could do is extract the camera-embedded preview out of the raw file using something like EXIFToolGUI:

                                  Latest ExiftoolGUI version 5.16

                                  • 14. Re: Odd RAW corruption problem on master+backup hard drive
                                    trshaner Adobe Community Professional & MVP

                                    matija1492 wrote:

                                     

                                    Here's crystal disk info:

                                    I have the the 2TB version of the same drive (WD Black) with almost identical numbers and I've experienced zero issues to date.

                                     

                                    matija1492 wrote:

                                    2) The "created" and "date modified" fields are identical. They are the times when the image was taken (camera time), not when it was copied to hard drive.

                                    So unlikely the files were over-written to the backup drive.

                                    • 15. Re: Odd RAW corruption problem on master+backup hard drive
                                      trshaner Adobe Community Professional & MVP

                                      What backup software and drive are you using?

                                      • 16. Re: Odd RAW corruption problem on master+backup hard drive
                                        matija1492 Level 1

                                        Yes, it's too late for recovery off original cards, they were overwritten many times. Recovering from embedded previews is my only chance, I tried it on two sample images and it worked, so hopefully it will work for the rest as well. With all the diagnostics I just can't find the time to get down to it.

                                        • 17. Re: Odd RAW corruption problem on master+backup hard drive
                                          matija1492 Level 1

                                          The dates are identical on the master drive, but not on the original drive. I traced back the last copying of the files to 12 days ago. It's hard to say what triggered the copying over the "old originals" again, but my only current theory is that I might have renamed the original folder. I do this rarely and I usually rename the folders manually on both drives (so no re-copying occurs), but I can't categorically exclude this option. And I honestly can't think of anything else.

                                           

                                          I am using Windows Explorer for initial backup (when the files are fresh off the memory cards) and later Karen's Replicator. It basically just compares master and backup folder with a set of rules. In my case it replicates deletions and looks for new files and modified files (by date and file size).

                                           

                                          So if we suppose that we figured out how the corrupt copies were spread from master to backup, the final thing that remains is how they got corrupt in the first place. Can anyone recommend software which would stress-test the disk and detect possible physical faults with the plates, heads or electronics? I presume the problem should be with the plates, since files were not modified at any time so other parts of the chain could corrupt the data stream through reading and re-writing. But if Windows CheckDisk couldn't detect even one issue with the surface scan, how can the plates be the culprits?

                                          • 18. Re: Odd RAW corruption problem on master+backup hard drive
                                            trshaner Adobe Community Professional & MVP

                                            matija1492 wrote:

                                            I am using Windows Explorer for initial backup (when the files are fresh off the memory cards) and later Karen's Replicator. It basically just compares master and backup folder with a set of rules. In my case it replicates deletions and looks for new files and modified files (by date and file size).

                                            I can highly recommend FreeFileSync as it provides much more capability and reporting than available in Karen's Replicator. After doing the initial Compare of Source to Backup drives it provides a detailed report of files to be added, removed, or updated. You can easily see files and folders that will be deleted or updated during the backup Sync operation and do a quick sanity check.

                                             

                                            matija1492 wrote:

                                            Can anyone recommend software which would stress-test the disk and detect possible physical faults with the plates, heads or electronics? I presume the problem should be with the plates, since files were not modified at any time so other parts of the chain could corrupt the data stream through reading and re-writing. But if Windows CheckDisk couldn't detect even one issue with the surface scan, how can the plates be the culprits?

                                            All drives use CRC data checking during Read & Write to confirm data integrity. Also with platter degradation data errors would show in one or more of the SMART information readouts. Your drive has none. It's highly possible that you experienced an intermittent system error due to overload and/or low AC mains voltage, over-heating, etc. You can download the free PassMark DiskCheckup SMART utility and run the extended self test.

                                             

                                            You can use the free HW Monitor to check basic system temperatures, voltages, and fan speeds. If you ran Memtest86 only and not MemTest86+ you may want to run it. Unfortunately, these memory testers all run in standalone mode without the operating system loaded. This may sound like a good idea so that you can access and test ALL memory locations, but it may not reveal memory issues that only occur with the OS and applications loaded and running in memory.

                                             

                                            If you're not experiencing any other system problems or errors and only the two folders of imported image files are affected it may have been a fluke-event. The best suggestion I can provide to help prevent this from happening again is to NOT move or rename files or folders after they are imported into LR. Use LR's Rating, Keywording, and Collection tools and the Filter Bar to organize and locate your image files....not folders. A UPS is also good protection against even infrequent power-loss or brown-outs.

                                            • 19. Re: Odd RAW corruption problem on master+backup hard drive
                                              trshaner Adobe Community Professional & MVP

                                              matija1492 wrote:

                                              Recovering from embedded previews is my only chance, I tried it on two sample images and it worked, so hopefully it will work for the rest as well.

                                              Jeffrey Friedl's Blog » Jeffrey’s “Extract Cached Image Previews” Lightroom Plugin

                                              • 20. Re: Odd RAW corruption problem on master+backup hard drive
                                                matija1492 Level 1

                                                Thanks for the wealth of information!

                                                 

                                                Regarding file syncing, I decided to get a Drobo 5D which should avoid flukes like these in the future and avoid synchronizing folders altogether. It has a small battery which should help in case of black-outs. UPS is a good idea, but one thing at a time

                                                 

                                                A friend suggested manufacturer's tool for testing the drive (WDDiag) which I will leave overnight to check the disk again. It's results are valid for warranty purposes with WD so I guess it can't hurt.

                                                I really am not experiencing any other system issues whatsoever, so it might be a fluke event. Although that sounds scary - with all the precautions taken and regular backups, that a fluke becomes my undoing. It's kind of ironic that running a regular backup was the final nail in the coffin.

                                                 

                                                The tool you suggested would be very useful if the 1:1 previews hadn't expired after 30 days. I did rebuild them to evaluate the damage from thumbnails, but they're useless now. I guess the embedded previews are my last resort now.

                                                 

                                                Anyhow, back to damage control and extracting previews. Thanks again!

                                                • 21. Re: Odd RAW corruption problem on master+backup hard drive
                                                  ssprengel Adobe Community Professional & MVP

                                                  If you extract the camera-embedded preview JPGs, unless ExifToolGUI already does this, you'll want to update the extracted jpgs with the EXIF from the raw files:

                                                   

                                                  http://ninedegreesbelow.com/photography/exiftool-commands.html#raw-to-jpeg

                                                   

                                                  exiftool -tagsfromfile %d%-.8f.cr2 -all:all -r -overwrite_original -ext jpg /media/images/digikam/rawpeg