16 Replies Latest reply: Dec 3, 2013 2:23 AM by trshaner RSS

    File corruption after import

    drtom Community Member

      I use and love Lightroom. Currently on  4.x

      Here is the problem. Suddenly many of my images are being corrupted. They seem to acquire a colorful striped pattern. Sometimes it happens on import, sometimes not for a few weeks. You will be looking at the filmstrip, and actually watch them be redrawn with the error. When I import the same raw file into another program (FastStone), I am able to see it properly and export it to jpeg. With LR, the export maintains the corruption. I have tried it on two different hard drives with the same results. I don't think its a hardware issue. I am attaching two sample images. One processed with FastStone, the other with LR. I had to use paint application to shrink them to an uploadable size, but they do represent the problem correctly._MG_9285fastStone.jpg_MG_9285lr.jpg

        • 1. Re: File corruption after import
          dj_paige Community Member

          This is almost always a hardware problem. If it happens weeks after the photo was imported into Lightroom, then there is a good chance that your hard disk is experiencing problems. I suggest you run some diagnostics on it.

           

          Also, make sure you have backups of these photos and backups of your catalog file on another physical disk (not a different partition of the same disk); consider purchasing a new disk.

          • 2. Re: File corruption after import
            drtom Community Member

            Thank you for your input. I have two separate hard drives on my system. The OS runs on a SSD, then I have the main files on the HDD. I have tested the raw images on both drives with the same results. Also, when I use another program on the same drive, the problem does not occur. Wouldn't that suggest a problem with LR rather than the drive?

            • 3. Re: File corruption after import
              dj_paige Community Member

              Lightroom reads the photo data from the hard drive. Lightroom does not modify the photo portion of your photo files on your hard drive. So, if the photo data shows up corrupted, then it is the hard drive that is sending corrupted data to Lightroom.

               

              (Of course, it is also possible that the data arrived on the hard drive from the camera card corrupted, but since you said the corruption sometimes happens weeks after import, its hard to claim that the data arrived at the hard drive corrupted)

              • 4. Re: File corruption after import
                drtom Community Member

                I am in the process of scanning my hard drives now. I will report back.

                • 5. Re: File corruption after import
                  Geoff the kiwi CommunityMVP

                  What Fastone uses is the jpeg preview in the file not the raw data hence the clean look. Lightroom reads the preview first then the raw data.

                  It could well be that this occurred during d/load as well. How did you d/load??

                  • 6. Re: File corruption after import
                    DdeGannes Community Member

                    I would also check your RAM at the same time. RAM modules can also cause these problems. I have had to change "flacky" ram on my computer twice.

                    • 7. Re: File corruption after import
                      drtom Community Member

                      I have completed the scans on both drives. They both check out fine, no problems found. I also did a mem check of the RAM, also fine, no problems.

                      • 8. Re: File corruption after import
                        areohbee Community Member

                        Are you 100% certain the files were not corrupted upon transfer? I mean, if disks are really OK, and ram, then usually this problem occurred upon transfer, e.g. bad card reader (or card), or cable, or usb port, ...

                         

                        Reminder: viewing jpeg preview, e.g. Fastone, doesn't count (has no bearing on whether raw is corrupt).

                         

                        What is the filename extension of corrupt files?

                         

                        Rob

                        • 9. Re: File corruption after import
                          areohbee Community Member

                          dj_paige wrote:

                           

                          Lightroom reads the photo data from the hard drive.

                          More specifically, the OS reads it from the drive, and Lightroom reads it from RAM - thus bad ram can also be a culprit...

                           

                          Put another way: ram is the buffer between hard drive and Lightroom, so ram failure can mean bad data in Lightroom even if hard drive is fine.

                          • 10. Re: File corruption after import
                            drtom Community Member

                            Hi All,

                             

                             

                            I have done some more testing.

                            I used the same card reader to transfer multiple "good" and "bad" files to the computer. Then I imported them into LR. Same results. The files that were good previously remained good, the "bad" files remained bad.

                             

                            I repeated this many times. Same results.

                             

                            Next I hooked up the camera to the computer directly with the factory USB cord. All images transferred and imported into LR were fine.

                             

                            Repeated with card reader. Got same errors. Then on subsequent tests with card reader, all images appear fine in LR.

                             

                            So, is the culprit the card reader?  I am guessing that LR, the Hard drive, Ram, and the Camera and SD card are all fine. Not sure if there is anything left to test. Wish I hadn't permnanently deleted some of the "bad" images before more testing. Lesson learned.

                            • 11. Re: File corruption after import
                              trshaner Community Member

                              As has already been mentioned LR uses the camera file's embedded preview for display until  Standard or 1:1 Previews are built. This would explain what you saw, "You will be looking at the filmstrip, and actually watch them be redrawn with the error." The initial image is of the smaller embedded preview. As the larger Standard or 1:1 previews are built using raw data the corrupted images are revealed.

                               

                              SUGGESTION

                              When importing images select 'Build Previews' as Standard or 1:1. Leave the images on the memory card until 1) Preview Building has completed in the progress bar and 2) you've reviewed each image file at least once in the Library module's Loupe view mode. I would also suggest keeping the images on the memory card until you've done initial editing on all of the images. This will require using additional memory cards, which I explained in the below post is very cheap insurance:

                               

                              Would you spend 16 cents for “temporary backup” of 1,000 perhaps irreplaceable images?

                               

                              Try your card reader again using a new memory card before condemning it and also try plugging it into a different port. The culprit may actually be a failing memory card (or USB port), which simply works better using you camera's memory card controller. This will help you determine the defective component(s).

                              • 12. Re: File corruption after import
                                drtom Community Member

                                Thanks for the troubleshooting and great advice.

                                • 13. Re: File corruption after import
                                  batjake Community Member

                                  We initially had trouble with the 7D. A few weeks ago - the 5Dii is having the same issue.  We have started shooting RAW+Jpg as a backup.  Glad we did.  The two 30D's haven't had any trouble at all.  Pretty much the last thing I want to see is a beautiful photo that my wife took of a bride with a big pink block going through the side of her head. 

                                   

                                  We're using high speed, high quality cards.  Bought new ones when the trouble started but got the same results (both Lexar and Sandisk).  We use an Extreme USB 2.0 CF reader.  My guess is that the reader is the culprit (by the way, nobody seems to have a real answer - just guesses.  It's annoying ).  I've recovered the same corrupt, deleted files using the software that came with one of the cards.  The ones that were giving me trouble turned out fine.  Also no issues so far if I copy/paste with the camera actually hooked up to the computer. 

                                   

                                  Planning on upgrading to the Lexar Professional USB 3.0 Dual-Slot Reader (UDMA 7) and not deleting anything until we review each file.  For some reason, the huge 17-22mpix CR2 files are fragile. 

                                   

                                  Obvious advice: Be careful...and good luck...

                                  • 14. Re: File corruption after import
                                    trshaner Community Member

                                    If you're on a Windows system you can test your memory card reader and cards using the below test program. I downloaded and tested the utility and it works well on my Windows 7 system. There are similar utilities available for Mac OS X, but I'm only running Windows so can't recommend specific ones.

                                     

                                    http://www.ricksdailytips.com/flash-drive-card-tester/

                                     

                                    Testing flash memory devices like CF, SD, and SSD can be confusing. All of these devices have the ability to reallocate bad sectors when detected. They also use wear-leveling, which means cards that appear good may over time exhibit errors. Wear-leveling attempts to control the number of memory cell read/write cycles by "distributing" them across the card's full storage-space. In a typical shoot you probably use less than the memory card's full capacity, but the card's wear-leveling algorithm is constantly reassigning different previously unused memory sectors. If any memory cells in that “previously unused” storage space are defective you may not experience file corruption until you’ve used the memory card numerous time.


                                    Flash memory devices also have the ability to map out and "reallocate" bad sectors using "spare blocks" reserved for this purpose. This means the next time you use the card it may work fine. This is also one of the reasons why it is  impossible to test a memory card with 100% assurance, since you can't physically access and test the cells in these "spare blocks."

                                     

                                    That said you still can run the above flash memory test utility, which will write, read, and compare all of the card's "accessible" memory cells for errors. I suggest running this test utility (or a similar diagnostic program) with every new memory card you purchase or those you suspect have issues. It's also a good test of your memory card reader. If you have more than one memory card that exhibits errors then your memory card reader is probably the cause. Try testing the same cards using a different card reader.

                                     

                                    WARNING: Prior to testing make sure you remove and backup all image files that are on the memory card. This test utility erases all data on the card! Also it is a good idea to reformat all memory cards inside your camera that you have tested with this utility or any other diagnsotic program.

                                     

                                    This Flash Memory Guide from Kingston has  information on flash memory devices applicable to most all manufacturers' devices.

                                    • 15. Re: File corruption after import
                                      trshaner Community Member

                                      Just thought this might be of interest.

                                       

                                      I just had an opportunity to test a new SanDisk Extreme 32GB 120MB/s UDMA7 card received today.  Using the Flash Drive Test Utility v1.14 (see above link)  11 errors were detected. I went back and rechecked SanDisk Extreme 16GB, 8GB, and 4GB cards that tested good when new. They all tested good indicating the 32GB card obviously has some issues. I ran the test two more times and got errors at the exact same 11 LBN addresses, so they are hard unrecoverable errors. The card is being being returned for a replacement! I'll report test results for the replacment card when recieved.

                                      Flash Memory Test Failure.jpg

                                      • 16. Re: File corruption after import
                                        trshaner Community Member

                                        Update On 32GB CF Card Errors

                                         

                                        I received a replacement SanDisk Extreme 32GB 120MB/s UDMA7 card from B&H and it exhibits  11 Fatal Read Errors at the exact same LBN locations as the first card using Flash Drive Test Utility v1.14. It appears this test utility is not compatible with 32GB UDMA7 cards. I sent an email to SanDisk Tech Support and received two responses:

                                         

                                        I would like to inform you that the software that you are using is not a suggested software to test our cards so I would recommend you to us the card in the device that you have. If you face any issues while you are using this card on your camera you can contact us back again and we will surely go ahead and assist you. You can also revert back to this email for further assistance or contact us back again for further assistance.

                                         

                                        What software do you recommend that I can use to test this card?

                                         

                                        I understand your concern regarding the card showing errors however would like to inform you that we do not have any suggested software to test our cards and if you face any issues while using this card on your camera then please contact us back again with your incident ID1735950 or revert back to this email for further assistance and we would be glad to replace the card for you.

                                         

                                        Although it's good to know SanDisk will replace defective cards that won't help end-users recover corrupted files on the card!

                                         

                                        A search on Google pulls up scant few Flash Memory test diagnostics, freeware or paid. I did find one called 'CheckFlash 1.16' written by a programmer based in the Ukraine. It is fast and uses two data patterns (Hex5, HexA) to insure all memory cells can be written to both 1 and 0 and read back correctly. The Flash Drive Test Utility 1.14 and others write a single test pattern and then read back and compare the data. This is fine to check the actually memory size to find "fake" cards and USB drives, but won't insure data integrity.

                                         

                                        There is only a Windows version, but it runs fine on my Windows 7 64bit system. The SanDisk Extreme 32GB 120MB/s UDMA7 card passed with no issues! Select your memory card in the dropdown 'Drive' selector and then use the settings as in the below screenshot. THIS WILL OVERWRITE ALL DATA, so first backup any files on the card to your hardrive.

                                         

                                        CheckFlash Diagnostic.jpg

                                         

                                        The 'Small pattern test' is alternating 1010 data followed by 0101 data on the 2nd pass. This insures insure all memory cells on the card are functioning properly. The 'Full pattern set' is much more extensive and will take quite a long time to run on large cards. If you have a card that has issues and it passes the 'Small pattern test' then you might want to run the 'Full pattern test' it before tossing the card in the garbage. If it passes both tests you probably have other issues.

                                         

                                        Use a good high-speed card reader, preferably USB3 or FireWire. You will need to Format the card when you replace it in the camera.

                                         

                                        Message was edited by: trshaner Corrected 0101 data test pattern.