33 Replies Latest reply: Aug 20, 2012 10:51 AM by sarangiman RSS

    LR4 not identifing duplicates?

    paulphotography Community Member

      Hi!

       

      I just upgraded to LR 4.0 and the  identify duplicate function does not appear to be identifing duplicates.

       

      I was cleaning off cards and wanted to make sure all the photos had been added to the catalog  when I realized that photos which have been imported, will reimport.

       

      The "don't import suspected duplicates" box is checked.  I verified one photo, _img_2597.cr2 , was both on the card and in the catalog.

      LR4 re-imported it - putting a duplicate in the catalog.

       

      I then added some new photos to the catalog and then tried to re-load them: the detection of duplicates worked.  It apears that photos from the 3.6 catalog are the issue.

       

      I have:

      rebooted the system

      opened up a backup catalog

      synchronize folders

       

      All to no avail. 

       

       

      Any ideas.

       

      Thanks!

       

      Paul 

        • 1. Re: LR4 not identifing duplicates?
          Jasonized Community Member

          If you're having problems with the LR3 catalog structure (which might be it, indeed), then try creating a new catalog, and using "import from catalog", import the old catalog.  That might "fix" the problem.


          Be sure you don't check "import negatives", as that would duplicate all of your images!

           

          Good luck!

          • 2. Re: LR4 not identifing duplicates?
            paulphotography Community Member

            Jason,

             

             

             

            Thanks for the fast reply.  I'm giving it a try.  Unfortunately in order to

            run LR 4 I needed to upgrade to Windows 7 ( from xp).  My backups are a bit

            challenged.  If the re-build doesn't work any other suggestions?

             

             

             

            Thanks!!

             

             

             

            Paul

            • 3. Re: LR4 not identifing duplicates?
              Jasonized Community Member

              Uhmm..  I'm at a loss here.  You've already upgraded to LR4, so that means your system must be running Win7 (or similar).  So what is the problem?

               

              You don't need to make "backups", as you will not be touching your old catalog in any fashion at all.  It will be safe and good.

               

              Create a new _blank_ catalog.  Import from your old catalog.  All that does is copy the information from the old catalog into the new one, it won't touch your old one or make any other changes.

               

              Or am I missing something?

               

              If it works, and you're happy, rename the catalog to the old one and you're back to "normal"!

               

              Cheers!

              • 4. Re: LR4 not identifing duplicates?
                Lee Jay Community Member

                I think images that go through the catalog upgrade process aren't detected consistently.  Once you've got those images off the cards, you'll probably see few, if any problems.  But I agree that's a bug.

                • 5. Re: LR4 not identifing duplicates?
                  paulphotography Community Member

                  Simply stated: I either can't find or have deleted the LR 3.6 catalog.  I'm

                  looking on the Carbonite site to see how recent that backup is, but it

                  look's rather old.

                   

                   

                   

                  I tried building a new catalog by importing the existing LR 4 catalog, but

                  the issue still exists: all photos loaded under 3.6 do not show up as a

                  duplicate.   Time for tech support?

                  • 6. Re: LR4 not identifing duplicates?
                    b_gossweiler Community Member

                    Paul,

                     

                    Any chance the images in your LR4 catalog got there via an "Import from Catalog" operation? There's a known bug in LR (3 and 4), which leads to non-recognition of duplicates after an import from catalog.

                     

                    Beat

                    • 7. Re: LR4 not identifing duplicates?
                      paulphotography Community Member

                      Beat,

                       

                       

                       

                      Between upgrading from XP to windows 7 and LR 3.6 to 4, the catalog could

                      have been made from an "import from Catalog".    I seem to recall making

                      the catalog twice - once on conversion then again, maybe via import.

                      Normally this would not be an issue.  Unfortunately I have about 15 cards I

                      need to clean from a recent trip to Turkey and I wanted to double check that

                      I didn't miss any photos.  I guess I'll just have to double check them

                      manually.

                       

                       

                       

                      Any rumors if Adobe is planning to fix the bug?

                       

                       

                       

                      Thanks!!

                       

                       

                       

                      Paul

                      • 8. Re: LR4 not identifing duplicates?
                        b_gossweiler Community Member

                        paulphotography wrote:

                         

                        Any rumors if Adobe is planning to fix the bug?

                        It is an accepted bug, but obviously not with high priority.

                         

                        Background information:

                         

                        Column importHash in catalog table AgLibraryFile does not get imported while doing an "Import from Catalog". This column contains a string made up of original filename, capture timestamp and file size and seems to be used to detect suspected duplicates.

                         

                        Beat

                        • 9. Re: LR4 not identifing duplicates?
                          Billzabub

                          Arghhhh... this explains a lot...

                           

                          I only just stumbled across this when I tried importing some new photos on a CF card which also had some from before the weekend (when I upgraded to LR4).

                           

                          Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?

                           

                          i.e. I had my LR4 catalog all looking ok - but with only recent edits in (lots of photos but no edits pre 2011). I then imported from old catalog, it found some duplicates (not sure how) but also lots of new photos (28,0000...)

                           

                          Dsiregarding the validity of that import for a moment (that's next weekends problem to solve)... If I now import from another old catalog which share some of the same photos as the previous import - will it find the duplicates or not?

                           

                          -> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'

                           

                          Bill

                          • 10. Re: LR4 not identifing duplicates?
                            b_gossweiler Community Member

                            Bill,

                             

                            Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?

                             

                            "Import from Catalog" does not seem to be affected by the problem I described above, just "normal" Import.

                             

                            -> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'

                            You might want to have a look at the Duplicate Finder plug-in.

                             

                            Beat

                            • 11. Re: LR4 not identifing duplicates?
                              Billzabub Community Member

                              Thanks Beat (again )

                               

                              I bought duplicate finder but I seem to have found a basic flaw - (maybe I'm not using correctly though...) Because a lot of my shots are taken at Continuous High speed shutter, Duplicate finder thinks that 4 consecutive shots are the same as they share the same time stamp {Nikon D3 at 9 fps). So that blows that..

                               

                              I just sent an email to the developer asking whether there could be an option to use the 'shutter count' exif field on Nikon - that would solve that problem....

                               

                              Thanks,

                               

                              Bill

                              • 12. Re: LR4 not identifing duplicates?
                                paulphotography Community Member

                                Let me state up front this is somewhat of a rant based upon my original post.  I have just attempted to clean some memory cards.  Rather than a quick and easy process it took me 5 minutes of checking and double checking to ensure I had transferred all the photos before deleting them from the card.  ( Here's the rant part)  Why wasn't this considered a severity 1 bug?  It's reduced functionality!  There was no notice which stated this upgrade does not contain all the functionality in 3.6.  A simple warning that stated the duplicate photo capability does not work in 4.0 with old catalogs: please ensure you have cleaned your old cards before upgrading.  

                                 

                                Releasing software with bugs in it is common.  In general the bugs are not a reduction in previous functions, maybe the new features are buggy, but not the old ones.  The release managers should be a ashamed of themselves.

                                 

                                Also, to Bill's point, if the software had a solid duplicate finder then you could at least clean up the mess poor software development made.

                                 

                                 

                                Thanks!

                                 

                                Paul

                                 

                                PS I feel much better now.

                                • 13. Re: LR4 not identifing duplicates?
                                  b_gossweiler Community Member

                                  Just to put something straight here:

                                   

                                  This bug is not related or confined to LR4 (it's happening in LR3.6 also) nor does it happen during a regular catalog upgrade from an LR3 catalog to an LR4 catalog. During a regular catalog upgrade the importHash information is converted into the LR4 catalog.

                                   

                                  The problem only happens when doing an explicit "Import from Catalog", in LR3 or LR4.

                                   

                                  Beat

                                  • 14. Re: LR4 not identifing duplicates?
                                    Dorin Nicolaescu-Musteață Community Member

                                    Yep. An Lightroom 1.0 as well.

                                     

                                    Btw, here's a similar bug report from 2 years ago: http://forums.adobe.com/message/2936668

                                     

                                    And another related (same old importHash) one that fails on images shot before 21st century. UPDATE: Ok, the second one has been corrected.

                                    • 15. Re: LR4 not identifing duplicates?
                                      paulphotography Community Member

                                      Beat,

                                       

                                      I did a catalog upgrade from LR 3.6 catalog to LR 4 catolog does this mean that

                                      the duplicate function should be working? 

                                       

                                       

                                      Thanks!

                                       

                                      Paul

                                      • 16. Re: LR4 not identifing duplicates?
                                        b_gossweiler Community Member

                                        Paul,

                                         

                                        Yes, to my knowledge it should, unless your LR3 catalog (or part of it) was created using an "Import from Catalog" at some point of time.

                                         

                                        Beat

                                        • 17. Re: LR4 not identifing duplicates?
                                          paulphotography Community Member

                                          Beat,

                                           

                                           

                                           

                                          Seems like I'm back where I started: an LR 4.0 catalog without the duplicate

                                          function working.  Not exactly clear how I got to this point, bad

                                          conversion, original created from a catalog, etc., but I'm stuck.   Is there

                                          anyway to get the functionality to work?  Would tech support have any

                                          utilities to 'correct' the catalog?

                                           

                                           

                                           

                                          Somehow it just seems wrong that I would loose functionality and there would

                                          be no method to fix it.  Thoughts and suggestions are welcomed.

                                           

                                           

                                           

                                          Thanks!!

                                           

                                           

                                           

                                          Paul

                                          • 18. Re: LR4 not identifing duplicates?
                                            b_gossweiler Community Member

                                            Paul,

                                             

                                            I don't know if Adobe support has a utility to fix, I suspect the information to perfectly fix it is not present in the catalog anymore (if you're hit by the problem I'm talking about).

                                             

                                            If you want to find out whether your LR4 catalog suffers from the bug I'm refering to, you can inspect your catalog using an SQLite utility. Check AgLibraryFile.importHash. If it is null, that's the reason for your problems. To be filled correctly, it needs to contain something looking like this:

                                            273140615:DSC_3204.NEF:13422285
                                            

                                             

                                            Beat

                                            • 19. Re: LR4 not identifing duplicates?
                                              paulphotography Community Member

                                              Beat,

                                               

                                               

                                               

                                              Thanks!  I'll take the catalog to work and look at via SQL Server.  Not sure

                                              why the importHash field can't be re-built.  The basic information, date of

                                              photo, original file name, etc. is contained within the catalog.  I'll post

                                              what I find.

                                               

                                               

                                               

                                              Many thanks!!!

                                               

                                               

                                               

                                              Paul

                                              • 20. Re: LR4 not identifing duplicates?
                                                b_gossweiler Community Member

                                                paulphotography wrote:

                                                 

                                                Not sure

                                                why the importHash field can't be re-built.  The basic information, date of

                                                photo, original file name, etc. is contained within the catalog.

                                                I kind of doubt it, but then I don't know all the information that's contained within the catalog. For example, AFAIK, when renaming during import, there's no original file name recorded in the catalog.

                                                 

                                                Beat

                                                • 21. Re: LR4 not identifing duplicates?
                                                  paulphotography Community Member

                                                  Beat,

                                                   

                                                  Used SQLite and took a look at the tables.  The AgLibraryFile.importHash is blank.  It appears that I can write a query to populate it.  I just need to know where file size is stored.

                                                   

                                                  Thanks!

                                                   

                                                  Paul

                                                  • 22. Re: LR4 not identifing duplicates?
                                                    b_gossweiler Community Member

                                                    I don't want to play devil's advocate here, Paul, but I still doubt you will be able to reconstruct importHash for the following reasons (all AFAIK):

                                                    • The original filename recorded in the catalog is - in case of renaming during import - not the filename as on the card at time of import
                                                    • The file size does not seem to be recorded in the catalog

                                                     

                                                    Also, you'd have to know the algorithm to calculate the time stamp used in importHash (I have no idea about it).

                                                     

                                                    3 hurdles to take for 3 subfields of importHash sounds like a lot to me ....

                                                     

                                                    Beat

                                                    • 23. Re: LR4 not identifing duplicates?
                                                      paulphotography Community Member

                                                      Beat,

                                                       

                                                       

                                                       

                                                      I think you're right.  I'll just push thru cleaning the cards.  Many thanks

                                                      for all your help and information.

                                                       

                                                       

                                                       

                                                      Paul

                                                      • 24. Re: LR4 not identifing duplicates?
                                                        gruhl28 Community Member

                                                        I am having the same problem as the original poster, and all I have done is upgrade from LR3 to LR4; I have not done any explicit Import from Catalog. And the thing is, it's not just photos that I imported before the upgrade that are not being recognized as duplicates; even ones I've imported after the upgrade are not being recognized. So this does indeed seem to be something new, not a pre-existing bug with Import from Catalog. Lightroom does seem to be recognizing a few duplicates, but not most.

                                                         

                                                        P.S. This is my first time using the forum, and I also had a problem replying to this thread; even though I was logged and it showed my username at the very top of the page, Reply was not shown, and on the left near Ask a Question it showed the Login/Register link. When I clicked that, it asked me to create a screenname, but would not accept mine because it had already been used (by me!). It did allow me to post a new question, though, and once I'd done that, it now allows me to reply to this thread. Any ideas? This is definitely secondary to the Duplicates problem, though.

                                                         

                                                        Thanks.

                                                        • 25. Re: LR4 not identifing duplicates?
                                                          paulphotography Community Member

                                                          Here's where I came out: the duplicate function does not work for photos

                                                          added under any version prior to LR 4.0 (for me).  The cause doesn't matter,

                                                          whether it is a conversion bug or a bad catalog, I'm still in the same

                                                          place: the function doesn't work.  After looking at the data base tables, it

                                                          appears that the data base design doesn't store the individual elements

                                                          which would allow the key field, importhash, to be re-created.  I have yet

                                                          to read of anything where adobe would acknowledge that they have a design

                                                          issue followed by an implementation issue.

                                                           

                                                           

                                                           

                                                          You may want to try to rebuild your LR4 catalog and see if that helps.  You

                                                          can also download SQLite and take a look at the tables (use a backup) to see

                                                          if the field is being populated.

                                                           

                                                           

                                                           

                                                          Sorry I couldn't help more. 

                                                           

                                                           

                                                           

                                                          Paul

                                                           

                                                           

                                                           

                                                          PS If anybody has any other suggestions please offer them up.

                                                          • 26. Re: LR4 not identifing duplicates?
                                                            b_gossweiler Community Member

                                                            gruhl28,

                                                             

                                                            I'm only aware of a problem created by importing as catalog. I cannot reproduce what you're seeing when directly importing into LR4. Something must be special in your case. Where are you importing from (camera, card, HD?).

                                                             

                                                            Beat

                                                            • 27. Re: LR4 not identifing duplicates?
                                                              gruhl28 Community Member

                                                              Hi b_gossweiler,

                                                               

                                                              I'm importing from a card. You don't have the problem? I'm on a Mac, running Snow Leopard, if that makes any difference.

                                                               

                                                              Thanks,

                                                              Glenn

                                                              • 28. Re: LR4 not identifing duplicates?
                                                                gruhl28 Community Member

                                                                Ok, I think I figured it out. In addition to Lightroom trying to import my images, Canon's ImageBrowser software was starting up and trying to import also, and that seemed to be interfering with Lightroom (that was happening when I was using LR3 also, but it didn't seem to get LR3 confused). After a bit of searching for how to stop ImageBrowser from starting up (I already had the option turned off in the software), I found that you have to go into Apple's Image Capture and turn off all applications there. I think LR4 is now recognizing duplicates correctly.

                                                                • 29. Re: LR4 not identifing duplicates?
                                                                  b_gossweiler Community Member

                                                                  Now that's odd ... and good to know. Thanks for reporting back!

                                                                   

                                                                  Beat

                                                                  • 30. Re: LR4 not identifing duplicates?
                                                                    sandro.franchi

                                                                    A customer of mine reported an issue with LR4 where the "suspected duplicates", checker or not, seems not to be working at all. He has all his files in folder \A (and subfolders) (already imported in the current catalog) and want to do a kind of "reorganization" (using the option to organize the folders by date in the right panel) and massive conversion to DNG, importing the same files with "Copy as DNG" to the folder "\B" in a different drive.

                                                                    No matters if the ignore duplicates is checked or not, most of the images are grayed in the central grid, without the checkbox needed to import it or not. Checking and unchecking the "ignore duplicates" does just nothing, there is no change at all. Any suggestion?

                                                                     

                                                                    PS: Everything seems to work "normal" with a new, blank catalog, of course, because there is no way to find a duplicate...

                                                                    • 31. Re: LR4 not identifing duplicates?
                                                                      Cornelia-I Community Member

                                                                      I have doubts about your scheme:

                                                                      It relies on importing images again. This can work if you start with an empty new catalog, and go for Import-Copy as DNG-destination date-based folders.

                                                                       

                                                                      But for this you would need to go via metadata saved to the image files in your current catalog. And you would loose all data which cannot be saved to the file, like collection-membership, virtual copies, stacking. (Virtual copies you could first export as DNG, so you would have real copies to re-import.)

                                                                       

                                                                       

                                                                      Normally to do a conversion to DNG you do NOT import them again, via "copy as DNG".

                                                                      You convert to DNG via menu Library - Convert Photos to DNG, check "Delete originals after successful conversion".

                                                                       

                                                                      I would not create a date-based FOLDER structure at all, but a date-based COLLECTION-structure, using smart collections.

                                                                      (Collapse the folder panel in library module. They are less useful than collections anyway as not accessible from all the modules.)

                                                                      This is fool-proof and way less work.

                                                                      Remember: folders are unimportant in LR, they just need to be handy storage buckets.

                                                                       

                                                                      Whatever you decide: do a proper backup of the current catalog and store that safely away, to have a restarting point in case of any mess.

                                                                      • 32. Re: LR4 not identifing duplicates?
                                                                        sandro.franchi Community Member

                                                                        I do agree with all you say Cornelia, thanks for your time!

                                                                        • 33. Re: LR4 not identifing duplicates?
                                                                          sarangiman

                                                                          I have this issue as well.

                                                                           

                                                                          However, it affects all new photos as well, so it's definitely *not* isolated to photos that were in my LR3 catalog.

                                                                           

                                                                          In other words, I've had LR4 for months. Last week I shot some photos, downloaded them, didn't erase the card, shot some more photos, then popped the card back into my computer today, & LR4 still shows the photos from the 'Previous Import' in the Import dialog under 'New Photos'.

                                                                           

                                                                          Sometimes LR4 corrects this & finds the duplicates if I exit out of the Import dialog & go back in again.

                                                                           

                                                                          But this time, it's just not working. It still lists all these photos as 'new', even though they're sitting their in the catalog, even under 'Previous Import'.

                                                                           

                                                                          I know the philosophy is to always wipe your card, but, y'know, it just doesn't happen sometimes b/c you forget.

                                                                           

                                                                          And then this manual comparison becomes pretty annoying.

                                                                           

                                                                          It'd be great if Adobe prioritized this... it was fine up until LR4.