Skip navigation
paulphotography
Currently Being Moderated

LR4 not identifing duplicates?

Mar 17, 2012 11:31 AM

Tags: #lr4

Hi!

 

I just upgraded to LR 4.0 and the  identify duplicate function does not appear to be identifing duplicates.

 

I was cleaning off cards and wanted to make sure all the photos had been added to the catalog  when I realized that photos which have been imported, will reimport.

 

The "don't import suspected duplicates" box is checked.  I verified one photo, _img_2597.cr2 , was both on the card and in the catalog.

LR4 re-imported it - putting a duplicate in the catalog.

 

I then added some new photos to the catalog and then tried to re-load them: the detection of duplicates worked.  It apears that photos from the 3.6 catalog are the issue.

 

I have:

rebooted the system

opened up a backup catalog

synchronize folders

 

All to no avail. 

 

 

Any ideas.

 

Thanks!

 

Paul 

 
Replies
  • Currently Being Moderated
    Mar 17, 2012 11:56 AM   in reply to paulphotography

    If you're having problems with the LR3 catalog structure (which might be it, indeed), then try creating a new catalog, and using "import from catalog", import the old catalog.  That might "fix" the problem.


    Be sure you don't check "import negatives", as that would duplicate all of your images!

     

    Good luck!

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 17, 2012 2:12 PM   in reply to paulphotography

    Uhmm..  I'm at a loss here.  You've already upgraded to LR4, so that means your system must be running Win7 (or similar).  So what is the problem?

     

    You don't need to make "backups", as you will not be touching your old catalog in any fashion at all.  It will be safe and good.

     

    Create a new _blank_ catalog.  Import from your old catalog.  All that does is copy the information from the old catalog into the new one, it won't touch your old one or make any other changes.

     

    Or am I missing something?

     

    If it works, and you're happy, rename the catalog to the old one and you're back to "normal"!

     

    Cheers!

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 17, 2012 2:17 PM   in reply to paulphotography

    I think images that go through the catalog upgrade process aren't detected consistently.  Once you've got those images off the cards, you'll probably see few, if any problems.  But I agree that's a bug.

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 17, 2012 5:59 PM   in reply to paulphotography

    Paul,

     

    Any chance the images in your LR4 catalog got there via an "Import from Catalog" operation? There's a known bug in LR (3 and 4), which leads to non-recognition of duplicates after an import from catalog.

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 17, 2012 6:42 PM   in reply to paulphotography

    paulphotography wrote:

     

    Any rumors if Adobe is planning to fix the bug?

    It is an accepted bug, but obviously not with high priority.

     

    Background information:

     

    Column importHash in catalog table AgLibraryFile does not get imported while doing an "Import from Catalog". This column contains a string made up of original filename, capture timestamp and file size and seems to be used to detect suspected duplicates.

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 20, 2012 8:29 AM   in reply to b_gossweiler

    Arghhhh... this explains a lot...

     

    I only just stumbled across this when I tried importing some new photos on a CF card which also had some from before the weekend (when I upgraded to LR4).

     

    Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?

     

    i.e. I had my LR4 catalog all looking ok - but with only recent edits in (lots of photos but no edits pre 2011). I then imported from old catalog, it found some duplicates (not sure how) but also lots of new photos (28,0000...)

     

    Dsiregarding the validity of that import for a moment (that's next weekends problem to solve)... If I now import from another old catalog which share some of the same photos as the previous import - will it find the duplicates or not?

     

    -> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'

     

    Bill

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 20, 2012 1:00 PM   in reply to Billzabub

    Bill,

     

    Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?

     

    "Import from Catalog" does not seem to be affected by the problem I described above, just "normal" Import.

     

    -> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'

    You might want to have a look at the Duplicate Finder plug-in.

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 20, 2012 8:05 PM   in reply to b_gossweiler

    Thanks Beat (again )

     

    I bought duplicate finder but I seem to have found a basic flaw - (maybe I'm not using correctly though...) Because a lot of my shots are taken at Continuous High speed shutter, Duplicate finder thinks that 4 consecutive shots are the same as they share the same time stamp {Nikon D3 at 9 fps). So that blows that..

     

    I just sent an email to the developer asking whether there could be an option to use the 'shutter count' exif field on Nikon - that would solve that problem....

     

    Thanks,

     

    Bill

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 21, 2012 5:45 AM   in reply to paulphotography

    Just to put something straight here:

     

    This bug is not related or confined to LR4 (it's happening in LR3.6 also) nor does it happen during a regular catalog upgrade from an LR3 catalog to an LR4 catalog. During a regular catalog upgrade the importHash information is converted into the LR4 catalog.

     

    The problem only happens when doing an explicit "Import from Catalog", in LR3 or LR4.

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 21, 2012 6:24 AM   in reply to b_gossweiler

    Yep. An Lightroom 1.0 as well.

     

    Btw, here's a similar bug report from 2 years ago: http://forums.adobe.com/message/2936668

     

    And another related (same old importHash) one that fails on images shot before 21st century. UPDATE: Ok, the second one has been corrected.

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 21, 2012 3:49 PM   in reply to paulphotography

    Paul,

     

    Yes, to my knowledge it should, unless your LR3 catalog (or part of it) was created using an "Import from Catalog" at some point of time.

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 21, 2012 4:27 PM   in reply to paulphotography

    Paul,

     

    I don't know if Adobe support has a utility to fix, I suspect the information to perfectly fix it is not present in the catalog anymore (if you're hit by the problem I'm talking about).

     

    If you want to find out whether your LR4 catalog suffers from the bug I'm refering to, you can inspect your catalog using an SQLite utility. Check AgLibraryFile.importHash. If it is null, that's the reason for your problems. To be filled correctly, it needs to contain something looking like this:

    273140615:DSC_3204.NEF:13422285
    

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 22, 2012 1:26 AM   in reply to paulphotography

    paulphotography wrote:

     

    Not sure

    why the importHash field can't be re-built.  The basic information, date of

    photo, original file name, etc. is contained within the catalog.

    I kind of doubt it, but then I don't know all the information that's contained within the catalog. For example, AFAIK, when renaming during import, there's no original file name recorded in the catalog.

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 22, 2012 2:53 PM   in reply to paulphotography

    I don't want to play devil's advocate here, Paul, but I still doubt you will be able to reconstruct importHash for the following reasons (all AFAIK):

    • The original filename recorded in the catalog is - in case of renaming during import - not the filename as on the card at time of import
    • The file size does not seem to be recorded in the catalog

     

    Also, you'd have to know the algorithm to calculate the time stamp used in importHash (I have no idea about it).

     

    3 hurdles to take for 3 subfields of importHash sounds like a lot to me ....

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 26, 2012 7:16 AM   in reply to b_gossweiler

    I am having the same problem as the original poster, and all I have done is upgrade from LR3 to LR4; I have not done any explicit Import from Catalog. And the thing is, it's not just photos that I imported before the upgrade that are not being recognized as duplicates; even ones I've imported after the upgrade are not being recognized. So this does indeed seem to be something new, not a pre-existing bug with Import from Catalog. Lightroom does seem to be recognizing a few duplicates, but not most.

     

    P.S. This is my first time using the forum, and I also had a problem replying to this thread; even though I was logged and it showed my username at the very top of the page, Reply was not shown, and on the left near Ask a Question it showed the Login/Register link. When I clicked that, it asked me to create a screenname, but would not accept mine because it had already been used (by me!). It did allow me to post a new question, though, and once I'd done that, it now allows me to reply to this thread. Any ideas? This is definitely secondary to the Duplicates problem, though.

     

    Thanks.

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 26, 2012 4:06 PM   in reply to gruhl28

    gruhl28,

     

    I'm only aware of a problem created by importing as catalog. I cannot reproduce what you're seeing when directly importing into LR4. Something must be special in your case. Where are you importing from (camera, card, HD?).

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 27, 2012 5:21 PM   in reply to b_gossweiler

    Hi b_gossweiler,

     

    I'm importing from a card. You don't have the problem? I'm on a Mac, running Snow Leopard, if that makes any difference.

     

    Thanks,

    Glenn

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 27, 2012 8:07 PM   in reply to gruhl28

    Ok, I think I figured it out. In addition to Lightroom trying to import my images, Canon's ImageBrowser software was starting up and trying to import also, and that seemed to be interfering with Lightroom (that was happening when I was using LR3 also, but it didn't seem to get LR3 confused). After a bit of searching for how to stop ImageBrowser from starting up (I already had the option turned off in the software), I found that you have to go into Apple's Image Capture and turn off all applications there. I think LR4 is now recognizing duplicates correctly.

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 28, 2012 1:04 AM   in reply to gruhl28

    Now that's odd ... and good to know. Thanks for reporting back!

     

    Beat

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 11, 2012 12:08 PM   in reply to paulphotography

    A customer of mine reported an issue with LR4 where the "suspected duplicates", checker or not, seems not to be working at all. He has all his files in folder \A (and subfolders) (already imported in the current catalog) and want to do a kind of "reorganization" (using the option to organize the folders by date in the right panel) and massive conversion to DNG, importing the same files with "Copy as DNG" to the folder "\B" in a different drive.

    No matters if the ignore duplicates is checked or not, most of the images are grayed in the central grid, without the checkbox needed to import it or not. Checking and unchecking the "ignore duplicates" does just nothing, there is no change at all. Any suggestion?

     

    PS: Everything seems to work "normal" with a new, blank catalog, of course, because there is no way to find a duplicate...

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 11, 2012 1:29 PM   in reply to sandro.franchi

    I have doubts about your scheme:

    It relies on importing images again. This can work if you start with an empty new catalog, and go for Import-Copy as DNG-destination date-based folders.

     

    But for this you would need to go via metadata saved to the image files in your current catalog. And you would loose all data which cannot be saved to the file, like collection-membership, virtual copies, stacking. (Virtual copies you could first export as DNG, so you would have real copies to re-import.)

     

     

    Normally to do a conversion to DNG you do NOT import them again, via "copy as DNG".

    You convert to DNG via menu Library - Convert Photos to DNG, check "Delete originals after successful conversion".

     

    I would not create a date-based FOLDER structure at all, but a date-based COLLECTION-structure, using smart collections.

    (Collapse the folder panel in library module. They are less useful than collections anyway as not accessible from all the modules.)

    This is fool-proof and way less work.

    Remember: folders are unimportant in LR, they just need to be handy storage buckets.

     

    Whatever you decide: do a proper backup of the current catalog and store that safely away, to have a restarting point in case of any mess.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 11, 2012 2:15 PM   in reply to Cornelia-I

    I do agree with all you say Cornelia, thanks for your time!

     
    |
    Mark as:
  • Currently Being Moderated
    Aug 20, 2012 10:51 AM   in reply to paulphotography

    I have this issue as well.

     

    However, it affects all new photos as well, so it's definitely *not* isolated to photos that were in my LR3 catalog.

     

    In other words, I've had LR4 for months. Last week I shot some photos, downloaded them, didn't erase the card, shot some more photos, then popped the card back into my computer today, & LR4 still shows the photos from the 'Previous Import' in the Import dialog under 'New Photos'.

     

    Sometimes LR4 corrects this & finds the duplicates if I exit out of the Import dialog & go back in again.

     

    But this time, it's just not working. It still lists all these photos as 'new', even though they're sitting their in the catalog, even under 'Previous Import'.

     

    I know the philosophy is to always wipe your card, but, y'know, it just doesn't happen sometimes b/c you forget.

     

    And then this manual comparison becomes pretty annoying.

     

    It'd be great if Adobe prioritized this... it was fine up until LR4.

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)