LR4 not identifing duplicates?

If you're having problems with the LR3 catalog structure (which might be it, indeed), then try creating a new catalog, and using "import from catalog", import the old catalog.  That might "fix" the problem.

Be sure you don't check "import negatives", as that would duplicate all of your images!

Good luck!

Uhmm..  I'm at a loss here.  You've already upgraded to LR4, so that means your system must be running Win7 (or similar).  So what is the problem?

You don't need to make "backups", as you will not be touching your old catalog in any fashion at all.  It will be safe and good.

Create a new _blank_ catalog.  Import from your old catalog.  All that does is copy the information from the old catalog into the new one, it won't touch your old one or make any other changes.

Or am I missing something?

If it works, and you're happy, rename the catalog to the old one and you're back to "normal"!

Cheers!

I think images that go through the catalog upgrade process aren't detected consistently.  Once you've got those images off the cards, you'll probably see few, if any problems.  But I agree that's a bug.

Paul,

Any chance the images in your LR4 catalog got there via an "Import from Catalog" operation? There's a known bug in LR (3 and 4), which leads to non-recognition of duplicates after an import from catalog.

Beat

paulphotography wrote:

 

Any rumors if Adobe is planning to fix the bug?

It is an accepted bug, but obviously not with high priority.

Background information:

Column importHash in catalog table AgLibraryFile does not get imported while doing an "Import from Catalog". This column contains a string made up of original filename, capture timestamp and file size and seems to be used to detect suspected duplicates.

Beat

Arghhhh... this explains a lot...

I only just stumbled across this when I tried importing some new photos on a CF card which also had some from before the weekend (when I upgraded to LR4).

Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?

i.e. I had my LR4 catalog all looking ok - but with only recent edits in (lots of photos but no edits pre 2011). I then imported from old catalog, it found some duplicates (not sure how) but also lots of new photos (28,0000...)

Dsiregarding the validity of that import for a moment (that's next weekends problem to solve)... If I now import from another old catalog which share some of the same photos as the previous import - will it find the duplicates or not?

-> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'

Bill

Bill,

Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?

"Import from Catalog" does not seem to be affected by the problem I described above, just "normal" Import.

-> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'

You might want to have a look at the Duplicate Finder plug-in.

Beat

Thanks Beat (again )

I bought duplicate finder but I seem to have found a basic flaw - (maybe I'm not using correctly though...) Because a lot of my shots are taken at Continuous High speed shutter, Duplicate finder thinks that 4 consecutive shots are the same as they share the same time stamp {Nikon D3 at 9 fps). So that blows that..

I just sent an email to the developer asking whether there could be an option to use the 'shutter count' exif field on Nikon - that would solve that problem....

Thanks,

Bill

Just to put something straight here:

This bug is not related or confined to LR4 (it's happening in LR3.6 also) nor does it happen during a regular catalog upgrade from an LR3 catalog to an LR4 catalog. During a regular catalog upgrade the importHash information is converted into the LR4 catalog.

The problem only happens when doing an explicit "Import from Catalog", in LR3 or LR4.

Beat

Yep. An Lightroom 1.0 as well.

Btw, here's a similar bug report from 2 years ago: http://forums.adobe.com/message/2936668

And another related (same old importHash) one that fails on images shot before 21st century. UPDATE: Ok, the second one has been corrected.

Paul,

Yes, to my knowledge it should, unless your LR3 catalog (or part of it) was created using an "Import from Catalog" at some point of time.

Beat

Paul,

I don't know if Adobe support has a utility to fix, I suspect the information to perfectly fix it is not present in the catalog anymore (if you're hit by the problem I'm talking about).

If you want to find out whether your LR4 catalog suffers from the bug I'm refering to, you can inspect your catalog using an SQLite utility. Check AgLibraryFile.importHash. If it is null, that's the reason for your problems. To be filled correctly, it needs to contain something looking like this:

273140615:DSC_3204.NEF:13422285

Beat

paulphotography wrote:

 

Not sure

why the importHash field can't be re-built.  The basic information, date of

photo, original file name, etc. is contained within the catalog.

I kind of doubt it, but then I don't know all the information that's contained within the catalog. For example, AFAIK, when renaming during import, there's no original file name recorded in the catalog.

Beat

I don't want to play devil's advocate here, Paul, but I still doubt you will be able to reconstruct importHash for the following reasons (all AFAIK):

• The original filename recorded in the catalog is - in case of renaming during import - not the filename as on the card at time of import
• The file size does not seem to be recorded in the catalog

Also, you'd have to know the algorithm to calculate the time stamp used in importHash (I have no idea about it).

3 hurdles to take for 3 subfields of importHash sounds like a lot to me ....

Beat

I am having the same problem as the original poster, and all I have done is upgrade from LR3 to LR4; I have not done any explicit Import from Catalog. And the thing is, it's not just photos that I imported before the upgrade that are not being recognized as duplicates; even ones I've imported after the upgrade are not being recognized. So this does indeed seem to be something new, not a pre-existing bug with Import from Catalog. Lightroom does seem to be recognizing a few duplicates, but not most.

P.S. This is my first time using the forum, and I also had a problem replying to this thread; even though I was logged and it showed my username at the very top of the page, Reply was not shown, and on the left near Ask a Question it showed the Login/Register link. When I clicked that, it asked me to create a screenname, but would not accept mine because it had already been used (by me!). It did allow me to post a new question, though, and once I'd done that, it now allows me to reply to this thread. Any ideas? This is definitely secondary to the Duplicates problem, though.

Thanks.

gruhl28,

I'm only aware of a problem created by importing as catalog. I cannot reproduce what you're seeing when directly importing into LR4. Something must be special in your case. Where are you importing from (camera, card, HD?).

Beat

Hi b_gossweiler,

I'm importing from a card. You don't have the problem? I'm on a Mac, running Snow Leopard, if that makes any difference.

Thanks,

Glenn

Ok, I think I figured it out. In addition to Lightroom trying to import my images, Canon's ImageBrowser software was starting up and trying to import also, and that seemed to be interfering with Lightroom (that was happening when I was using LR3 also, but it didn't seem to get LR3 confused). After a bit of searching for how to stop ImageBrowser from starting up (I already had the option turned off in the software), I found that you have to go into Apple's Image Capture and turn off all applications there. I think LR4 is now recognizing duplicates correctly.

Now that's odd ... and good to know. Thanks for reporting back!

Beat

A customer of mine reported an issue with LR4 where the "suspected duplicates", checker or not, seems not to be working at all. He has all his files in folder \A (and subfolders) (already imported in the current catalog) and want to do a kind of "reorganization" (using the option to organize the folders by date in the right panel) and massive conversion to DNG, importing the same files with "Copy as DNG" to the folder "\B" in a different drive.

No matters if the ignore duplicates is checked or not, most of the images are grayed in the central grid, without the checkbox needed to import it or not. Checking and unchecking the "ignore duplicates" does just nothing, there is no change at all. Any suggestion?

PS: Everything seems to work "normal" with a new, blank catalog, of course, because there is no way to find a duplicate...

I have doubts about your scheme:

It relies on importing images again. This can work if you start with an empty new catalog, and go for Import-Copy as DNG-destination date-based folders.

But for this you would need to go via metadata saved to the image files in your current catalog. And you would loose all data which cannot be saved to the file, like collection-membership, virtual copies, stacking. (Virtual copies you could first export as DNG, so you would have real copies to re-import.)

Normally to do a conversion to DNG you do NOT import them again, via "copy as DNG".

You convert to DNG via menu Library - Convert Photos to DNG, check "Delete originals after successful conversion".

I would not create a date-based FOLDER structure at all, but a date-based COLLECTION-structure, using smart collections.

(Collapse the folder panel in library module. They are less useful than collections anyway as not accessible from all the modules.)

This is fool-proof and way less work.

Remember: folders are unimportant in LR, they just need to be handy storage buckets.

Whatever you decide: do a proper backup of the current catalog and store that safely away, to have a restarting point in case of any mess.

I do agree with all you say Cornelia, thanks for your time!

I have this issue as well.

However, it affects all new photos as well, so it's definitely *not* isolated to photos that were in my LR3 catalog.

In other words, I've had LR4 for months. Last week I shot some photos, downloaded them, didn't erase the card, shot some more photos, then popped the card back into my computer today, & LR4 still shows the photos from the 'Previous Import' in the Import dialog under 'New Photos'.

Sometimes LR4 corrects this & finds the duplicates if I exit out of the Import dialog & go back in again.

But this time, it's just not working. It still lists all these photos as 'new', even though they're sitting their in the catalog, even under 'Previous Import'.

I know the philosophy is to always wipe your card, but, y'know, it just doesn't happen sometimes b/c you forget.

And then this manual comparison becomes pretty annoying.

It'd be great if Adobe prioritized this... it was fine up until LR4.