-
1. Re: LR4 not identifing duplicates?
Jasonized Mar 17, 2012 11:56 AM (in response to paulphotography)If you're having problems with the LR3 catalog structure (which might be it, indeed), then try creating a new catalog, and using "import from catalog", import the old catalog. That might "fix" the problem.
Be sure you don't check "import negatives", as that would duplicate all of your images!Good luck!
-
2. Re: LR4 not identifing duplicates?
paulphotography Mar 17, 2012 1:43 PM (in response to Jasonized)Jason,
Thanks for the fast reply. I'm giving it a try. Unfortunately in order to
run LR 4 I needed to upgrade to Windows 7 ( from xp). My backups are a bit
challenged. If the re-build doesn't work any other suggestions?
Thanks!!
Paul
-
3. Re: LR4 not identifing duplicates?
Jasonized Mar 17, 2012 2:12 PM (in response to paulphotography)Uhmm.. I'm at a loss here. You've already upgraded to LR4, so that means your system must be running Win7 (or similar). So what is the problem?
You don't need to make "backups", as you will not be touching your old catalog in any fashion at all. It will be safe and good.
Create a new _blank_ catalog. Import from your old catalog. All that does is copy the information from the old catalog into the new one, it won't touch your old one or make any other changes.
Or am I missing something?
If it works, and you're happy, rename the catalog to the old one and you're back to "normal"!
Cheers!
-
4. Re: LR4 not identifing duplicates?
Lee Jay Mar 17, 2012 2:17 PM (in response to paulphotography)I think images that go through the catalog upgrade process aren't detected consistently. Once you've got those images off the cards, you'll probably see few, if any problems. But I agree that's a bug.
-
5. Re: LR4 not identifing duplicates?
paulphotography Mar 17, 2012 2:20 PM (in response to Jasonized)Simply stated: I either can't find or have deleted the LR 3.6 catalog. I'm
looking on the Carbonite site to see how recent that backup is, but it
look's rather old.
I tried building a new catalog by importing the existing LR 4 catalog, but
the issue still exists: all photos loaded under 3.6 do not show up as a
duplicate. Time for tech support?
-
6. Re: LR4 not identifing duplicates?
b_gossweiler Mar 17, 2012 5:59 PM (in response to paulphotography)Paul,
Any chance the images in your LR4 catalog got there via an "Import from Catalog" operation? There's a known bug in LR (3 and 4), which leads to non-recognition of duplicates after an import from catalog.
Beat
-
7. Re: LR4 not identifing duplicates?
paulphotography Mar 17, 2012 6:13 PM (in response to b_gossweiler)Beat,
Between upgrading from XP to windows 7 and LR 3.6 to 4, the catalog could
have been made from an "import from Catalog". I seem to recall making
the catalog twice - once on conversion then again, maybe via import.
Normally this would not be an issue. Unfortunately I have about 15 cards I
need to clean from a recent trip to Turkey and I wanted to double check that
I didn't miss any photos. I guess I'll just have to double check them
manually.
Any rumors if Adobe is planning to fix the bug?
Thanks!!
Paul
-
8. Re: LR4 not identifing duplicates?
b_gossweiler Mar 17, 2012 6:42 PM (in response to paulphotography)paulphotography wrote:
Any rumors if Adobe is planning to fix the bug?
It is an accepted bug, but obviously not with high priority.
Background information:
Column importHash in catalog table AgLibraryFile does not get imported while doing an "Import from Catalog". This column contains a string made up of original filename, capture timestamp and file size and seems to be used to detect suspected duplicates.
Beat
-
9. Re: LR4 not identifing duplicates?
Billzabub Mar 20, 2012 8:29 AM (in response to b_gossweiler)Arghhhh... this explains a lot...
I only just stumbled across this when I tried importing some new photos on a CF card which also had some from before the weekend (when I upgraded to LR4).
Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?
i.e. I had my LR4 catalog all looking ok - but with only recent edits in (lots of photos but no edits pre 2011). I then imported from old catalog, it found some duplicates (not sure how) but also lots of new photos (28,0000...)
Dsiregarding the validity of that import for a moment (that's next weekends problem to solve)... If I now import from another old catalog which share some of the same photos as the previous import - will it find the duplicates or not?
-> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'
Bill
-
10. Re: LR4 not identifing duplicates?
b_gossweiler Mar 20, 2012 1:00 PM (in response to Billzabub)Bill,
Ok so I get that any recent photos (on CF or whatever) I have to watch out for... however - what is the experience when using 'import from catalog' multiple times?
"Import from Catalog" does not seem to be affected by the problem I described above, just "normal" Import.
-> I would really love a LR built in 'De-duplicator' function - then I could get rid of all my duplicate photos without LR getting totally confused. Preferably with some simple rules - e.g. 'get rid of version without edits'
You might want to have a look at the Duplicate Finder plug-in.
Beat
-
11. Re: LR4 not identifing duplicates?
Billzabub Mar 20, 2012 8:05 PM (in response to b_gossweiler)Thanks Beat (again )
I bought duplicate finder but I seem to have found a basic flaw - (maybe I'm not using correctly though...) Because a lot of my shots are taken at Continuous High speed shutter, Duplicate finder thinks that 4 consecutive shots are the same as they share the same time stamp {Nikon D3 at 9 fps). So that blows that..
I just sent an email to the developer asking whether there could be an option to use the 'shutter count' exif field on Nikon - that would solve that problem....
Thanks,
Bill
-
12. Re: LR4 not identifing duplicates?
paulphotography Mar 21, 2012 4:02 AM (in response to paulphotography)Let me state up front this is somewhat of a rant based upon my original post. I have just attempted to clean some memory cards. Rather than a quick and easy process it took me 5 minutes of checking and double checking to ensure I had transferred all the photos before deleting them from the card. ( Here's the rant part) Why wasn't this considered a severity 1 bug? It's reduced functionality! There was no notice which stated this upgrade does not contain all the functionality in 3.6. A simple warning that stated the duplicate photo capability does not work in 4.0 with old catalogs: please ensure you have cleaned your old cards before upgrading.
Releasing software with bugs in it is common. In general the bugs are not a reduction in previous functions, maybe the new features are buggy, but not the old ones. The release managers should be a ashamed of themselves.
Also, to Bill's point, if the software had a solid duplicate finder then you could at least clean up the mess poor software development made.
Thanks!
Paul
PS I feel much better now.
-
13. Re: LR4 not identifing duplicates?
b_gossweiler Mar 21, 2012 5:45 AM (in response to paulphotography)Just to put something straight here:
This bug is not related or confined to LR4 (it's happening in LR3.6 also) nor does it happen during a regular catalog upgrade from an LR3 catalog to an LR4 catalog. During a regular catalog upgrade the importHash information is converted into the LR4 catalog.
The problem only happens when doing an explicit "Import from Catalog", in LR3 or LR4.
Beat
-
14. Re: LR4 not identifing duplicates?
dorin_nicolaescu Mar 21, 2012 6:24 AM (in response to b_gossweiler)Yep. An Lightroom 1.0 as well.
Btw, here's a similar bug report from 2 years ago: http://forums.adobe.com/message/2936668
And another related (same old importHash) one that fails on images shot before 21st century. UPDATE: Ok, the second one has been corrected.
-
15. Re: LR4 not identifing duplicates?
paulphotography Mar 21, 2012 7:57 AM (in response to b_gossweiler)Beat,
I did a catalog upgrade from LR 3.6 catalog to LR 4 catolog does this mean that
the duplicate function should be working?
Thanks!
Paul
-
16. Re: LR4 not identifing duplicates?
b_gossweiler Mar 21, 2012 3:49 PM (in response to paulphotography)Paul,
Yes, to my knowledge it should, unless your LR3 catalog (or part of it) was created using an "Import from Catalog" at some point of time.
Beat
-
17. Re: LR4 not identifing duplicates?
paulphotography Mar 21, 2012 4:03 PM (in response to b_gossweiler)Beat,
Seems like I'm back where I started: an LR 4.0 catalog without the duplicate
function working. Not exactly clear how I got to this point, bad
conversion, original created from a catalog, etc., but I'm stuck. Is there
anyway to get the functionality to work? Would tech support have any
utilities to 'correct' the catalog?
Somehow it just seems wrong that I would loose functionality and there would
be no method to fix it. Thoughts and suggestions are welcomed.
Thanks!!
Paul
-
18. Re: LR4 not identifing duplicates?
b_gossweiler Mar 21, 2012 4:27 PM (in response to paulphotography)Paul,
I don't know if Adobe support has a utility to fix, I suspect the information to perfectly fix it is not present in the catalog anymore (if you're hit by the problem I'm talking about).
If you want to find out whether your LR4 catalog suffers from the bug I'm refering to, you can inspect your catalog using an SQLite utility. Check AgLibraryFile.importHash. If it is null, that's the reason for your problems. To be filled correctly, it needs to contain something looking like this:
273140615:DSC_3204.NEF:13422285
Beat
-
19. Re: LR4 not identifing duplicates?
paulphotography Mar 21, 2012 6:59 PM (in response to b_gossweiler)Beat,
Thanks! I'll take the catalog to work and look at via SQL Server. Not sure
why the importHash field can't be re-built. The basic information, date of
photo, original file name, etc. is contained within the catalog. I'll post
what I find.
Many thanks!!!
Paul
-
20. Re: LR4 not identifing duplicates?
b_gossweiler Mar 22, 2012 1:26 AM (in response to paulphotography)paulphotography wrote:
Not sure
why the importHash field can't be re-built. The basic information, date of
photo, original file name, etc. is contained within the catalog.
I kind of doubt it, but then I don't know all the information that's contained within the catalog. For example, AFAIK, when renaming during import, there's no original file name recorded in the catalog.
Beat
-
21. Re: LR4 not identifing duplicates?
paulphotography Mar 22, 2012 10:09 AM (in response to b_gossweiler)Beat,
Used SQLite and took a look at the tables. The AgLibraryFile.importHash is blank. It appears that I can write a query to populate it. I just need to know where file size is stored.
Thanks!
Paul
-
22. Re: LR4 not identifing duplicates?
b_gossweiler Mar 22, 2012 2:53 PM (in response to paulphotography)I don't want to play devil's advocate here, Paul, but I still doubt you will be able to reconstruct importHash for the following reasons (all AFAIK):
- The original filename recorded in the catalog is - in case of renaming during import - not the filename as on the card at time of import
- The file size does not seem to be recorded in the catalog
Also, you'd have to know the algorithm to calculate the time stamp used in importHash (I have no idea about it).
3 hurdles to take for 3 subfields of importHash sounds like a lot to me ....
Beat
-
23. Re: LR4 not identifing duplicates?
paulphotography Mar 22, 2012 4:48 PM (in response to b_gossweiler)Beat,
I think you're right. I'll just push thru cleaning the cards. Many thanks
for all your help and information.
Paul
-
24. Re: LR4 not identifing duplicates?
gruhl28 Mar 26, 2012 7:16 AM (in response to b_gossweiler)I am having the same problem as the original poster, and all I have done is upgrade from LR3 to LR4; I have not done any explicit Import from Catalog. And the thing is, it's not just photos that I imported before the upgrade that are not being recognized as duplicates; even ones I've imported after the upgrade are not being recognized. So this does indeed seem to be something new, not a pre-existing bug with Import from Catalog. Lightroom does seem to be recognizing a few duplicates, but not most.
P.S. This is my first time using the forum, and I also had a problem replying to this thread; even though I was logged and it showed my username at the very top of the page, Reply was not shown, and on the left near Ask a Question it showed the Login/Register link. When I clicked that, it asked me to create a screenname, but would not accept mine because it had already been used (by me!). It did allow me to post a new question, though, and once I'd done that, it now allows me to reply to this thread. Any ideas? This is definitely secondary to the Duplicates problem, though.
Thanks.
-
25. Re: LR4 not identifing duplicates?
paulphotography Mar 26, 2012 12:14 PM (in response to gruhl28)Here's where I came out: the duplicate function does not work for photos
added under any version prior to LR 4.0 (for me). The cause doesn't matter,
whether it is a conversion bug or a bad catalog, I'm still in the same
place: the function doesn't work. After looking at the data base tables, it
appears that the data base design doesn't store the individual elements
which would allow the key field, importhash, to be re-created. I have yet
to read of anything where adobe would acknowledge that they have a design
issue followed by an implementation issue.
You may want to try to rebuild your LR4 catalog and see if that helps. You
can also download SQLite and take a look at the tables (use a backup) to see
if the field is being populated.
Sorry I couldn't help more.
Paul
PS If anybody has any other suggestions please offer them up.
-
26. Re: LR4 not identifing duplicates?
b_gossweiler Mar 26, 2012 4:06 PM (in response to gruhl28)gruhl28,
I'm only aware of a problem created by importing as catalog. I cannot reproduce what you're seeing when directly importing into LR4. Something must be special in your case. Where are you importing from (camera, card, HD?).
Beat
-
27. Re: LR4 not identifing duplicates?
gruhl28 Mar 27, 2012 5:21 PM (in response to b_gossweiler)Hi b_gossweiler,
I'm importing from a card. You don't have the problem? I'm on a Mac, running Snow Leopard, if that makes any difference.
Thanks,
Glenn
-
28. Re: LR4 not identifing duplicates?
gruhl28 Mar 27, 2012 8:07 PM (in response to gruhl28)Ok, I think I figured it out. In addition to Lightroom trying to import my images, Canon's ImageBrowser software was starting up and trying to import also, and that seemed to be interfering with Lightroom (that was happening when I was using LR3 also, but it didn't seem to get LR3 confused). After a bit of searching for how to stop ImageBrowser from starting up (I already had the option turned off in the software), I found that you have to go into Apple's Image Capture and turn off all applications there. I think LR4 is now recognizing duplicates correctly.
-
29. Re: LR4 not identifing duplicates?
b_gossweiler Mar 28, 2012 1:04 AM (in response to gruhl28)Now that's odd ... and good to know. Thanks for reporting back!
Beat
-
30. Re: LR4 not identifing duplicates?
sandro.franchi Apr 11, 2012 12:08 PM (in response to paulphotography)A customer of mine reported an issue with LR4 where the "suspected duplicates", checker or not, seems not to be working at all. He has all his files in folder \A (and subfolders) (already imported in the current catalog) and want to do a kind of "reorganization" (using the option to organize the folders by date in the right panel) and massive conversion to DNG, importing the same files with "Copy as DNG" to the folder "\B" in a different drive.
No matters if the ignore duplicates is checked or not, most of the images are grayed in the central grid, without the checkbox needed to import it or not. Checking and unchecking the "ignore duplicates" does just nothing, there is no change at all. Any suggestion?
PS: Everything seems to work "normal" with a new, blank catalog, of course, because there is no way to find a duplicate...
-
31. Re: LR4 not identifing duplicates?
Cornelia-I Apr 11, 2012 1:29 PM (in response to sandro.franchi)I have doubts about your scheme:
It relies on importing images again. This can work if you start with an empty new catalog, and go for Import-Copy as DNG-destination date-based folders.
But for this you would need to go via metadata saved to the image files in your current catalog. And you would loose all data which cannot be saved to the file, like collection-membership, virtual copies, stacking. (Virtual copies you could first export as DNG, so you would have real copies to re-import.)
Normally to do a conversion to DNG you do NOT import them again, via "copy as DNG".
You convert to DNG via menu Library - Convert Photos to DNG, check "Delete originals after successful conversion".
I would not create a date-based FOLDER structure at all, but a date-based COLLECTION-structure, using smart collections.
(Collapse the folder panel in library module. They are less useful than collections anyway as not accessible from all the modules.)
This is fool-proof and way less work.
Remember: folders are unimportant in LR, they just need to be handy storage buckets.
Whatever you decide: do a proper backup of the current catalog and store that safely away, to have a restarting point in case of any mess.
-
32. Re: LR4 not identifing duplicates?
sandro.franchi Apr 11, 2012 2:15 PM (in response to Cornelia-I)I do agree with all you say Cornelia, thanks for your time!
-
33. Re: LR4 not identifing duplicates?
sarangiman Aug 20, 2012 10:51 AM (in response to paulphotography)I have this issue as well.
However, it affects all new photos as well, so it's definitely *not* isolated to photos that were in my LR3 catalog.
In other words, I've had LR4 for months. Last week I shot some photos, downloaded them, didn't erase the card, shot some more photos, then popped the card back into my computer today, & LR4 still shows the photos from the 'Previous Import' in the Import dialog under 'New Photos'.
Sometimes LR4 corrects this & finds the duplicates if I exit out of the Import dialog & go back in again.
But this time, it's just not working. It still lists all these photos as 'new', even though they're sitting their in the catalog, even under 'Previous Import'.
I know the philosophy is to always wipe your card, but, y'know, it just doesn't happen sometimes b/c you forget.
And then this manual comparison becomes pretty annoying.
It'd be great if Adobe prioritized this... it was fine up until LR4.