Help restoring original photo library after accidental deletion

[DaveDuprePhoto]

I have a Lightroom catalog with over 38,000 photos and videos in it. I have it backed up twice, but recently had a perfect storm that resulted in me accidentally deleting all the original photos. Both of my backups failed, so I was cleaning off a disk to make room to make a copy of everything before I setup my backups on new media again. I screwed up and deleted my main photos folder (this is on a Mac). Cue lots of colorful metaphors and cries to have cloud backup.

I immediately quarantined my external drive so nothing changed on it, and I used a recovery tool to try and get things back. Of my 38205 files in my original catalog, the tool restored 34821 file with their original file name and folder location.  It also found over 40k raw files, where raw files do not have the original filename or folder location (these are not raw files in the photo sense). Only the file content is there. Looking around, it looks like I have just about all the files I need, but I'm now looking for some advise on how best to rebuild my catalog and relink all the files.

My questions:

After I put back the 34821 restored files, I will have two issues. First, there will be a lot of missing files which I'm hoping to find in the raw file content  from above. Second, since this process restored deleted files, there will be a lot of files in the folder that are not in the catalog.

1. Does Lightroom have a way to locate files in the main photo folders that are not in the catalog? I know I can just ignore them, but the perfectionist in me will struggle with that.

2. How best can I restore all those missing links? I have most of the original content, but the filenames are all changed. My original thought was to import all those renamed files into a new folder and hope that Lightroom will clear out all the duplicates. That would at least limit how many files I need to search for the missing photo.

Thanks!

[john beardsworth]

DaveDuprePhoto  wrote

Second, since this process restored deleted files, there will be a lot of files in the folder that are not in the catalog.

You mean files that you had deliberately deleted?

DaveDuprePhoto  wrote

1. Does Lightroom have a way to locate files in the main photo folders that are not in the catalog? I know I can just ignore them, but the perfectionist in me will struggle with that.

What you can do is synchronize the parent photo folder and import those files. They will then all be shown in Previous Import. However, be careful with synchronizing the folder - it's not a way to dig yourself out of a hole and can often worsen the problem.

I would make certain to run a backup at each stage of resolving this, just in case you make another mistake.

DaveDuprePhoto  wrote

2. How best can I restore all those missing links? I have most of the original content, but the filenames are all changed. My original thought was to import all those renamed files into a new folder and hope that Lightroom will clear out all the duplicates. That would at least limit how many files I need to search for the missing photo.

What filenames does the original content have? Is it using the filename from the camera? If so, is the original filename shown in the catalogue's Metadata panel? Test this with a few images at first. If so, I can let you have a script that puts a dummy file in the missing files' place, letting you rename the dummy files back to the original filenames - you then copy the original files over the dummy files and LR will now be pointing to the real files with their original names. But before you grab this lifebelt, answer my questions

[DaveDuprePhoto]

Yes, the files I deliberately deleted from Lightroom while reviewing a bunch of new photos. This would have been before I screwed up and emptied my trash.

The original content basically has random filenames. Since the directory metadata is all gone, my recovery gives the files names like 207780854-207796083001.JPG or CANON_281299472-2813005971026.JPG. Note, the vast majority of the lost files are JPGs from my early days before I had a DSLR, or they are scanned negatives/slides. I think the early files got clobbered in the directory info as part of moving things to the trash.

Guess what I just bought today? Yes, a cloud backup solution. Obviously, a TimeMachine backup to a NAS and a synced second HD are not enough to protect me from myself given the TimeMachine backup decided it needed to start over, and the second external disk died at the same time.

[dj_paige]

After I put back the 34821 restored files, I will have two issues. First, there will be a lot of missing files which I'm hoping to find in the raw file content  from above ... Looking around, it looks like I have just about all the files I need, but I'm now looking for some advise on how best to rebuild my catalog and relink all the files.

If you do it right, there won't be any missing files. Just put them in the exact same folder they were in before the deletion. If your backups mirror the folder structure of the originals (and they should), then this is a trivial thing to do, takes you a minute or two and everything is ready to go. No rebuilding the catalog, no re-linking, no importing.

[DaveDuprePhoto]

There will be missing files. My original /Photos folder had ~38000 files in it according to Lightroom. My restored /Photos folder has ~35000 files in it. Lightroom should automatically find the 35k files I restored. Then I will be left with a bunch of files I explicitly deleted -- Lightroom doesn't know about them, and I will have a bunch of files whose filenames and original locations are unknown. I will need to find them by image and capture date. It might be worth writing a script given that capture date to the second should be close enough to match most any image I have. That's what I'm hoping anyway.

[DaveDuprePhoto]

I think I have a way to remove the already explicitly deleted files. If I do an import/add from my main /Photos directory, and I ensure that duplicates are not imported, I should be able to add a keyword to all the added files. Once things are imported, I can view the keyword, and delete the files (again).

I can't try it yet though. My files are still copying from my restore disk.

Regarding the script I mentioned above, I'm a software developer so scripting is not scary for me. However, I'm not aware of tools that will allow me to script against a Lightroom catalog. Are there any?

Edit: Looks like I could reverse-engineer the SQLite database that is Lightroom's catalog, or I figure out how to write my own plugin to do this. Any other options?

[john beardsworth]

In general, I would leave the database alone. It's horrid.

Lightroom has an SDK - it's in Lua. See the attached which does what I described earlier - copies  a dummy file (save a file called dummy.jpg on your desktop) into the location of each missing file so you can then restore the original filenames and replace the dummies with the real files. It may not help in your case, but would give you an idea or two.

John

--[[

Fixes where user has deleted files renamed in catalogue but has copies with original filenames

Copies dummy file into missing file locations

You then rename the dummy files in LR using the original filename

Finally, in Explorer/Finder, delete the dummy files and put the unrenamed files in their place

SETUP INSTRUCTIONS

Temporarily save this file onto your desktop and call it "YOURFILENAME.lua".

Now you need to create a scripts folder:

  In Lightroom, choose Lightroom > Preferences (Mac OS) or Edit > Preferences (Windows).

  Choose the Preset tab and select Show Lightroom Presets folder.

  Create a folder in the Lightroom folder called Scripts.

  Copy the "YOURFILENAME.lua" script into the Scripts folder.

  Quit and reopen Lightroom.

You should now see a little scripts menu to the right of the Help menu. It has a single item Title case keywords.

--]]

------------------------------DON'T CHANGE BELOW

local LrPathUtils = import 'LrPathUtils'

local LrFileUtils = import 'LrFileUtils'

local LrTasks = import 'LrTasks'

local LrDialogs = import 'LrDialogs'

local SEP

if MAC_ENV then SEP = '/' else SEP = '\\' end

LrTasks.startAsyncTask( function()

srcPath = LrPathUtils.getStandardFilePath( 'desktop' ) .. SEP .. "dummy.jpg"

if LrFileUtils.exists( srcPath ) ~= "file" then

LrDialogs.message("Create a dng called dummy.dng and put it on your desktop", srcPath)

return false

end

local catalog = import "LrApplication".activeCatalog()

photos = catalog:getTargetPhotos()

filePaths = {}

for i, photo in ipairs(photos ) do

if photo:checkPhotoAvailability() == false then

destPath = photo:getRawMetadata('path')

LrFileUtils.copy( srcPath, destPath )

end

end

  LrDialogs.showBezel("Finished", 5)

end)

[DaveDuprePhoto]

Thank you John! I will give it a try.

Dave

[DaveDuprePhoto]

So, I have a way to find all the duplicates via metadata (vs filename), and it looks like I can restore the vast majority of the missing files. I just have to write the process of copying the file to the original location and renaming it. I'm glad I converted most of my new Sony raw files to dng because it seems that the recovery software didn't know how to deal with Sony raw files. They all have pink bands. Thankfully, I still have previews for them (many are full-size.)

Am I correct in assuming that a script could pull out the largest preview it can find and make it the master image? Many of the lost images are ones I had no plan on sharing anyway (1 and 2 stars), but there are a few that are cherished family photos. I'll take whatever I can get. Especially since several are so old that the preview is about the size of the original image anyway.

[john beardsworth]

Your script would probably have to call photo:requestJpegThumbnail() and then convert the data to a file.

[john beardsworth]

Somewhere - Jeffrey Friedl? - there is a script to recover previews of missing files.

[DaveDuprePhoto]

There is one! Straight from Adobe: Script to extract previews of lost or deleted photos

[DaveDuprePhoto]

Unfortunately, it doesn't seem to work. It asks for the folder to write to, but nothing ever gets written.

[john beardsworth]

Here's the Friedl version Jeffrey’s “Extract Cached Image Previews” Lightroom Plugin . It's likely to be more reliable.

[DaveDuprePhoto]

So with a combination of a find duplicate plugin, some manual processing, and a quick script I wrote...

1. Import all the restored files that didn't have folder or filename info.

2. Find all missing photos

3. Use a find duplicate (by metadata) plugin to find all the duplicates of the missing files

4. Create a collection of file pairs: missing file, restored file, missing file, restored file, ... All the odd photos in the collection are my original catalog entries with a missing file. All the even photos are duplicates found in the restored files. These all have crazy names.

5. Run the script below. I usually comment out the modification parts the first time just to be sure.

It will be a lot of manual work creating the restoration collection, but at least I get most all of my files back.

Thank you all for your help!!

Dave

local LrFileUtils = import 'LrFileUtils'

local LrTasks = import 'LrTasks'

local LrDialogs = import 'LrDialogs'

local LrLogger = import 'LrLogger'

local myLogger = LrLogger( 'recoverLogger' )

myLogger:enable( "logfile" )

LrTasks.startAsyncTask( function()

    local catalog = import "LrApplication".activeCatalog()

    local photos = catalog:getTargetPhotos()   

    local nphotos = table.getn(photos)

    myLogger:trace(nphotos .. ' photos')

    for i = 1, nphotos, 2 do

        destPhoto = photos

        srcPhoto = photos[i+1]

        if srcPhoto:checkPhotoAvailability() == true and destPhoto:checkPhotoAvailability() == false then

            destPath = destPhoto:getRawMetadata('path')

            srcPath = srcPhoto:getRawMetadata('path')

            myLogger:trace(i .. ': ' .. srcPath .. ' --> ' .. destPath)   

            index = string.find(destPath, "/[^/]*$")

            path = string.sub(destPath, 0, index-1)

            LrFileUtils.createAllDirectories(path)

            LrFileUtils.copy(srcPath, destPath)

            destPhoto:saveMetadata()

        else

            myLogger:trace(destPath .. ' exists!')

        end

    end

    LrDialogs.showBezel("Finished", 5)

end )