7 Replies Latest reply on Feb 18, 2018 2:49 PM by Jerry and Lois

    Keyword: Collection of projects spanning over 100tb, 20 external hard disks

    Jerry and Lois Level 1

      Hello!

       

      While we've been shooting professionally since 2004, we have neglected a critical process in our workflow and need to correct this.

       

      We have some 20 or so external disks, spanning some 100tb, each completely filled and dedicated to the storage of all the photo projects we've ever shot.  Some of our event projects are quite massive, with well over 25,000 images from the capture (=> 50,000 NEF+JPG combos), spanning hundreds of gigs.  And that's before we ever produced finished images. 

       

      The challenge is that we've never keyworded our images.  We can find projects easily enough (info contained in an "intelligent" spreadsheet), but we need to get our act together.

       

      We subscribe to the Adobe "Photographer" package, but have always used Photo Mechanic in our workflow, not LR.

       

      My understanding has always been that LR had limitations on the total number of files it can catalog.

       

      Can anyone help with:

       

      1. Can LR handle a very large inventory of projects and files, stored across 20 disks?  We only keep one or two disks active at any given time (all are external USB), the rest are safely stored away until needed again.
      2. If LR can handle this, how to install/configure it for the environment described in #1?  Do we need another stand-alone disk just to support whatever LR's cataloging info may require?
      3. Is LR the proper choice to keyword all these images?
      4. If yes to #3, what's the best practice to start keywording this huge number of files?
      5. Any instructional video tutorials that you can recommend?


      I know - massive work effort, but needed.

       

      Many thanks in advance for your insights!
      Jerry

        • 1. Re: Keyword: Collection of projects spanning over 100tb, 20 external hard disks
          Just Shoot Me Adobe Community Professional

          LR has no limit on the number of image files it cam record in the catalog, Database, file.

          • 2. Re: Keyword: Collection of projects spanning over 100tb, 20 external hard disks
            johnrellis Most Valuable Participant

            With that many photos, you might consider automatic keywording.  It won't be nearly as precise as human keywording, but it will be much, much faster.  As a practical matter, you might get something useable from automatic keywording but never find the time to do manual keywording. 

             

            The new LR CC can automatically identify the content of your photos, but you have way too many photos for the current capabilities of LR CC.  For LR 7 (Classic), you might take a look at Excire or my Any Vision plugin, which uses Google's image-recognition technology.

            1 person found this helpful
            • 3. Re: Keyword: Collection of projects spanning over 100tb, 20 external hard disks
              Jerry and Lois Level 1

              johnrellis, thanks for the quick feedback.  I've never spent any significant time with LR, and always found myself uninstalling it.

               

              1. Do you have any best practices/recommendations on a "keywording 101" you'd recommend?  Our primary goal here is simply to make our image search far more efficient than relying on the aforementioned spreadsheet and our own memories.

               

              2. Any suggestions on how to configure LR (CC), in terms of where it creates/maintains its database?  We've dedicated a separate external drive for helping with this.  Hoping that it won't try to make duplicates of all our image files - only thumbs.

               

              3. Any way of calculating how large the db will be?  Does LR simply generate thumbnails for every image it becomes aware of?

               

              4. Any issues with LR dealing with external drives?  I won't have all 20 of them plugged into my PC at one time, in fact, probably only one at a time.

               

              Thanks again!
              Jerry

              • 4. Re: Keyword: Collection of projects spanning over 100tb, 20 external hard disks
                john beardsworth Adobe Community Professional & MVP

                #1 There are lots, and Peter Krogh's The DAM Book is the leading one (the 1st or 2nd editions are fine) and is more about the basic principles of managing photo collections - principles that you then apply in LR.

                #2 You can put the catalogue (and its previews or thumbnails folder) wherever you want. It won't duplicate your files providing you import with the Add option, which records the photos where they are.

                #3 Not really. It will try to create thumbnails for every image.

                #4 No problems. In fact, maybe the opposite. Unlike PM, LR records details of photos stored on EHDs that aren't attached or powered up.

                 

                It strikes me that you need to learn a bit more about using LR before you focus on keywording best practices. Remember that LR isn't just a keywording tool - it is a combination of cataloguing, adjustment, and output, all in one integrated package. In many cases, an entire shoot can be processed, tagged with keywords and other metadata, and then printed or exported entirely within LR. Keywording is just one feature. LR is designed to be the hub and you benefit most from working with its design, not fighting it.

                 

                In cases like yours, I often encounter a sort of analysis-paralysis where the size of the total task means people don't start doing any keywording at all. Or they think they must buy external keyword lists ("controlled vocabularies") and then struggle with thousands of irrelevant keywords. Start small, maybe on current jobs, gain experience, gather speed, realize you can be more ambitious. The journey of a thousand miles begins with a single step....

                 

                Don't think keywording is a one-time exercise. It's perfectly normal to add a few keywords, then realize another term describes the pictures and add it too. And only add as many keywords as you need - don't make perfect the enemy of the possible. Some other thoughts here.

                 

                I agree with the comment about automated methods. My Search and Replace plugin can copy fields like title over to keywords (often metadata is in more than one place) or it can take folder names and make those into keywords, for example.

                 

                I'd suggest your strategy should be:

                1. Import all your photos with the Add option - get your entire picture collection under control in one place and start using LR as the place where you manage your photography
                2. Start keywording small current jobs (and don't forget other fields like title, caption, job)
                3. Put your effort into actually applying the keywords to photos, not into moving keywords into neatly-structured hierarchies - save housekeeping stuff like that for later, if at all.
                1 person found this helpful
                • 5. Re: Keyword: Collection of projects spanning over 100tb, 20 external hard disks
                  Jerry and Lois Level 1

                  @john beardsworth:

                  Thank you so much for your time and information.  This gets my basic concerns out of the way (and yes, analysis-paralysis is an old friend ).  I needed to get the foundation set correctly before launching in.

                   

                  For DAM, it's been a long time since I took a token look at that, and definitely need to review.

                   

                  For our workflow, we have a fairly efficient system - albeit lacking with the keywording parameter.  But, review is always a good thing.

                   

                  Your suggested strategy is great, and parallels what I'd been planning on, once I had the basic concerns about EHDs and catalog limitations addressed..  We've used Photo Mechanic in lieu of LR for years on a project by project tool, and it's worked great for us.  But not for a global all-projects manager, which is what I hope to migrate us towards with LR.

                   

                  Thanks again!

                  • 6. Re: Keyword: Collection of projects spanning over 100tb, 20 external hard disks
                    Conrad C Adobe Community Professional & MVP

                    You've gotten a lot of good advice already, but here are a few more bits to fill in the answers:

                    https://forums.adobe.com/people/Jerry+and+Lois  wrote

                    1. Can LR handle a very large inventory of projects and files, stored across 20 disks?  We only keep one or two disks active at any given time (all are external USB), the rest are safely stored away until needed again.

                    This should not be a problem. Lightroom can easily catalog multiple volumes, connected and unconnected. When a disk is not connected, you can still enter metadata for those images even though you'll only be able to see the thumbnails and any generated previews for those images. When you connect that disk, the images become fully available for all types of edits.

                     

                    You know how you see different folders at the top level of the Lightroom Folders panel? If you were to catalog your 20 disks, the 20 volume names would be listed at the top level of the Folders panel. I have one Lightroom catalog tracking images on multiple internal and external drives, loose USB stick drives, and even on other computers over the network. Works great.

                     

                    You said you have 25,000 images. That's considered a small catalog to many of us here. My personal catalog has almost 4x that many images, spanning many digital images and scanned film images.

                     

                    https://forums.adobe.com/people/Jerry+and+Lois  wrote

                    2. If LR can handle this, how to install/configure it for the environment described in #1?  Do we need another stand-alone disk just to support whatever LR's cataloging info may require?

                    No - I hope it's clear from what I said that Lightroom can catalog all of your images where they stand. There is no reason to move anything to different drives, no need to consolidate them to one drive. Plug in a drive, start the Import dialog box in Lightroom, select the Add option (this is critical!) so that files are left where they are and simply are recorded at that drive and folder location (not copied or moved). They'll be added to the catalog, clearly marked in the Folders panel as being under that specific drive. Then do the next drive, and so on.

                     

                    https://forums.adobe.com/people/Jerry+and+Lois  wrote

                    3. Is LR the proper choice to keyword all these images?

                    It can be, for two reasons. Lightroom provides so many ways to enter keywords that even if you don't like two of the ways, you might like the other three ways. Some people prefer to do it with the mouse, so there are several ways to add keywords using point-and-click methods. Some people are keyboard-oriented, so there are several ways to add multiple keywords by typing, including an auto-complete feature. You can configure panels of preset keyword buttons that can also be applied with keyboard shortcuts.

                     

                    The other reason Lightroom can be a good choice is that it's standards compliant like Photo Mechanic, so every image you keyword can carry that metadata to other applications. If you export a keyworded image, you have control over whether the keywords are included. If you upload an image containing keywords to any website that supports them, like Flickr, they'll show up there. Images containing embedded keywords can even be found using the system search features on Mac and Windows.

                     

                    https://forums.adobe.com/people/Jerry+and+Lois  wrote

                    4. If yes to #3, what's the best practice to start keywording this huge number of files?

                    5. Any instructional video tutorials that you can recommend?

                    The best way to start is to closely study all Lightroom keyword features. Try the Keyword section in the book Adobe Lightroom Classic CC – The Missing FAQ by Victoria Bampton. Understand which ways of entering keywords you would prefer to design your workflow around.

                     

                    Peter Krogh, already mentioned, has a number of good books and videos, such as The DAM Book, 2nd Edition. DAM stands for Digital Asset Management, which is more about best practices for managing a large digital archive with Lightroom, not just about keywording.

                     

                    Depending on what your photos are about, automatic keywording and face recognition might help, but not always. If your subjects are too obscure or personal, automatic keywording won't know what they are. If your photos aren't of people, face recognition won't help.

                    1 person found this helpful
                    • 7. Re: Keyword: Collection of projects spanning over 100tb, 20 external hard disks
                      Jerry and Lois Level 1

                      Conrad C:

                      Thanks for the additional comments!  This helps tremendously, in addition to previous info.

                       

                      FWIW, I've not used LR enough to be familiar with it.  I've installed it a few times, but becuase I didn't RTFM nor set up preferences (e.g. "Add"), it was more of a pain than a utility for us.  We've been using Photo Mechanic and PS since about PS 3.0 or so.

                       

                      One clarification:  We don't have "just" 25,000 images total.  We have some 30 or so projects each containing that many images, just in the RAW filesets alone (think very large, all-inclusive beauty pageants, 2 cameras running non-stop).  In just the dog world (where we started shooting back around 2000 or so), we now have roughly 700,000 raw images.  And since we've been shooting FT since 2004, the overall number of captures is in the millions spanning about 100tb of collective disk space on dedicated EHDs. 

                       

                      Hence the concerns about catalog or database limitations (with respect to upper number of files allowed, db performance, etc.)

                       

                      We've also shot many tens of thousands of images in the rock and roll (performance, portraiture, etc.) and celebrity worlds where keywording is probably a good thing again .  I'm guessing that "Roger Daltrey", or "Jimmy Fallon" are good things to tag.  It'll be interesting to see how/if the automated tools handle some of the more recognizable individuals!!

                       

                      We have almost never uploaded our images to other photo-based websites (e.g. Flickr), so this whole keyword thang wasn't an obvious issue until now; we were happily self-contained.

                       

                      One of the ultimate goals for us - we're gearing up to completely retool our website, and hope to have keywording in place (and migrated to the ALT fields) for SEO.  But for now, baby steps - just learning the process first.

                       

                      Again, very very appreciative on your time and clear information!!!

                      Jerry