22 Replies Latest reply on Apr 14, 2016 10:04 AM by Bob Somrak

    Struggling to grasp the need for what appears to be duplicated effort when publishing.

    Mike CG

      First of all, a warning (or a plea) - I am a relatively new user.  This should probably be posted as 2 questions but they are so related, going to submit as 1.

       

      A bit of background:  I have recently gone thru the painstaking process of figuring out a disk structure to organize my ~36,000 photos.  I know some swear to avoid setting up a disk structure, but I must be old-school, I like organized files versus relying on a 'Catalog' entirely.  I also like to be able to quickly open a file browser and know where exactly I can find an image if I need to.  But that is not my point here.

      Also after much investigation, I have settled upon using SmugMug has my online sharing site, as it allows for nested albums / structure much like what I have on my drive.

       

      First a quick overview of how I have my images structured:

       

      Albums (main folder)

      • Adventures
        • 2016-01-23 ~ Exploring Vanderwater Conservation Area
        • 2015-12-11 ~ Cross-country skiing in Heberdown
        • etc
      • Family
        • Anniversary's
        • Birthdays
          • Ian's 54th in 2007
          • Mike's 45th in 2014
          • etc
        • Christmas
          • 2015 at Home
          • 2014 at Parents
          • etc
        • Funerals
        • New Year's Eve
        • etc
      • Projects
        • New Back Deck
        • Kitchen Reno
        • etc
      • Scenes
        • 03-05 Spring
        • 06-08 Summer
        • 09-11 Fall
        • 12-02 Winter

       

      This has worked out perfectly, everything is organized, easy to figure out / find things.  I realize there are some areas that would overlap, for example 'Adventures' would contain family members etc - I figure that could be resolved with tags down the road and create collections.  But I digress...

       

      My struggle / question (finally).  My mission is to create the same structure on SmugMug via the SmugMug Publisher.  However, I'm finding it insanely tedious to have to make a 'copy' of every 'album' in my Folder Structure as a new Gallery in the Publisher.  This is what I was referring to as duplicated effort.  I was expecting to be able to right-click a folder and say Publish using the SmugMug Publisher.  I'm really doing everything twice:

       

      1.  Import new pictures from camera

      2.  Create folder and place images into folder

      3.  Select all images in new folder

      4.  Create gallery and include new images

      5.  Publish

       

      Is there a faster way?

       

      And then - bonus question.  If I, later on down the road find a new image and decide to toss it into one of my already-existing folders, how do easily find it and know I need to also add it to the corresponding Published Album?  Are there any plugin's available to help ease this?

       

      Sorry for the run-on here - but any observations, insight or best-practices would be much appreciated!

       

      Thanks,

      Mike

        • 1. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
          zoladd Level 1

          Hi Mike,

           

          You can make some easy adjustments to your process by:

           

          a) Combining your steps 1 and 2 and importing your photos using Lr Import with Copy and creating the folders (if needed) using Lr at the time of import.

          b) Combinging your steps 3 and 4 by creating Smart Galleries based on folder(s).  If in future you add photos to a folder the smart galleries will pick them up automatically if you used step a) above.  If you did not import them using Lr then you still get the Smart Gallery benefit but only after you add them to the Lr Catalog some other way.

           

          Smart Galleries are similarly useful if you decide to create galleries of family members etc., or have a need to have photos appear in multiple galleries.

           

          I can see why you have setup a disk folder structure and I can understand why you might want to place tags into image files.  But you need to realise that this places an additional workload on you to keep the external folder structures, file names, tags etc., consistent with the metadata inside Lr (or any other editing/cataloging program).

           

          Others have suggested a simple folder structure (eg date based) and do everything else as metadata inside the Lr Catalog.  This makes day to day operations more simple but at the potential cost of being more locked into Lr in future. 

          • 2. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
            Jao vdL Adobe Community Professional & MVP

            I would really advise against setting up a folder structure in this way on your hard disk. In my experience, you are generally far better off having a folder structure organized by date and using keywords, collections and smart collections to organize them in a higher level and more flexible structure. The problem really is that you don't want to duplicate files on your hard disk and you run into trouble when one file really belongs in multiple categories. A simple folder structure organized by date is far more flexible if you then organize using keywords and collections/albums. Invariably a folder structure will get you stuck at some point. This is the exact reason why libraries employ a simple cataloging system to organize the books on the shelves and have indices (now all computerized of course) to allow you to find them and to group them according to different criteria.

             

            Within Lightroom you can generate a simple nested collection solution and use smart galleries in the smugmug plugin to refer back to your nested collections or simply generate the nested collections directly within the publish services side panel. That way they will always be mirrored inside Lightroom and on smugmug and you don't have to think about it. Bonus is that you can create much smarter galleries on smug mug this way where your image shows up in multiple places. You could have all pictures of a person for example show up in their own gallery and those same pictures show up in the events galleries (e.g. Christmas 2015).

            • 3. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
              trshaner Adobe Community Professional & MVP

              Mike CG wrote:

              A bit of background:  I have recently gone thru the painstaking process of figuring out a disk structure to organize my ~36,000 photos.  I know some swear to avoid setting up a disk structure, but I must be old-school, I like organized files versus relying on a 'Catalog' entirely.  I also like to be able to quickly open a file browser and know where exactly I can find an image if I need to.

              As long as you just "view" the image files from outside LR this may be a benefit. The problem is using subject named folders for organizing files makes it very tempting to do some of that "organizing" from outside LR (i.e. Bridge, Finder, Explorer). Doing so will cause LR to "lose" the files & folders inside the Library module and require manually reconnecting them in the LR catalog file. Files & folders that have been imported into LR should only be renamed, moved, or deleted from inside LR. Doing any of this from outside LR is going to create very big headache. You may already be aware, but we all forget from time to time and do it anyway.

               

              If your concerned about "finding" specific images from outside LR use keywords, ratings, labels, EXIF & IPTC metadata to organize the images. LR can be setup to write this metadata to XMP sidecars for raw files and directly into JPEG, TIFFs, etc. The only potential downside is that any metadata change or addition will flag the file as having been changed. This will cause backup applications to "update" every image with metadata changes to the backup storage location(s). XMP files will not take very long since they are relatively small, but large non-raw files (no XMP sidecars) will take considerably longer. Many LR users have requested the option to use XMP sidecars with non-raw files for this very reason, but to date it has not been implemented. More info here:

               

              XMP Sidecar Files « Julieanne Kost's Blog

               

              Peter Krogh has a good eBook on the subject: Organizing Your Photos with Lightroom 5 - The DAM Book

              • 4. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                dj_paige Level 9

                A couple of comments in addition to the very good advice above:

                 

                I also like to be able to quickly open a file browser and know where exactly I can find an image if I need to.

                As long as you want to view the unedited photo, file browsers work. And I guess I don't really see the need in my work to be able to go to a file browser and view the unedited photo, if I want to see the photo, I want to see the edited version. Why is viewing the unedited photo via file browser necessary in your work? If you want to view the edited photo, or actually use the edited photo, Lightroom is required.

                 

                While I understand that it is tedious to recreate this folder hierarchy as Publish Collections, I don't see any way around it. I don't think that as ever the point of the Lightroom Library module, to set up meaningful folder hierarchy and then publish the contents of the individual folders; and so you are essentially trying to do something that the software was never designed for. Maybe this plugin will be helpful here (or maybe not, I don't know). Jeffrey Friedl's Blog » Jeffrey’s “Folder Publisher” Lightroom Plugin

                 

                With regards to zoladd saying you want "Smart Galleries", this will save you some time in the case where you add photos to a folder (except that the actual Lightroom term for this is "Smart Publish Collection")

                • 5. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                  Mike CG Level 1

                  Thanks everyone for your replies - lots of good information and cases for not having the disk structure set up as I have it.  I agree fully it is too much to maintain.  My needs to grabbing something out of it are too few and far between to make it worth it.

                   

                  I think my 'need' for a directory-based structure comes purely from fear of a 'virtual' structure only and 'what if something goes wrong' with it.  Catalog backups, yes for sure...  but still.  Whereas a directory structure always means something, no matter what tool you look at it with.

                   

                  I think this topic of "Disk Structure" vs "Virtual Structure" has been debated a lot and as much as it scares me, I think going forward a virtual structure is probably the best way to go.

                   

                  Now being the OCD type person I am when it comes to files and data, I'm going to probably go insane trying to switch my collection around.  I won't be able to stand having part of it like it is now and all stuff going forward in a YYYY\MM-DD type folder setup - for me it has to be all or none.  lol   (I am my own worst enemy when it comes to creating headaches).

                   

                  So, where does one start?  I already have a whole bunch of stuff uploaded to SmugMug - but just as regular galleries, not Smart Galleries.  Will all that have to be erased and then re-uploaded or will re-org'ing stuff be able to re-sync those photos and save me the upload pain?  (I am on a very slow Internet service which often gets < 1Mbps upload speed).

                   

                  But aside from the uploading issue...   is there a (fast / easy) way I can 'Keyword' all my organized photos with the current Folder name?  It would be a great start to re-creating what I have virtually as a start - and from there I could easily add more keywords for better organization.  I have written a program that will move all the files into a date-based structure for me - and I guess I can re-sync the Catalog after (or something).....   but I won't do that until I know what I am doing for certain.

                   

                  So, any recommendations / steps on how to accomplish this without destroying the world?

                   

                  I have 626 folders in my structure now - hopefully there is some automation available.  Please say yes there is.

                   

                  And - if there are a lot of 'Smart Collections' set up - either in the main Collections section and/or the Publishers section - does that kill performance of Lr?

                  • 6. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                    dj_paige Level 9

                    Despite the advice given above, which I agree with, that I believe applies strongly to people who have no or little organization before their first use of Lightroom, that does not apply to you. Since you have already created the folder structure, it seems pointless and silly to eliminate this portion of your organization and replace it or duplicate it with keywords. Use the capture date folders for future imports and of course, add keywords and metadata to those future imports.

                     

                    If you already have a structure (in this case folders) that allow you to find photos by certain criteria, there is no need to also have keywords that allow you to find the photos by the corresponding keyword for that folder. This is truly unnecessary duplication. Unless the keywords are needed for both existing photos and future imports.

                     

                    Since Publish collections must be created one-by-one, as far as I know, there is no other choice at this time for creating them, it is duplication of effort in a certain sense.

                    • 7. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                      Mike CG Level 1

                      I agree it seems like a lot of effort, however, it will drive me crazy just knowing I have this in my Photos directory:

                       

                      2016

                      • 01-01
                      • 01-02
                      • etc

                      Adventures

                      • 2016-01-23 ~ Exploring Vanderwater Conservation Area
                      • etc

                       

                      Given that I am OCD (or CDO if you alphabetize it) - I'd like to take the painful effort up front and just 'undo' the organization physical structure and somehow re-create it virtually (as a first step).

                       

                      So creating Collections is manual...  ok, I can probably live with that.  It will take some time one-by-one, but I guess that is a tolerable one-time event.

                       

                      How about somehow marking my existing 'folders' with a marker / keyboard so that I can easily re-select them into Smart Collections after the existing structure is 'normalized' to dates?  Anything to help with that?

                      • 8. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                        Bob Somrak Level 5

                        Jao vdL wrote:

                         

                        I would really advise against setting up a folder structure in this way on your hard disk. In my experience, you are generally far better off having a folder structure organized by date and using keywords, collections and smart collections

                        Why do you need to have a folder structure organized by date.  You can filter easily for capture dates or ranges of capture dates in Lightroom. 

                        • 9. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                          Jao vdL Adobe Community Professional & MVP

                          Of course you don't need to but it makes for very easy moving of folders to external hard disks when your internal fills up. That's really the only reason I do it.

                          • 10. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                            Mike CG Level 1

                            Hi Bob, at this point it is just for a matter of consistency on the drive itself.  I don't want to have part organized (as noted above) up until 'today' and then future images being in a flatter date structure.  I'd like to have the same drive structure for everything and handle a structure virtually within Lr.  That way, when I import, the files automatically fall under the right place - and I handle them via collections.  The issue now is how to sanely go about the transformation from one to the other.

                             

                            PS...  I never remember when I took a picture so ultimately the dates alone would do me no good.  Hence my folders right now have meaningful titles.

                            • 11. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                              dj_paige Level 9

                              Hi Bob, at this point it is just for a matter of consistency on the drive itself.

                              In my opinion (but maybe not yours), this is a false goal. One that causes extra work. One that causes extra work that adds no benefit (yes, I understand if you are OCD then maybe the "neatness" is a benefit, but it really isn't, in my opinion). The goal for me is to be able to find my photos. The goal for me is NOT to be able to find my photos and simultaneously have perfect consistency in my folder hierarchy. My disk hierarchy is inconsistent. Some photos were originally put into folders named by subject. Then photos named by capture date, in format 1, and then later using different software, photos by capture date in format 2. And I can find any one of my photos quickly, because I don't search by capture date or folder name, I search by keywords and other metadata, where I do have great consistency.

                              PS...  I never remember when I took a picture so ultimately the dates alone would do me no good.  Hence my folders right now have meaningful titles.

                              Again, in my opinion, you have a misunderstanding. The point of using capture date folders is not to enable you to find the photos easily. No one can remember the dates of thousands (tens of thousands) (hundreds of thousands) of photos. The capture date folder names are simply for storage. Because its not a good idea to put all your photos into a single HUGE folder. The folder names by capture date are not there to enable you to find the photos. The metadata, and descriptive folder names (like you have now), enable you to find your photos. There's nothing wrong with a combination, as long as you can find your photos. Which, in my case, has been proven, I have a combination of folder naming schemes, but great consistency in how I apply metadata.

                              • 12. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                trshaner Adobe Community Professional & MVP

                                Mike CG wrote:

                                 

                                But aside from the uploading issue...   is there a (fast / easy) way I can 'Keyword' all my organized photos with the current Folder name?

                                John Beardsworth wrote a script to do this Copy Mac Finder paths to keyword hierarchy? | Lightroom Forums. If you need more information I suggest contacting him via his website: Lightroom etc – John Beardsworth

                                 

                                It would be a great start to re-creating what I have virtually as a start - and from there I could easily add more keywords for better organization.  I have written a program that will move all the files into a date-based structure for me - and I guess I can re-sync the Catalog after (or something).....   but I won't do that until I know what I am doing for certain.

                                 

                                As I mentioned in my reply #3 this will create a HUGE MESS inside LR that will be very difficult to "re-sync." Even if you could find a LUA script to do this from inside LR I would advise NOT using it! Why? When "moving" a large number of files or folders on disk from inside LR (or even outside LR) files can become corrupted or lost. There have been numerous reports in this forum of users moving entire folder structures only to find out some or all of their files have been deleted. Things can go wrong for numerous reasons beyond the scope and subject of this post!

                                 

                                So what's best for you? IMHO leave the existing subject named folder structure as is, add the directory path as keywords for these files inside LR using John Beardsworth script, and then add the same associated keyword types to new files going forward on Import into LR, but use a date named folder structure. You will then be able to use a keyword search to find the files in both the subject named folders and date named folders. It won't look neat & tidy in Finder, Explorer, and the LR Library module Folders panel, but you will be able to locate ALL files with the same keyword search criteria.

                                 

                                I'm not sure how this impacts your Smugmug gallery going forward. Perhaps someone else can provide suggestions.

                                • 13. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                  Mike CG Level 1

                                  Yes, it truly is a false goal - but as I said, I'm OCD about my file system.  Always have been as I know once you leave a 'mess' (in my opinion) - it tends to get real ugly fast.  So, while there is no benefit from it, it is consistent and clean.

                                   

                                  No misunderstanding on the date file structure, I know not to put all my eggs in one basket - and I do have a running backup and I know by filing by date has no value for finding things.  I plan on updating the metadata of the records based on their structure prior to re-organizing the files themselves.

                                  • 14. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                    Mike CG Level 1

                                    Well, that is worrisome then.  So, what I'm thinking instead - forget about doing this in Lr.  I have some idea of code I can write myself to add keywords to my files based on my current file structure and can move the files using the OS via this same code into the final date-based structure.  (I will just copy it all to a new location first so I have the original if things go south).  Additionally, I found some material on the actual Lr catalog - it might be possible to programatically (not using LUA) to update the catalog itself and automatically create my collections.  Going to look into it...

                                     

                                    Either way - I can just re-import the final disk structure as a new catalog at the end.  I don't have anything in there except the organization.  The SmugMug plugin appears to be able to re-sync the photos - will worry about that after though.

                                    • 15. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                      dj_paige Level 9

                                      Mike CG wrote:

                                       

                                      Yes, it truly is a false goal - but as I said, I'm OCD about my file system.  Always have been as I know once you leave a 'mess' (in my opinion) - it tends to get real ugly fast.  So, while there is no benefit from it, it is consistent and clean.

                                      Well I don't have OCD, but organizing in another way (keywords and metadata) cleans up the mess. There is no mess. Your photos are perfectly organized and perfectly find-able.

                                       

                                      I point out that my inconsistent file system does not "tend to get ugly real fast", it remains in the same inconsistent state as it was years ago, and is in no way worse than it was before. You almost make it sound like the inconsistent file system gets worse on its own, which is simply not true. It only gets worse if you choose to do something that makes it worse.

                                       

                                      On the one hand, you are frustrated by the work you will have to do to make your Publish Collections match your folder structure, you call it a duplication of effort; yet on the other hand you are willingly going to do a lot of work to revamp your entire folder system even though it's not strictly necessary. It's your time, it's your decision, but if it was me, I wouldn't do it.

                                      • 16. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                        trshaner Adobe Community Professional & MVP

                                        Mike CG wrote:

                                        Either way - I can just re-import the final disk structure as a new catalog at the end.  I don't have anything in there except the organization.  The SmugMug plugin appears to be able to re-sync the photos - will worry about that after though.

                                        That is the ONLY way I would advise doing it. Just make sure all of your folder hierarchy names have been added as EXIF keyword metadata before moving or renaming any folders.

                                        • 17. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                          PatGarrett Level 1

                                          I too recommend using dates as the HDD folder organization.  Here's why:

                                           

                                          1) backup - when folders are date named RESTORE is easy (so is backup). I adopted digital imaging back in the late 90's and have never lost an image in spite of several HDD crashes.

                                          2) KEYWORDS on import (easier? than changing the target of the import) should be routine. The strategy to accomplish what you want is to assign KEYWORDs during import that are like your proposed structure.  Smart Collections can then auto-populate collections that mimic your proposed structure. IMPORTANT FACT - keywords, and stars can be seen in FINDER and EXPLORER, so all your structure work is maintained EVEN if you decide to leave LR.

                                          3) Sooner or later you'll need multiple HDD to store your photos.  I put a physical label the external drives that shows the dates of the files on the drive. So when I use LR and desired photo is off-line I know exactly which drive contains the file.  Which drive has families, projects, etc. just doesn't work easily.

                                           

                                          • 18. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                            PatGarrett Level 1

                                            If you have multiple external HDD organizing by dates makes segregation of images easy. The problem with using names is that the size of the named group is variable. Projects might span multiple drives. When dates are used, just cut-off when the disk is nearly full (I never fill a drive 100%... I target 80=90%)

                                            • 19. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                              Bob Somrak Level 5

                                              1). How does date folders make backup/restore any easier than any other naming convention?

                                               

                                              . IMPORTANT FACT - keywords, and stars can be seen in FINDER and EXPLORER, so all your structure work is maintained EVEN if you decide to leave LR.

                                               

                                              Does finder and explorer read the RAW xmp data?  I don't have Lr here to try it on.

                                              • 20. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                                trshaner Adobe Community Professional & MVP

                                                In Windows 7 using a raw file with XMP and 'Save Metadata to File' I can see LR Ratings, but no Keywords.

                                                • 21. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                                  Bob Somrak Level 5

                                                  Thanks Todd.  Do the keywords show on Jpg an Tiff using Save Metadata to File?  I don't have a REAL computer handy, using an Ipad

                                                  • 22. Re: Struggling to grasp the need for what appears to be duplicated effort when publishing.
                                                    PatGarrett Level 1

                                                    It's easier because all the files are stored together and backup/restore can be done with a simple copy.

                                                     

                                                    Those features (stars and keywords) don't show in RAW files in Finder/Explorer, but do in JPGs, so I stand corrected.  However, I do use the features frequently when I export large numbers of JPGs for printing, or posting on the web.