Skip navigation
Currently Being Moderated

Need advice. Is it necessary to put images in folders and subfolders with LR?

Apr 14, 2011 4:21 AM

It seems that the concept of spending hours putting files into folders, sub-folders and sub-sub-folders is not needed  with LR. Creating all these folders takes hours, especially if you are like me and need a major reorganization of image organization. My thought is that all images can go into one massive folder as long as they are properly tagged. I can then use LR Smart Collections to essentially virtually organize all my images. I can create many more sorting options using Smart Collections than I can by putting images into a folder. This flies in the face of everything I've learned about storing data on a computer so I'm nervouse to head this direction because it starts with unorganizing my folders and removing all tags so I can start fresh.

 

I don't want to take this on only to learn I didn't think of a gotcha issue. One drawback is that you are now committed to find anything by using LR or tags only. What do you think?

 
Replies
  • Currently Being Moderated
    Apr 14, 2011 5:23 AM   in reply to ECI-Tampa

    Hmm.. Well, let's take this a bit at a time.


    First, is it _required_ that you make sub folders to store images?  No, but depending on your operating system, you will run into problems somewhere between 8 and 13 thousand images, at least given from the problems some people have posted here.  The operating system doesn't like having that many in one folder, it causes problems (not to mention how slow accessing a folder with that many items in it will be).


    Next, is it hard to make all those folders? No, LR will make them for you on import, if you ask it (you don't even have to ask politely!).  So it sounds like you haven't done much reading on import techniques.

     

    Quite a few of us are proponants of using date to store your images, keywords and collections to sort them.  Here's how I do it:

     

    Major_project->year->year_mon->year_mon_day

     

    In the expanded import window, over on the right side for destinations, you can select the method LR uses to create the path.    These sub folders get made automatically, no interaction on my part (other than selecting the major_project part).

     

    Now, after import, you have it easy to find by date;  after you've keworded, you find them easilly by keywords and collections.  And you'll not run into system level dependencies like how many folders you can fit in one folder view...

     

    Anyway, I don't think very many people here would advocate one giant folder; in fact, it will be hardly anyone (there's always _one_ for anything, it seems!).  You might try a basic book,  Martin Evening's is well thought of, and so is Victoria's.

     

    Cheers!

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 5:27 AM   in reply to ECI-Tampa

    Oh, and as dj_paige has mentioned elsewhere, stay away from Kelby's book...    It has some really odd sounding recommendations, especially for newbies!


    Cheers!

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 5:35 AM   in reply to Jasonized

    Thanks, Jasonized.

     

    There are many ways to set folders up, and there is no one right way. Compared to Jasonized, I don't have a "Major Project" folder. My folder hierarchy is simpler, it is just

     

    Nikon Pictures->year_mon_day

     

    Whatever works for you. I agree that thousands of photos in a single folder may cause problems.

     

    An important concept (to me, anyway) that the original poster has already noticed is that storage can be (and in my mind should be) entirely separate from Organization/categorization/search. You set up your files in folders to meet whatever storage needs you have. You do not set up your folders to provide you with a strong system to find your photos. My folder arrangement is minimal, as you can see. I almost never look in my folders, and the only time I ever think about moving photos or folders is when I want them on a new hard disk.

     

    In Lightroom, using keywords, metadata and (optional) collections, you set up your photos for Organization/categorization/search. Using these Lightroom tools is much more powerful than anything you can come up with using only folders ... a photo can be in only one folder, but it can have an many keywords as you want and can have a huge amount of metadata assigned to the photo. Searching using either smart collections or the filter bar or the keywording panel is extremely quick once you have done a thorough job of keywording and assigning metadata.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 5:42 AM   in reply to dj_paige

    Hey Dj!

     

      So... year_mon_day..  hmm.. And after 1 year, you have 365 in that one folder... two years... 10 years, 3650 folders! Yikes!  I'd hate to try and scroll down looking for one particular day doing that! 

     

    But then, as you said, you don't go back...  I sometimes need to find images for something shot in particular, and date is best remembered (since it's logged when I shoot anyway).

     

    Cheers!

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 6:19 AM   in reply to Jasonized

    I don't shoot photos 365 days a year, probably less than 100 days a year. But your point is still a good one, I had thought of changing to Nikon Photos->Year->Year_Month_Day. Not sure I'm going to make the change, but I'm thinking about it.

     

    Your brain must work differently than mine. I never remember dates the photos were taken (unless its a holiday or a birthday). Usually, I cannot even remember the year the photos were taken. So I almost never need to find things by capture date, and if I did (Christmas pics, for example), I could use the Lightroom Filter Bar.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 6:25 AM   in reply to ECI-Tampa

    ECI-Tampa wrote:


    My thought is that all images can go into one massive folder as long as they are properly tagged. I can then use LR Smart Collections to essentially virtually organize all my images. I can create many more sorting options using Smart Collections than I can by putting images into a folder.

     

    A lot of people recommend that approach.  I don't.  What if you decide at some future time to stop using LR and start using another tool?  What if your catalog goes away?  What if (I hope not), Adobe stops making LR?

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 6:53 AM   in reply to dj_paige

    I am using the very same folder structure for quite some time now and I am very happy with it so far.

    Pictures \ Year \ Year-Month-Day

     

    .. my 0.02

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 7:00 AM   in reply to Lee Jay-ZyZk56

    I don't think many recommend that approach at all! You probably read what you wanted to read.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 7:49 AM   in reply to john beardsworth

    I have never yet seen anyone recommend putting all pictures into a single big folder

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 3:12 PM   in reply to ECI-Tampa

    Folders don't exist. They are notional things. The operating system puts files anywhere it wants on the disk, not even in adjacent sectors a lot of the time. It knows where the files are because it has an indexing system.   So, if you have a file c:\folder\pic.dng and a file c:\folder2\pic.dng they are merely different file names since what we call the path is really just a long file name. If you saved the same 2 files as c:\folder\pic.dng and c:\folder\pic2.dng that again is just another file name and it is all, the same to the operating system and its index. The files in the same 'folder' will not be placed next to one another on the disk.   It follows that putting all your files in one large folder or a series of smaller ones is all the same thing to the computer. If it has some significance in your mind, then put them where you fancy. It makes even less difference to Lightroom, of course which is a pure database not concerned with the physical positioning at all, even down to the what and where of the storage media.  There is no limit to the number of files you can put in a 'folder', of course, in spite of what some people think. This notion is a hangover from limitations of early operating systems and their memory addressing capabilities.  In terms of organising files, that is what Lightroom is for and the keywords and structure should come from those and exif. Since these are embedded in the files themselves and are pretty much standard, they will not be lost if Lightroom goes belly up. I can even read mine in several other programs I use so I have no fear of losing their organisation and structure. And with daily backups of our precious files no fear of losing the LR catalogs either.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 3:49 PM   in reply to ECI-Tampa

    I think it's also good for photographers to think beyond themselves when it comes to organizing their archive.

     

    For a professional, their archive is a business asset. When the business is put up for sale, buyers will like to see that things are organized and accessible. A buyer may or may not use LR, so a basic folder structure with metadata saved out to the files can be a good selling point.

     

    For the hobbyist, their archive will eventually be passed on to their heirs, who would be even less likely to know about LR. So again, having a simple but consistent folder stucture will be very helpful to others digging through your old pictures.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 4:01 PM   in reply to glugglug

    glugglug wrote:

     

    Folders don't exist. They are notional things. The operating system puts files anywhere it wants on the disk, not even in adjacent sectors a lot of the time. It knows where the files are because it has an indexing system.

     

    Correct, but the difference is that every tool everywhere understands that system because its so pervasive.  So there's basically no risk of loss of compatibility.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 6:15 PM   in reply to glugglug

    glugglug wrote:

     

    It follows that putting all your files in one large folder or a series of smaller ones is all the same thing to the computer.

     

    In the abstract, maybe, but each common filesystem in use today makes some strong distinctions between file and directory objects for some file operations, and for various housekeeping duties. It makes a very great difference indeed to the computer how a large amount of data is stored in a filesystem that has real-world limitations.

     

    This will change as RDBMS style filesystems become more common, but for now the abstract inode-to-data mapping is subject to certain realities and limitations.

     

    This OPs query was answered correctly already, I think. If you have no reason to use some special directory structure, just use a simple dated structure. Lr makes this stupid easy to do via an import preset. Most modern filesystems have specific limitations and caveats regarding too many file-type objects in a single directory-type object. Plainly put: you do not want thousands and thousands of files in the same directory, if you can help it. It's bad for performance, and bad for how the filesystem allocates space for file objects.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 14, 2011 10:11 PM   in reply to ECI-Tampa

    ECI-Tampa wrote:

     

    I can find a lot more images quickly using LR over 10 years of collecting images than anyone sorting through folders no matter how they are organized. Putting them on multiple drives or dvd only compounds the issues with using folders.

    I don't think having a heirachy of folders has anything to do with this..  you want to wade through 10 years of photos without folders?  simply click on the top level folder and "include subfolders".  There you are.  All in one view.

     

    And as far as I know, the only way to create backups or move images you no longer want to look at is to create something to put them in... even a DVD or tape backup is "another kind of folder" as far as the OS is concerned.

     

    Gluglug, who apparently knows just enough about file systems to figure out where the files live on a disk, but not enough to understand how a folder "file" is constructed to know it's limitations makes several points about folders that have nothing to do with the actual discussion.  Yep, files can and do live anywhere on the disk.  That level of abstraction has nothing to do with how a directory actually functions or is constructed.  There is a limitation on how many files you can put in a "folder" file, and it is different with each OS and sometimes applications accessing the OS (because of the way the OS returns data structures about the directory in question).

     

    In any event, after you've put some multiple of thousands of photos into  one folder and the OS stops showing you new images, please don't come  back and start complaining about how LR won't show you images!  That's a limitation of the operating system, not LR.

     

    Cheers!

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 15, 2011 6:16 AM   in reply to ECI-Tampa

    The bad news is that you aren't going to be the first.  I suppose the real question is not can you, but why would you want to?  Doesn't take any more effort to do both at the same time.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 15, 2011 6:49 AM   in reply to ECI-Tampa

    ECI-Tampa wrote:

     

    The idea is not to wade at all but to use tags and smart collections to do the work for you. This idea is predicated on tagging your photos correctly. If this step is skipped, you're screwed trying to go through this many images not in folders.

     

    No one is suggesting that these two things, maintaining metadata and using a simple directory structure to store managed file objects, are exclusive. They are not particularly related as, since you point out, metadata like keywords and such have nothing to do with how the data is stored as file objects. This is why it is called /metadata/.

     

    Metadata helps you slice across a massive data store easily so you can narrow down on some selection that is important to you. This is the paradigm of our times: it is easy to collect and store data, but expensive to find what you want in that data.

     

    Making our personal data stores easier to query is, of course, important. Lr gives you a lot of great tools to assist with that when it manages your photos. It is inevitable that filesystems will become more and more like RDBMS so our other data can also be managed easier. (Both Microsoft and Apple will introduce such filesystems over the next few releases, and other systems like BeOS and Linux have already had such filesystems available for some time. But we are not there yet, and there are costs associated with treating a regular inode-type filesystem as an RDBMS.

     

    This is regardless of how the data is represented to the user by the OS as it makes calls to the filesystem.

     

    The principle of KISS applies here:

     

    - Use a simple directory structure if you do not need anything more complex. A dated structure works just fine.

    - Use metadata to slice across the collection.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 15, 2011 7:17 AM   in reply to clvrmnky

    clvrmnky wrote:

     

    The principle of KISS applies here:

     

    - Use a simple directory structure if you do not need anything more complex. A dated structure works just fine.

    - Use metadata to slice across the collection.

     

    I don't use either of those.  I have a neatly organized complex folder structure, storing around 200,000 untagged images.  The time it would take me to tag those images effectively is, frankly, unthinkable.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 15, 2011 7:27 AM   in reply to Lee Jay-ZyZk56

    Find all your best images of waterfalls in Colorado, please, and then your rock paintings from all SW states. Do come back when you've done so....

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 16, 2011 6:28 AM   in reply to Lee Jay-ZyZk56

    Lee Jay wrote:

     

    clvrmnky wrote:

     

    The principle of KISS applies here:

     

    - Use a simple directory structure if you do not need anything more complex. A dated structure works just fine.

    - Use metadata to slice across the collection.

     

    I don't use either of those.  I have a neatly organized complex folder structure, storing around 200,000 untagged images.  The time it would take me to tag those images effectively is, frankly, unthinkable.

     

    Note that this comment is a follow-on from the OP and other comments, and I mention "if you do not need anything more complex." I never suggested that everyone should drop what they are doing and switch to this method. However, if you were starting from scratch, you could do much, much worse.

     

    Like a neatly organized complex folder structure.

     

    This does raise an interesting issue, however. Metadata is only useful if it is both present and reasonably complete across a dataset. A lot of organizations with a large legacy datastore have had to hire people to add basic metadata when updating those datastores. I recall reading somewhere about a museum or collection of some kind that has recently had to do this.

     

    The fact is that the larger your data store grows, the less useful a plain directory structure becomes. Or, more idiosyncratic, anyway.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 15, 2011 6:01 PM   in reply to john beardsworth

    johnbeardy wrote:

     

    Find all your best images of waterfalls in Colorado, please, and then your rock paintings from all SW states. Do come back when you've done so....

     

    I have no such images.  I shoot events.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 16, 2011 1:44 AM   in reply to Lee Jay-ZyZk56

    Kinda predictable....

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 16, 2011 7:58 AM   in reply to john beardsworth

    Okay, then be honest.  What would you do if you were me?

     

    I have about 70,000 untagged images at work.  Only about 30,000 of them need to be tagged, the rest are parts of a long time-lapse.

     

    I have a bout 130,000 images at home.  Only about 110,000 of them need to be tagged, the rest are duplicates (raw+JPEG or saved exports).

     

    I estimate that tagging 140,000 images comprehensively would take 500 to 1000 hours of work (15-30 seconds each).  That's totally impractical.  I'd have to quit my job for several months to complete such a project.

     

    I could start tagging images now as I collect them.  But then I'd have all those other images that aren't tagged and all the new ones that are.  That means using the smart-collection approach would mean all the new ones were in neat collections, and all the old ones were in a gigantic "untagged" collection with over 100,000 images in it.

     

    I could tag them in some sort of automatic way, perhaps using the folder names as tags using some sort of script.  But that wouldn't add any information to the images they don't already have - the path names are already metadata associated with the images.  It also wouldn't add any uniquely identifiable information to the images so cross-folder searching would still be unavailable.

     

    Meanwhile, I can already find all my images efficiently in seconds in nearly all cases.  I estimate that there is about one situation per year where having a comprehensively keyworded image store would be helpful in saving me time - perhaps 15 minutes each time that happens.

     

    So - give me an honest recommendation on a go-forward path (no "coulda, woulda, shoulda's").

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 17, 2011 4:02 AM   in reply to Lee Jay-ZyZk56

    Lee Jay wrote:

    Okay, then be honest.  What would you do if you were me?

    ....

    So - give me an honest recommendation on a go-forward path (no "coulda, woulda, shoulda's").

     

    Honestly, I always start off from assessing what you really need or what value you could get from keywording. I wouldn't advise you to add keywords for their own sake.

    Lee Jay wrote:

    Meanwhile, I can already find all my images efficiently in seconds in nearly all cases.  I estimate that there is about one situation per year where having a comprehensively keyworded image store would be helpful in saving me time - perhaps 15 minutes each time that happens.

    Let's pretend that this isn't quite as black and white as you always put it. Let's say we're mainly talking weddings - typically wedding photographers add few keywords, if any. And let's say you do need to find certain images more often - each time you go to meet prospective clients you want to put on your iPad (you?) a portfolio of your best work for their broad type of wedding. So you might have categories like indoors / outdoors / subaqua, particular venues, age groups, religions through humanist,  formal / weird. Let's assume you do judge it worthwhile to increase the number of keywords.

    Lee Jay wrote:

    I could start tagging images now as I collect them.  But then I'd have all those other images that aren't tagged and all the new ones that are.  That means using the smart-collection approach would mean all the new ones were in neat collections, and all the old ones were in a gigantic "untagged" collection with over 100,000 images in it.

    Actually, that is the way I would advise you to go. Don't let the perfect be the enemy of the possible. It's not at all uncommon that people are put off beginning doing some keywording by the scale of the complete task. But beginning with the images that you're currently working on isn't too much extra effort. Even if it turns out that you never eventually do go back and keyword the old stuff, in a year's time you'll have a whole year's worth tagged. While doing so you'll probably adopt ways to make keywording less of a chore. For example you might add metadata presets to slap on overall keywords for the typical shoot "Wedding, Ceremonies, Christian" and then one or more keyword sets if you can justify putting time into varying the keywords between images - eg Bride, Kids, Cake. I wouldn't waste any time building up a hierarchy (do if you want, or not if you don't).

     

    I simply wouldn't worry about the 100,000 remaining untagged - they'll be easier to tackle once you're you've got the experience / benefits from keywording new stuff. I'd advise you to do it only as and when you need - it's too big a task to do anything else. Let's say you need your best shots of Jewish weddings and - as you're currently keeping a lot of metadata in your head - you know some are in that untagged mass. Find them through the folder system as you do now, whack on some keywords like "Wedding, Jewish, Non-orthodox". All those shots are of the bride - whack on "Bride" (maybe from a keyword set). Hey, she's got an obvious tattoo - let's just add that keyword just in case it'll be useful. And quickly move on. Whatever seems worthwhile.

     

    Although scripting could help, where's the script? Worthwhile to write/adapt one yourself? Instead for folder names I'd just just smart collections to gather image - eg folder name contains "weddings" - and then slap on the keywords. Sure, there will be some duplication between the pseudo-metadata in the folder names and in keywords, but duplication happens with metadata. In fact, it'll be quicker to search by the keyword "Wedding" than search for images with the keyword "Wedding" and / or with the folder name containing "Wedding".

     

    So that's how I'd advise you - start with the new work, and deal with the older stuff ad hoc.

     

    John

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 17, 2011 5:44 AM   in reply to Lee Jay-ZyZk56

    I'd suggest adding basic metadata on import, and then work on the keyword backlog in chunks knowing you won't necessarily be done.

     

    Your estimate for how long it will take per-photo is a little pessimistic, actually. Not only can you ignore large swaths of photos you aren't interested in right at this moment, a lot of keywording can be automated to some extent. For example, if you have a particular event you could keywords tens or even hundreds with a single gesture, and then burn through a sub-selection of those with a series of keywords sets.  This would average a lot less than the many seconds per photo in your estimate.

     

    I've done this myself for a few thousand photos -- I have a "No Keywords" smart collection that I have whittled down to zero over a few months.  Certainly not 100,000 (less than a quarter of that) but if we accept that we don't have to touch every single photo and concentrate on the ones that interest us at the moment the work will get done surprisingly fast.

     

    Metadata has a lot of benefits (discussed at length here and elsewhere) but chief among them is that your metadataset does not have to be complete to be useful. And the utility grows fast even with simply linear changes to the whole metadataset.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 17, 2011 6:41 AM   in reply to clvrmnky

    clvrmnky wrote:

    Metadata has a lot of benefits (discussed at length here and elsewhere) but chief among them is that your metadataset does not have to be complete to be useful. And the utility grows fast even with simply linear changes to the whole metadataset.

     

    So who said?

     

    "Metadata is only useful if it is both present and reasonably complete across a dataset."

     

    I was going to shoot that one down

     

    John

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 17, 2011 8:00 AM   in reply to john beardsworth

    I don't understand how an incomplete set is useful.  If I want to see all images of a certain type, having 90% of my images un-tagged means I can only see 10% of the images of a given type, not all of them.  What if the one I'm looking for is in that 90% set?  Further, what if I'm taggng images and I make a mistake.  For example, let's say I'm looking for all my P-51 images but I've failed to properly tag a few P-51s.  How will I ever find them again and how will I even know that some actual P-51 images aren't tagged as such?

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 17, 2011 8:17 AM   in reply to Lee Jay-ZyZk56

    You are making the perfect the enemy of the possible. One presumes you would realise that you haven't found the image, so find it the normal way, tag it and move on. Next time you'll find it straight away.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 17, 2011 3:31 PM   in reply to john beardsworth

    For the direction that this thread has taken, namely what to do with an untagged collection, a lot of value can be had from processing a subset of the data. Metadata is most useful when nearly complete (where "complete" is defined as something good enough, both in terms of quality and consistency) but is still tremendously useful if it is not (and this is known.)

     

    Especially if we have decided to process the data in chunks, and can use other existing metadata to help narrow things down into these chunks.

     

    When the entire dataset is stored with reasonable amounts of normalized metadata this is, of course, the best situation.

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points