7 Replies Latest reply on Feb 1, 2018 5:15 PM by Trevor.Dennis

    Best storage solution for large amount of photos and 4k video files?

    matthewh133

      Hi everyone, I do photography for a living and thus have terabytes upon terabytes of 42 megapixel photos and 4k video files (Sony A7R III, Canon 1DX II). Currently, I have a very inefficient method of storing these files on external hard drives and just having to plug in whatever hard drive contains the files that I need at that time.

       

      What I want to do is have all of my files stored on one device that is permanently attached to my computer so I can access them on a whim. Also, I want this device to be fast so that I am not having issues with Lightroom, Photoshop, Premier Pro etc lagging because of the connection type.

       

      I'm thinking that right now I want a device that can hold up to around 48tb+ (24tb for storage and 24tb for backup/copy).

       

      What would you suggest for my needs? Just an external USB 3 hard drive enclosure with 4 x 12tb hard drives? Would this be slow to edit on? I have a separate SSD in my PC that I can copy Premiere Pro projects and their corresponding video files onto to edit when working on a particular project and then move them off of afterwards, but I'm guessing this hard drive enclosure solution should be more than enough for photo editing off of and storage?


      Any thoughts would be greatly appreciated as I'm a little lost right now. I should mentioned I'm also hoping to spend around $1500 (maximum $2000) if possible.

        • 1. Re: Best storage solution for large amount of photos and 4k video files?
          Trevor.Dennis Adobe Community Professional

          Is there any reason why you don't have your external drives permanently connected?

          Do you use those little pocket drives that get their power via the USB lead, or drives with their own power?

          This

          https://regmedia.co.uk/2008/03/25/wd_elite_passport_1.jpg

          vs this

          I am now up to eight external drives. Three WD My Books, and five Seagate Expansion.  I have had three WD My book drives fail over the years, and the Seagate drives perform better as well at 180MB/s continuous, as against 120MB/s for the WD drives.  The power blocks were getting a bit OTT, so I now power them via the computer's PSU — it has made things a lot tidier with two less power boards behind the computer desk.

           

          So what I do is use fast internal Project drive for current projects, and bump content to the external drives as and when I need the space.  Up to now that was using two 256Gb Samsung PRO SSDs, and a 1Tb Samsung EVO SSD.  The new computer I hope to have completed any day now, with use two Samsung 500Gb 960 PRO NVMe drives, and I will move one of the 256Gb and the 1Tb SSDs to the new box plus five internal HDDs.

           

          For my DAM I i name folders 'year_month_day meaningful name' which causes them to sort in month order.  I just use Windows 10 to search.  It is fast and finds what I am looking for.  Purists like Bill on the Premiere Pro Hardware forum would tell you to keep indexing turned off — especially with so many drives.  I'm toying with the idea of having alternative logins.  A normal usage with all the processes running, and drive indexing turned on, and a lean-mean-editing-machine login.  I don't know how doable this will be, but I intend looking into it.

           

          I use Shadow Protect for backup, set to 15 minute incremental backups.  You can mount the backup at any of those points, and you can make an image of the boot drive and boot to it from a USB thumb drive or CD/DVD.

           

          I chose a motherboard with lots of USB3 sockets for my 7900X build (MSI X299 fatal1ty gaming pro i9) but you can get USB3 headers if you have spare PCIe sockets.  You ,might need to check that you have the PCIe lanes available to make the most these.  With so many drives, it can become a real issue.

           

          I also got a header to bring two internal USB3 sockets to the back pane.  This is on top of the two USB3 at the front of the case.  For the sake of a $10 header, it seems crazy not to make the most of all available connectivity.

          So I hope that helps, but please ask if unclear about anything. Also look at the Premiere Pro Hardware forum overview page, which has lots of useful information.  Bill and the other regulars know their stuff, (much better than me) and are always willing to help.

          Hardware Forum

          • 2. Re: Best storage solution for large amount of photos and 4k video files?
            matthewh133 Level 1

            Thanks a lot for your reply, truly appreciated. I could go that route, but I was kind of hoping to simplify and move away from so many cables, as well as have higher speeds. From what I can tell from my research, anything USB is not going to compare to the RAID solutions that have thunderbolt 3 connectivity, or NAS setups that have ethernet cable connection possibilities. It seems with a RAID or NAS I could have all of my files (even current projects) stored on them while being able to take advantage of the speeds to edit even 4k video no problems? The downside, is that I'm probably looking at around $2900 for a 48tb RAID setup (something like the OWC Thunderbay 4 48tb).

            • 3. Re: Best storage solution for large amount of photos and 4k video files?
              Test Screen Name Most Valuable Participant

              Network storage is going to be slower than USB3. RAID can’t go faster than your connectivity, it’s not usually a speed thing. Are you using USB3 devices, ports and cables now?

              • 4. Re: Best storage solution for large amount of photos and 4k video files?
                Test Screen Name Most Valuable Participant

                Hmm, and you want a single device for original and backup? That would be a pretty poor backup.

                • 5. Re: Best storage solution for large amount of photos and 4k video files?
                  matthewh133 Level 1

                  I was looking at these RAID systems:

                   

                  B&H Photo Video

                   

                  B&H Photo Video

                   

                  They are both thunderbolt 3. My PC doesn't currently support Thunderbolt 3, but I can definitely replace the motherboard with one that does.

                  • 6. Re: Best storage solution for large amount of photos and 4k video files?
                    matthewh133 Level 1

                    I would also have an off-site backup of all my files as well as the RAID.

                    • 7. Re: Best storage solution for large amount of photos and 4k video files?
                      Trevor.Dennis Adobe Community Professional

                      It's sad that Harm Millaard passed away, as he would surely have the ultimate solution using today's storage options.

                       

                      Choosing storage

                       

                      Even for 2015 Harm's nine SSD raid array had very tidy performance

                      Note that Harm and the PremPro Hardware guys always used proper raid controllers, that being the Areca ARC-1882ix-24 controller for Harm's 9 SSD array.   You'd pay $600 for a modern version of a card like that.

                       

                      This is for the 2Tb version of the Samsung 960 PRO MVMe drive.  That's a single drive not so far behind Harm's nine SSD raid array performance.  The times they are a changing, but no one os going to shell out $12K for 48Tb of NVMe storage, and nor would they need to.  A single 2Tb drive would keep you in project space for quite a while, unless you are Peter Jackson using two dozen RED Epic or Weapon 8K cameras.  Then bump your old projects to slower external storage.

                       

                      Incidentally, my eight external drives are in a single enclosure with a power break out box and cooling fan, but there are still eight USB cable between that enclosure and the computer.