13 Replies Latest reply on Jan 20, 2017 6:05 AM by davidpayne1981

    Verifying my understanding of Premiere, AE disk I/O structure

    Lovestar Level 1

      Hi guys! =)   I am currently building a new system to venture into video editing with. I'll be using Creative Cloud for my entire workflow because, well, a student sub to CC is cheaper than buying other people's products individually, haha. Marketing success!

       

      Anyway, I have done so much reading about video editing with Premiere / AE (including Harm Millard's guides) and wow, this process has been enlightening (and brain-exploding). I never knew building a PC could feel so complicated. But, I have basically everything selected for my system now. The place I'm held up is in the disk structure — this is really not something I want to mess up!

       

      I'm not asking you to tell me exactly what to buy, instead I want to make sure I can select the right products myself. I want to make sure I correctly understand the different potentially-conflicting I/O that goes on during PP and AE. I've read the white paper, Tweakers' Guide, etc. but some aspects still feel uncertain or vague to me.

       

      I'm sorry if these questions feel really lame / naïve, I'm a creative person but not a very analytic/technical one so some of the nuances of hardware & software operations go over my head (even if they seem obvious to you, haha).

       

      My current understanding of PP & AE I/O pathways is that all these categories of files can potentially conflict for disk access:

      BOTH PROGRAMS

      Windows / OS system files.

      PREMIERE PRO

        • Premiere application data (the installed program)

        • Project files

        • Source media / Raw footage

        • Media Cache

        • Previews

        • Exports

      AFTER EFFECTS

        • AE application data (the installed program)
        • Project files
        • Source media / Raw footage
        • Media Cache
        • Persistent Disk Cache
        • Previews
        • Renders
        • Exports

       

      I have some naïve questions about this, a few things are still slightly vague for me:

      • In this context, does 'Project files' refer strictly to the actual project save (like *.pproj), or is it a catch-all term for all 'files' used in the current 'project' (like audio samples, Illustrator files, etc.)?
      • Are 'Previews' and 'Renders' the same thing referred to in different ways, or are they actually distinct operations — ie, you could hypothetically have 1 disk for 'Previews' and another disk for 'Renders'?
      • Alternatively, is the 'Render' directory synonymous with the 'Export' directory? Or are these two distinct concepts?

       

      Next — my understanding of the Adobe recommended disk configuration (based on the white paper) is this:

      1. OS, programs
      2. Media Cache / Scratchdisk
      3. Source media / Raw footage
      4. Renders / Exports

       

      I'm still uncertain about where some file types into that scheme:

      • Which drive would Previews go on?
      • Where would the Persistent Disk Cache go? Is it OK to put it on the same drive as the Media Cache?
      • Where do 'Project files' belong?

       

      To clarify, I have seen guides like Harm Millard's and the white paper suggesting configurations for some of these (like putting Projects with Source Media, or Previews with Exports, etc) but unlike the "Adobe four" above, different forums and websites recommend different places for the 3 I'm confused about. That's why I'm asking — no one seems to agree on where Previews & Projects should go, and I can't tell whether 'Persistent Disk Cache' is lumped in with 'Media Cache' as a catch-all term in reference guides.

       

      Especially, about 'Project' files, I'm not sure exactly how they're 'used' by PP / AE:

      Are these accessed constantly (like the Media Cache) or only in short burst operations (reading once when loading the project, writing once  when saving/autosaving)? Like does PP just read all the 'Project' data when you load it up (to know how to switch all the different settings like you had it saved) and then ignore the Project data while actually working (until it gets saved again)? Or is it constantly checking the project data while working?

       

      Do Programs and Cache on the same SSD create access conflicts?

      I ask because someone had mentioned in a thread that I Google'd up, that Programs & Media Cache don't conflict with each other because disk access for the program files ends once PP / AE finish loading, while the Media Cache only begins access once the program is fully loaded. This sounds odd to me because I would think the program is accessing its code constantly while working, but then I'm not very smart when it comes to the guts of software so I'd rather ask than assume. =)

       

       

      Thanks guys, for any insight you can provide. I've spent weeks and weeks researching hardware/software/disk stuff (I know the difference between "socket" and "chipset" now!), and I've learned a lot, but these last few gaps are holding me up on making the final hardware decision (storage setup) before I start building. =)

        • 1. Re: Verifying my understanding of Premiere, AE disk I/O structure
          JEShort01 Level 4

          It sounds like you  are stressing too much about this!

           

          You need to tell the forum:

          - your workflow (media type, # layers, duration of projects - 3 minute YouTube or 1.5 hour feature!!!, number of layers)

          - what is the rest of your planned system? With Premiere Pro, and After Effects possibly to a lesser degree, you want to purchase drives that match the rest of your system and your price point.

          - what is your budget for drives?

           

          Anyone who answers what to do without this information would be assuming that one size fits and that is not the case with Adobe!

           

          Harm Millaard's articles on www.ppbm7.com and on this forum, "To RAID or not to RAID" address the subject of balance and suggested drive configurations spanning a wide range of cost and performance. You said you have read his guides, but there are so many he has contributed that I wanted to note these two specific items.

           

          Regards,

           

          Jim

          • 2. Re: Verifying my understanding of Premiere, AE disk I/O structure
            Lovestar Level 1

            Hi. =)   I'm not stressing! haha. I'm just doing due-diligence research before creating my system — I'd rather double-check my understanding now, than buy hardware I don't need.

             

            The reason I haven't included hardware information, workflow, budget, etc. is that I don't believe it's relevant here. Again, I'm not asking for specific instructions about what *I* should do — I understand the tech side enough to make those decisions once I have the full information.

             

            Instead, I want to make sure I understand the general theory / facts about NLE with PP / AE. If I have an accurate understanding of the I/O flow these programs use, I can build appropriately.

             

            I did read everything on PPBM7, as well as Adobe white papers and plenty of other Google soup. That's why I'm asking here — I have a good general understanding now, but wanted to verify / clarify some of the details I'm still uncertain about.

             

            All that said, I'm certainly not trying to be difficult (you're helping me, after all ), so if you really think my specific info is relevant here I'll be happy to update with those details.

            • 3. Re: Verifying my understanding of Premiere, AE disk I/O structure
              SebastiaanFP Level 1

              Hi Lovestar,

              You did your research well! The questions you have, I think can only exist because you haven't gotten the Adobe CC and your PC yet. So I am going to give a general answer, but because you are a starter / student, a think a general workflow & system is going to be fine for you? Here is a link to the Tweakerspage, where halfway down exactly is written how to distribute the various processes over the various amounts of disks.

               

              http://ppbm7.com/index.php/tweakers-page/84-disk-setup/95-disk-setup

               

              When you go into the preferences of PPRO and AE all these questions (meaning the last finetuning questions you asked in your post here about the different naming options used and the like...) will be very clearly presented to you as configurable options, and also each time when you open up a new project in these programs. So then each possible choice will be presented to you to configure by these programs themselves, with their respactive names, so you do not have to worry any further, it will fill in you last gaps of knowledge once you actually start working with these programs.

              Basically the more disks you can get the better, but 4 to 6 HDD already is going to do a lot for a general system. And if you have some RAID options, preferably in the hardware of the motherboard, instead of software only, that will make things even better. Ofcourse there are also the separate RAID controllers you can buy, but they have a price, or complete RAID-external HDD-sets with 2 to 6 HDD's

              Good luck, and I guess you are ready now to just start building your system and grow your last parts of understanding by actually sitting behind your PC with the Adobe CC installed.

               

              Greetz Sebastiaan

              • 4. Re: Verifying my understanding of Premiere, AE disk I/O structure
                Lovestar Level 1

                Hi, thanks. =)   I understand you can choose where to put all these individual items, but I'm trying to understand which ones should be separated from each other, and to where. Rather than just asking "Tell me what disks to put this on!", I'm trying to educate myself about the actual I/O system so I can make informed decisions about spacing out potential resource conflicts.

                 

                There's still a lot of stuff that reading PPBM7 and white papers & etc. have left me unclear about. Stuff like (kind of collating a shortlist from the OP):

                • Is it detrimental to put AE's Persistent Disk Cache and Media Cache on the same disk?
                • When guides refer to placement of 'Project' files, is this the literal project save (.pproj, etc) or a catch-all term for all files associated with the Project itself (like audio clips, etc)?
                • How frequently does your project access its save file while editing? Only during saves/loads, or more frequently (to load instructions for the Preview rendering, etc)?
                • What's the difference between 'Preview', 'Render', and 'Export', and should these each be on separate drives? I know you can 'render a preview', but AE guides also refer to 'rendering' in a finalistic sense which is separate from 'exporting'.
                • Is it detrimental to use the same SSD for both the Program installation and the various Caches/Scratches? Should OS/Programs be separated (diskwise) from the Caches, despite the SSD's fast transfer rate?

                 

                All of this stuff affects potential I/O issues. I'm sure you can edit 'fine' without sorting this stuff out, but since I'm building new and have a chance to make a clean storage organization, why not try to get it just right? =)

                 

                I really have read the guides, but they leave me still unclear in some areas, which is why I'm asking. For example, from PPBM7's Disk Setup that you linked:

                • It says SSDs don't suffer from fill-rate degradation — but everyone else says they do, because at fuller capacity they have to run more calculations to write a piece of data. (http://www.howtogeek.com/165542/why-solid-state-drives-slow-down-as-you-fill-them-up/)
                • It says 'Project files' includes 'project auto-saves' but it doesn't clarify if these are also in the same category as other project-related files (like still images and audio clips).
                • It says that the 'Project' file isn't accessed once you start writing for 'Export', which confuses me because doesn't the project file tell the export what to render/encode?
                • It lists multiple options — put 'Project' and 'Export' on the same disk, or put 'Media Cache, Previews, Export' on the same disk. So, since there's no 1 answer, I'm trying to understand the workings as much as possible and make my own best judgment.
                • It suggests putting Source Media and Project files on the same disk as an option, which leads me to wonder if this creates issues (with reading Source Media and Project file data at the same time) which is being compromised on, or if it just doesn't matter — hence why I ask about how and when 'Project' files are accessed.
                • It suggests that 'Media Cache' and 'Previews / Exports' can be RAID0'd together into one volume. This confuses me, because we just split them into separate volumes to avoid I/O conflicts. Now we're undoing that split, just with (modestly) faster read/write speeds (but wouldn't this still cause half-duplex I/O conflicts?). oO

                 

                Then there's the Adobe Hardware Performance White Paper, which advocates a 4-disk system (see page 16):

                • OS/Programs
                • Source Media / Project Files
                • Media Cache/Scratch Disk
                • Exports / Final Renders

                 

                But Adobe's own documentation here doesn't address, or ambiguously addresses, the 'correct' location for some major file types, such as:

                • Persistent Disk Cache (which is a separate entity from Media Cache, according to the AE help file)
                • Previews (which Harm Millard indicates are a major storage 'type', but the White Paper waffles about and doesn't even mention in the Storage summary)
                • What exactly 'Project files' is defined to mean in this context, and when/how often they're accessed while editing.

                 

                This is why I'm trying to narrow down and clarify some of these concepts — they seem a bit fuzzy / slippery compared to 'easy' stuff like the Media Cache, or OS/Programs, which everyone seems to agree on.

                 

                I really don't mean to be annoying. I'm just trying to learn as much as possible before committing to a system design and hardware purchase. =)

                • 5. Re: Verifying my understanding of Premiere, AE disk I/O structure
                  Steven L. Gotz Level 5

                  You are slightly overcomplicating it, but maybe this will help.

                   

                  When we say Project Files, it is the prproj files we are referring to. They are referenced often, but are not a huge bottleneck like video is. Project files are only a slight percentage of the size of a few minutes of video.

                   

                  The Source media is the video, audio, images, etc.

                   

                  Personally, I organize mine this way:

                   

                  C:\ has Windows and Programs (not the Windows Temporary files or the My Documents folder). Just the OS and Programs. So it seldom needs to be defragged and fits easily on an SSD. I back this up by cloning it.

                   

                  E:\ I keep Source files, video, audio, images, etc, on this RAID0. This gets backed up to an external drive dedicated to large files like these.

                   

                  F:\ I keep my cache and my preview files on this RAID0. All cache and all temporary files. This is never backed up and is often erased and defragged. That keeps it fast. It also holds my Windows Paging File.

                   

                  G:\ Main PC Storage. I use this for my projects and their auto-saves. This is where the My Documents folder resides as well. It gets backed up to a separate external drive occasionally, but I primarily rely on a cloud service to protect my data. These are files I might also be likely to need on a different PC or on my SmartPhone.

                   

                  So, basically I try to keep files that I will not back up separate from those that I back up by cloning, or those that I back up to a dedicated external drive. It also prevents bottlenecks by trying to pull too much or write too much to any one drive.

                   

                  Now, I must say that I have taken to exporting projects to the large, fairly empty F: drive now and then if I know they are temporary. Since I do not use preview files when exporting, it doesn't cause a problem as far as I can tell. Only final exports make it to the main PC storage drive.

                   

                  Edit: I missed mentioning that Preview files are what is created when you have a red line over your timeline so you press the Enter key to render the timeline. Temporary files are created to make it easier for Premiere Pro to play back your timeline in real time.

                  1 person found this helpful
                  • 6. Re: Verifying my understanding of Premiere, AE disk I/O structure
                    Lovestar Level 1

                    Thank you! I know this probably seems perplexing from your perspective, but that helped so much with putting all the pieces together conceptually. =)

                     

                    Let me just make sure I understand the last few concepts correctly:

                    • Project files are accessed frequently, but are so tiny that it doesn't really matter where they're located, because even if they technically cause an I/O conflict, it will be resolved in an eyeblink.
                    • "Rendering" — in this context — is any operation which basically looks at the Project file, pulls all its instructions together, and glues them into a composed product.
                    • You can "Render for Preview" (quick & dirty) or "Render for Export" (meticulous and precise to produce your final product). In both cases these are 'Renders' but they have totally different purposes.
                    • Rendering for Preview creates a temporary Preview file on-disk, so you don't have to render it over and over while working.
                    • Rendering for Export creates the final, encoded whatever that you either submit, upload, burn, or move further down the production pipeline.

                     

                    So that's why guides suggest Previews and Exports can go on the same drive — you're never rendering (writing) both of them at the same time, and there's no need to read Previews while writing an Export.

                     

                    Likewise, guides don't stress much about where Projects go because, while Projects are very important files, their impact on disk I/O is trivial.

                     

                    Have I got it all right? =)

                    • 7. Re: Verifying my understanding of Premiere, AE disk I/O structure
                      Steven L. Gotz Level 5

                      Well, most of use use "render" for creating preview files, and "export" for creating named files for some greater purpose, but basically, yes.

                       

                      The only time most people use previews for exporting is when they are using DV AVI. Then the preview files are acually DV AVI. Whereas, mostly I create preview files that are specifically NOT the same codec as I am going to export to.

                       

                      Project files can get rather large, but not in comparison to video. So yes, they can be just about anywhere. Be careful though. I accumulate huge quantities of Auto-Save files because I allow the program to create them. Just in case.

                       

                      But yes, I believe you understand it.

                      1 person found this helpful
                      • 8. Re: Verifying my understanding of Premiere, AE disk I/O structure
                        Lovestar Level 1

                        Thank you. You solved basically all the big questions I still had, and I feel confident assigning my storage strategy now. =)

                        • 9. Re: Verifying my understanding of Premiere, AE disk I/O structure
                          lloydpdx7 Level 1

                          Just now finding this thread as I too am making sense of Premiere's hardware needs, but I have a more basic curiosity for which I'm looking for resources: Just what is happening to which files during particular processes, especially export? (what is the interplay amongst the various kinds of files?) Maybe a slow-motion flow chart of I/O? Thanks for any pointers or references.

                          • 10. Re: Verifying my understanding of Premiere, AE disk I/O structure
                            R Neil Haugen Adobe Community Professional & MVP

                            In export, your program files pass commands to apply your current project's "instruction set" (all the choices you've made while editing, from clips/placement/effects/grading/et al) to the original footage as "placed" on the timeline, then output that to the particular codec specified in the output command to a particular location. Basically every frame is its 'own thing', so it's a pretty intensive process run thousands of times for a few minutes of output. It is constantly reading/writing bits & pieces out of quite a few files, and so having said files on different drives is a great thing to eliminate read/write bottlenecks.

                             

                            That's why you've seen the suggestions for having program/os on one drive, media on another, cache on another, renders/outputs on another ... and also many suggest the project files themselves have their own drive. And ... SSD's are better for program/os and cache drives, and either SSD's or RAID 0 for media and render/output drives. In an optimal world, of course. 

                             

                            Neil

                            • 11. Re: Verifying my understanding of Premiere, AE disk I/O structure
                              lloydpdx7 Level 1

                              Thanks, Neil, that's a concise helpful visual overview of what's happening during export. Helps a lot.

                               

                              I'm now trying to discern whether having discrete drives is ALWAYS optimal, or, in a multiple (3) SSD setup, if some file types can share their drive much more easily than with HDDs. I'm trying to maximize an i7 MacBook Pro for small scale editing and rendering, before spending cash I don't have for more of a workstation than I really need. Two internal 6g ssd's planned, and thinking about a single external thunderbolt ssd. With backup of course.

                              I've seen the generic disk setup guidelines posted somewhere here; just not sure if the same rules apply in an all-ssd setup.

                               

                              -Lloyd

                              • 12. Re: Verifying my understanding of Premiere, AE disk I/O structure
                                R Neil Haugen Adobe Community Professional & MVP

                                A lot of editing has been done on 3-drive machines, with all three being just decently fast/decent-cached standard drives. If you're planning on 3 drives of say two LARGE internal SSD's and an external thunderbolt ssd ... that should improve things rather notably. As to which files to 'associate' ... the three things most separated are os/program, cache files, and media. I think this figures that it gives you a better working environment, and not having a separate out-put drive ... well, you'll just plan on setting an output and going to dinner or bed.

                                 

                                In that sort of set-up, I've seen project files argued for both the os/program disk and the media disk. Cache files are mostly left by themselves. Again, plenty of disagreements are possible.

                                • 13. Re: Verifying my understanding of Premiere, AE disk I/O structure
                                  davidpayne1981 Level 1

                                  this is an interesting thread and sums up my thoughts for 23 hours of the day at the moment!

                                   

                                  I have a new iMac 5k and I have just purchased a thunderbolt 2 enclosure to use 2x 840 evo ssd's in RAID0 and that will be my video drive. I think the max speeds of those drives will be around 500MBps read and write.

                                   

                                  With that in mind do you guys think its worth it to utilise an existing super fast (2200MBps read and write) pci-e drive in a thunderbolt 2 enclosure as a cache drive?

                                   

                                  What I'm unsure of is whether premiere will be reading both video and cache drive simultaneously as if so then the 20GBPS max of the Thunderbolt bus will be split between these two drives and I dont want them to slow down.

                                   

                                  With that in mind, if the video drive just has a couple of medium speed SSD's in them then I assume there will be plenty of headroom left for the cache drive so in theory they shouldn't slow eachother down?

                                   

                                  Would love to hear your thoughts..