10 Replies Latest reply on Jun 9, 2013 2:45 PM by Harm Millaard

    What's it like to edit with an RPI of, say, 5?

    Clay201b

      I've spent quite a bit of time going over the results on the PPBM site. I spent today digging a little more deeply into the RPI figures. Obviously, editing on a machine with an RPI of 1 will give you the best possible experience; the fewest freezes, the least lag, the fastest renders. And an RPI of, say, 20 will be much, much slower. But what if your machine clocks in at, say, 5? You're not at the bottom of the heap, but neither are you close to the top. Does this mean that you're suffering through constant crashes? Do you have to transcode your AVCHD to a different format before editing can even begin? Do you give up entirely on certain effects or limit yourself to just one or two layers? Does scrubbing through the timeline make the baby Jesus cry?

       

      In your personal experience, how slow is too slow? As you slide down the RPI scale from 1 to 5, to 10, etc., at what point does the software become unusable?

       

      Obviously, this is a subjective question, but that's okay, because I'm hoping for some subjective answers.

        • 1. Re: What's it like to edit with an RPI of, say, 5?
          Harm Millaard Level 7

          Go to Benchmark Results and make sure you are registered and logged in and scroll down a bit till you see a chart like this:

           

          Ideal-RPI.png

          The image here is static, but on the page I linked to it is dynamic and is automatically adjusted when new entries are added.

           

          Notice the descriptions along the Y-axis and how they relate to the percentile values of the RPI. Say you have a system with a RPI of 500, it means this system is 5 times slower than the fastest system which has a RPI of 100. That is even slower than the Q1, lower quartile value of 415, so this system belongs to the bottom 15 - 20% of all systems and has around 80 - 85% systems in the benchmark that are faster.

           

          Should that worry you? If you built your system on a shoestring, a real budget system, no not really. There is some room for improvement and you can try to optimize your system a bit untill the RPI is around 400, but nothing to get all exited about. Things are different if you have a real costly system, say $ 5K, because then your score would normally have been in the upper quartile Q3 range, so there is a lot of improvement that can be achieved.

           

          Which number is tolerable, 200 or 500 or even 1000? That is purely personal. If you are a hobbyist, who doesn't mind some waiting from time to time and who enjoys a beer-thirty, then your tolerance level is distinctly different from a shop working under tight deadlines every day with clients looking over their shoulder impatiently.

           

          What factor, which test causes the RPI score as it is shown? Is that disk performance? If so, then one has to determine if that is relevant for the workflow in question. In the case of a wedding editor with long exports and impatient newly-weds it is different than a hobbyist who exports a vacation film and that is different from a shop that exports daily to a number of different formats. Is it the video card? Well that could be a real problem when clients are watching over your shoulder, but otherwise it may not be a problem at all.

           

          There is just no clear answer to your question. RPI is a result of the weighing of the individual tests and can be helpful to determine if improvements are needed. But ultimately it is the user who decides if he can live with the results from this test, how relevant it is for his or her workflow and what possible budgettary consequences there are when improving a system with the ultimate question: Is it worth it for me?

          • 2. Re: What's it like to edit with an RPI of, say, 5?
            Clay201b Level 1


            There is just no clear answer to your question.... Which number is tolerable, 200 or 500 or even 1000? That is purely personal. ....  ultimately it is the user who decides if he can live with the results from this test.... the ultimate question: Is it worth it for me?...

            As you point out, each user answers this question for him/herself. So there are *many* answers to my question out there. I'm hoping to hear from individuals who've tackled this question and can share their personal experiences.

             

            You certainly have a good point about the ultimate question. I'm hoping this discussion will be of help to people as they try to answer it in their individual circumstances.

            • 3. Re: What's it like to edit with an RPI of, say, 5?
              JEShort01 Level 4

              I truly value the PPBM tests that Bill and Harm have created and supported and here are my opinions...

               

              PPBM tests (PPBM5 and the relatively new PPBM6) are great for:

              • verifying that your system is working to its potential and identifying areas where it is strong and weak
              • comparing how various drive and file location configuration options perform - where you locate projects, media, scratch, cache, render outputs, etc.
              • comparing specific hardware - one cpu vs. another, laptops vs. desktops, etc.; this is not a perfect science with PPBM, but this resource beats any other resource or benchmark that I am aware of to understand how to give Premiere Pro what she wants and needs for various tasks

               

              Personally, I don't put much weight on the "bottom line" RPI number and here is why:

              • One person may care a lot about how long it takes to render to Blu-ray and another to DVD - they should worry about the hardware necessary to get what they need
              • While some aspects of editing could be acceptably slow (i.e. rendering for someone that does this once per month) whereas other aspects of editing (i.e. timeline playback with no lost frames without rendering) would impact what most users are doing most of the time

               

               

               

              Regards,

               

              Jim

              • 4. Re: What's it like to edit with an RPI of, say, 5?
                Clay201b Level 1

                Personally, I don't put much weight on the "bottom line" RPI number ....

                • While some aspects of editing could be acceptably slow (i.e. rendering for someone that does this once per month) whereas other aspects of editing (i.e. timeline playback with no lost frames without rendering) would impact what most users are doing most of the time

                 

                So you don't feel like the RPI number accurately reflects the timeline playback performance? That is to say, a machine that earns a 200 could have choppy playback while a 500 does just fine with the same footage, same codec?

                 

                I ask because playback is an area where issues frequently arise, but I can't find any benchmarks or tests which attempt to measure it. (And I don't know whether it's possible to create such a tool. I'm guessing probably not). Since the RPI is intended to be an overall measure of performance, I figured it was probably the closest I was going to come to such a metric. (It's also a nice shorthand way to express the distinctions we draw between relatively faster and slower machines). Beyond that, I was hoping that individual accounts of editing experiences would provide some insight into this (and other issues that don't lend themselves to statistics).

                • 5. Re: What's it like to edit with an RPI of, say, 5?
                  JEShort01 Level 4

                  Not necessarly

                   

                  Correct

                   

                  Not all media, filters, and workflow are alike.

                   

                  For example, someone doing playback on multiple layers of uncompressed might really be taxing drive speed.

                   

                  Whereas someone playing back layers of Red 4k which is highly compressed and very cpu intensive will be taxing even the fastest of cpus.

                   

                  And yet other workflows benefit greatly from the GPU (graphics card) MPE (Mecury Playback Engine) capabilities of CS5/5.5/6 for real-time playback and faster renders.

                   

                  PPBM is a great tool but cannot solve every question for every workflow.

                   

                  Jim

                  • 6. Re: What's it like to edit with an RPI of, say, 5?
                    jamesp2 Level 1

                    Clay201b wrote:

                     

                    I ask because playback is an area where issues frequently arise, but I can't find any benchmarks or tests which attempt to measure it.

                     

                    While you can probably assume that a machine with fast encoding and export times is unlikely to lag at playback -- even without a direct measure of time-line responsiveness-- predicting actual time-line editing performance is difficult because of issues which may not be reflected in the hardware or system tuning at all.

                     

                    The problems associated with playback of AVC and AVCHD in PPro is a prime example.  You can have a machine perfectly capable of handling that footage, and multiple streams of it, but the experience of editing such footage still leaves a lot to be desired, because of PPro and/or video driver issues.

                    • 7. Re: What's it like to edit with an RPI of, say, 5?
                      Bill Gehrke Adobe Community Professional & MVP

                      JEShort01 wrote:

                       

                      I truly value the PPBM tests that Bill and Harm have created and supported and here are my opinions...

                       

                      Personally, I don't put much weight on the "bottom line" RPI number and here is why:

                       

                      Regards,

                       

                      Jim

                      Thank you much Jim!  And as far as my personal use I rarely use the RPI score as the detailed tests are what I use.  And with the new PPBM6 benchmark you now have three even better strong indicators from within Premiere of:

                      1. Disk system performance (actually measured transfer rate in Megabytes/Second)
                      2. GPU performance
                      3. CPU/memory

                       

                      I really wish there would be a way to measure timeline playback, but I just do not know how to measure it.  So RPI is a good composite indication but as you pointed out you have to really get to the 4 basic tests that we now have done much better in PPBM6

                      • 8. Re: What's it like to edit with an RPI of, say, 5?
                        JEShort01 Level 4

                        Bill,

                         

                        Although the VB scripting may not be able to capture the results, CS6 has the ever so useful dropped frames counter which does indeed provide a measurable way to measure the timeline playback metric. (CS5 did not have this, I think CS5.5 does but I'm not certain)

                         

                        My personal opinion is that this could be very useful to record and report for future PPBMs. (i.e. play this difficult timeline and record dropped frames for full resolution, 1/2 resolution, 1/4 resolution)

                         

                        Regards,

                         

                        Jim

                        • 9. Re: What's it like to edit with an RPI of, say, 5?
                          Bill Gehrke Adobe Community Professional & MVP

                          Thank you again Jim, we will have to see what it takes to drop frames as both Harm and I have real heavy duty work horse computers.

                          • 10. Re: What's it like to edit with an RPI of, say, 5?
                            Harm Millaard Level 7

                            Clay,

                             

                            To give you some further insight in how to look at RPI and its merits, or lack thereof, one of the things I'm still struggling with is presenting the test results of the new PPBM6 benchmark in a new format, that makes it easier to interpret where areas of improvement may be with your specific system in comparison to other systems.

                             

                            Sidebar: This proves to be major hurdle to take, since it requires dynamic SQL queries and calculations of the database and presenting the results in these kind of graphs in a LightBox pop-up window for the selected record.

                             

                            Angular-alt.png

                            This may not be the final presentation form, but it will show in an eye-glance what this system does in comparison to other systems.

                             

                            Short explanation: Each bar shows the rank of this system on a specific test, compared to all other systems expressed as the percentage ranking. So, on Total Time this example ranks at 70%, with 30% scoring higher and 70% scoring lower. On RPI 25% score higher and 75% score lower.

                             

                            If I look at these results, the overall for Total Time and RPI are quite good for the system in question, but what immediately catches my eye is the excellent Disk I/O results, they are very, very good @ 85%. Additionally, the MPE results are very good @ 81%, but both the MPEG2-DvD and H.264 results are somewhat disappointing with 55% and 64% respectively. So, the overall results are positively influenced by the great Disk I/O performance and to a lesser degree by the good MPE results.

                             

                            If I were in the dark about this systems configuration, I would guess that this system has a dedicated raid controller with a large array causing this great Disk I/O performance and has a very good video card, but is held back on the other tests by other factors. The fact that H.264 is better than the MPEG2-DVD results shows me that this system is nicely tuned, no superfluous background processes running, is overclocked with a nice amount of memory and is better than a plain quad-core, probably a hexa-core or better, but the MPEG2-DVD results are held back because of the interaction between memory and VRAM, which may indicate an older generation CPU. Also, the degree of overclocking may be limited, increasing the latencies between memory and VRAM.

                             

                            To go back to my earlier post, if you have to export to multiple formats each day, this is a fabulous result, because of the excellent Disk I/O results, if you have clients looking over your shoulder regularly your video card is great, but if you were to export long timelines from AVCHD to MPEG2-DVD, then you may consider upping memory or overclocking a bit more to make a difference.

                             

                            Note that this is with the same RPI. Now if you compare different systems, you may find different points of improvement. See for example this presentation form:

                             

                            Gauges.png

                            Here you see essentially the same info, but presented in a different way.

                            I prefer this way with the speedometer over the first presentation form, but it is even more difficult to implement, since it entails six separate charts.

                             

                            Two significant differences in comparison to the earlier chart are the Disk I/O performance, which is lacking in this system in comparison to similar systems and the good MPE performance, which shows a very good video card for this system - but not overall - and which helps the MPEG2-DVD score to achieve nice results. This system is more balanced than the previous example, since the scores match better even when the performance is much lower. Note that the first example showed 55% for the MPEG2-DVD test and this one 51%, but the first one showed a RPI score of 75%, while this system shows 40%. That simply means the first system is pretty weak and this system is very good at exporting to MPEG2-DVD. The second system probably has the better BFTB (Bang-for-the-buck) value of these two systems for exporting to MPEG2-DVD.

                             

                            Again, I emphasize that RPI is a great way to see overall results compared to similar systems, but you have to delve deeper and investigate individual test results to draw conclusions to help improve your systems performance where it matters most for your workflow.

                             

                            I hope this helps to explain the value - or lack thereof - of the RPI measurement.

                             

                            PS. The new PPBM6 results table (you have to be registered and logged in to see these results) shows these kind of rankings in the overview, although not on a scale of 1 - 100%, but as absolute scores ranging from first to last observation.

                             

                            New results.png