I needed to reinstall W7 on my wife's machine and with CS6 coming out, I deciced to do the same with mine. I now have a fresh system with CS6 installed. But should I be able to compare results that I got before by running PPBM5?
I wanted to see if CS6 ran the PPBM5 benchmark faster or slower than CS5 or CS5.5.
I'll give the CS6 benchmark a try this weekend. I looked at benchmark results and the page is not operational; not sure if this is because there have not been any submissions or they are not yet ready to activate the results function.
You can run PPBM5.5 on CS6. There are a few changes in the user interface that will cause you a headache or two. Remember that there are changes in CS5 and CS5.5 that significantly increased our PPBM5 MPEG2-DVD benchmark scores between those two. Harm is working on a new much more stressful PPBM6 that will significantly increase all PPBM6 benchmark scores.
Message was edited by: Bill Gehrke
Not completely finallized yet
Yes. However, Bill and I are still working on a new test for CS6 with enhanced functionality that is not yet ready. For the moment you can use the old PPBM5 test as I have added version 6.0 as an option to the submission form. We will inform everybody here when the new test is ready, but for a sneek preview look at http://ppbm6.com/index.html
Take into consideration that Bill and I have not yet decided on the download test file, the instructions and scripts are not yet ready, some other files that we envisage to include to make your and our life easier with data submission have to be added.
The major issue we have to solve is how to find the balance between statistical accuracy, relevance of the tests, download size and expected test times, even on lesser powered systems.
Thanks Harm and Bill for your responses.
There is a download file on the PPBM6 page. I take it that this file is not finalized and may not be your final test file. It sounds like there is no sense in running it at this time. (I started the download before you replied.) Would you even want an upload?
I'll try the older test out of curiosity.
This is my test version and contains all the material Bill and I have been using for the moment. However, while the rendering test remains unchanged (once with MPE On and once with MPE Off), the disk test remains unchanged (only with a longer timeline), there are in my intended setup for the H.264 and MPEG2 test two essential changes in comparison to the previous ones in the sense that when exporting, both MRQ (Maximum Render Quality) and Frame Blending need to be turned on. That will cause hugely longer test times than with the previous version and may be prohibitive for people with slower systems.
If you want to help us, send me a PM and I will tell you what test settings to use and how to report your results.
I just ran the PPBM5 benchmark on my main system that's now running CS6. My results (as compared to the results I had previously achieved with CS5.5):
1) The MPE On/Off times and the Disk I/O times got reduced: Instead of 6 seconds and 70-ish seconds (with MPE On and MPE Off, respectively), I got 4 seconds and 50 seconds, respectively. The Disk I/O time also went down from 60 seconds to around 52 seconds, as well.
2) But while the MPE rendering and the Disk I/O performance got faster, the MPEG-2 DVD and H.264 Blu-ray encoding performance became slower. Instead of 70 seconds for the MPEG-2 DVD portion of the test (which I believe is now a fluke because something else in the system is caching the output - and because the MPEG-2 DVD times nearly doubled with my particular GTX 560 Ti 448 when I reverted from an overclocked 4.4GHz back to the stock 3.4GHz) in CS5.5, I got a result of 113 seconds in CS6 (at the same 4.4GHz overclock level). The H.264 Blu-ray times increased by about 12 seconds (from 60 seconds to 72 seconds).
I also ran PPBM5 on my auxiliary i5-2400 rig also running CS6, with a similar proportion of performance increase or decrease compared to CS5.5. The disk I/O and MPE Off times decreased (in this case, to 62 seconds and 83 seconds, respectively, versus 77 seconds and 103 seconds) while the MPEG-2 DVD and H.264 Blu-ray times increased (to 176 seconds and 101 seconds, respectively, versus 153 seconds and 88 seconds).
I will be repeating the tests with the CPU in my main system back to stock speeds to confirm my findings. I have been suspecting that the MPEG-2 DVD encoding performance times of most of the CS5.5 systems with heavily overclocked i7 CPUs and LGA 2011 CPUs and the higher-end GPUs are artificially low (read: unrealistically fast) due to the caching issue that I mentioned near the top of this post.
Thanks for the information.
I got similar results running PPBM5.5 with CS6. Now that I know someone else's results, I am less concerned about my new install of Win7.
Harm and Bill will sort this all out assuming they have enough uploads to look at and can match our prior results to current results by computer name.
None of this may make any difference as they are working on PPBM6.
I have been suspecting that the MPEG-2 DVD encoding performance times of most of the CS5.5 systems with heavily overclocked i7 CPUs and LGA 2011 CPUs and the higher-end GPUs are artificially low (read: unrealistically fast) due to the caching issue that I mentioned near the top of this post.
I recalculated my old CS5.5 results, and discovered that the MPEG-2 DVD encoding performance became 40 percent slower (I consider 100 percent slower as "completely stopped" or "totally dead") when the CPU clock speed is reduced by only 20 percent. This clearly means that there is caching going on somewhere in the systems running CS5.5 and an overclocked or high-end CPU with 16GB or more system RAM plus a non-reference GF110-based graphics card. Take away the caching, and that 60-second result from the GTX 580 would have increased to about 85 seconds. (This is because the second half of the encode is cached, which reduces the encode time by as much as 30 percent compared to without the caching.)
By a similar token, the MPEG-2 DVD encode scores in most of the i7-39xx PCs (as well as those PCs equipped with a GTX 680) on the PPBM5 results list are also about 40 percent faster than they really are.
This weird caching issue during the PPBM5 MPEG-2 encoding test with CS5.5 does not occur with all GPUs. The GTX 470, the GTX 560 and the GT 240 that I have on hand all performed as expected with even a heavily overclocked i7-2600K (whose MPEG-2 score with the GTX 470 is only slightly faster than with the same i7-2600K CPU running at its stock 3.5GHz Turbo), where the GPUs with fewer CUDA cores and/or low VRAM bandwidth are all predictably slower than those with more CUDA cores and relatively high VRAM bandwidth.
I know there must bea good reason for this so forgive me Harm, why does the test run the same material twice? If the caching causes these kind of issues, wouldn't the test be more accurate if it didn't run the same material twice?
As to your testing RmL1903655, I have a GTX 470. I overclock my 2600k to 4.8/4.7. It runs cool and has never given me a problem. Running CS 5.5 gave me relatively high numbers on the PPBM 5.5 MPEG-2 encoding. CS6 is much worse on the MPEG-2 PPBM 5.5 benchmark. The other test results were very good running CS5.5 (compared to higher end systems even) but are now about 10 seconds higher running CS6 on disk and AVI than I got before. This appears to be in line with your testing.
I found a file of a fast benchmark for a first comparison:
Maybe we get first results there.
Interesting is the fact, that the GTX580 seems to be faster than the GTX680 in rendering raytraced CUDA-Tasks in AE CS6.
There are now two submissions with CS6 as of today.
We can not make the download size prohibitively large, so we have to re-use the same clips over and over. In the H.264 test we have made one timeline with all kinds of differences, but for the MPEG test we simply replicated the sequence twice, because for statistical reasons we wanted both tests to take roughly the same amount of time. This was of course when it was designed with CS5. The different caching algorithms in CS5, CS5.5 and now CS6 have shown the weakness of the original test,.
For CS6 we are working on a completely new timeline that eliminates these weaknesses and replication and improves statistical and measurement accuracy at the cost of a larger download and much longer testing times.
That benchmark is only about Ray-tracing with AE, nothing to do with PR. It does not mention if the 680 was hacked or that the test is only done with 6.01.
I would like to see where CS6 is faster or slower than CS5.5, and whether changes in hardware have a greater effect in one or the other CSx version (which computers to upgrade to CS6? Who deserves a new video card?). I am therefore highly appreciative of being able to use PPBM5.5 on both CS6 and CS5, and with different computers. In looking at PPBM5.5 in CS6, I did run into "There are a few changes in the user interface that will cause you a headache or two". Could you list the settings you wish us to use in CS6?
Also, I would like to plug for a "PPBM6 mini" test that can be used with older hardware (old=2 yrs!). A test that allows one to check the difference between a bleeding edge hedgefund-level computer and a mere Cray supercomputer will be of great use to those having such systems, but appears to be overkill for many of us sitting in the middle of the hardware bell curve. And we are the probably the ones that need your test the most.
You echo thoughts that Bill and I have, to keep the PPBM5 in operation for both CS5 and CS6 for the hedgefund-level and 'median' system, effectively all systems, but also offer a PPBM6 test for a mere Cray computer. A 'PPBM Short' test, like PPBM5 for an easy check and a more complete 'PPBM Long test', like PPBM6 similar to the tests you can run with CPU-Z. We are still investigating those possibilities.
In my thinking for CS6 (it may be impractical for users with low-end machines) we include two presets for the AME encoding of H.264 and MPEG2 in the download so that we know everybody uses the same presets, but they include MRQ and Frame Blending in the AME encoding process. This will lead to massive increases of encoding times. Think a factor 10 longer and we are discussing whether that will be a deterrent to run the test. It will give a better indication of the benefits of CUDA enabled cards, but with MXF and RED 4K marerial added and now 6 tracks plus an adjustment layer, the lesser systems may be completely overwhelmed.
CS6 results will come in and as these results grow in numbers, we can draw conclusions.
At this moment it is nothing more than an internal test file used by Bill and myself for testing purposes. We still need to add the readme file, the script files, the AME preset files, and the program files for collecting the hardware info. Just be patient. We will announce when it is ready and then make a much smaller file with the missing parts available for separate download too, for those who have already downloaded the source material, to avoid that you need to download the stuff you already have again.
Harm, in an earlier message you said, "For the moment you can use the old PPBM5 test as I have added version 6.0 as an option to the submission form". Where would I find that version - under Benchmark Instructions, I can only find a v.4 for PPBM5 and v.1 for PPBM5.5, and I don't see any downloads under the submission tab?? Thanks for all your hard work, we are all looking forward to PPBM6,
Europe, Middle East and Africa