Many have wondered about the long silence about PPBM6
Let me give you an update on where we are with the new benchmark and explain the reasons for all the delays. In addition I like to tell you about the dilemmas we are facing and the choices we have made and still need to make, so you understand the what's, why's and how's of the new site.
The current PPBM5 Benchmark test, while suitable for all versions of Premiere Pro CS5+ including CS6, has a number of limitations:
- The timeline for the MPEG2-DVD test contains three identical sequences of the H.264-BR sequence. The reason for that was to reduce the download size of the test file, but gives problems with accuracy of the results, due to caching differences between CS5, CS5.5 and CS6. That is the reason that the MPEG2-DVD results are far better with CS5 than with subsequent versions. Once the first instance of the H.264-BR sequence was cached, the subsequent instances where completely handled from the cache and that is no longer the case with CS5.5 and CS6.
- The submission form required entry of hardware data but that led to many practical problems. First, sometimes the questions were not understood, so instead of replying, as instructed "In GB for a SINGLE physical disk, NOT the formatted capacity. Example: 2000" people answered with 931 (which is the formatted capacity of a 1000 (GB) disk, or they answered "1 TB" instead of 1000, or they answered 6000, thinking that was the correct answer for a 6 disk raid array, although the word SINGLE was written in bold in the explanation. For the video card one would see answers like "quattro 4000" or "quadra 4000" or "QUADRO4000" and similar different spellings and use of capitals or lower case. This meant that each and every entry had to be checked by hand. Not very efficient.
- The method used to determine elapsed times is based on Windows capability to differentiate between the creation timestamp of a file and the modification timestamp of that same file. That is what the .vbs script does. However, the accuracy of Windows using those timestamp's is limited to 1 second. With many results for the MPE on test in the range of 4, 5 and 6 seconds, there is a significant chance of measurement errors, that are sizable, thus reducing the validity of the the results.
The new approach
With these limitations in mind, we had to figure out ways to avoid the caching problems, improve on the data submission issues and improve the statistical reliability of results. This is what we have done:
- Designed a new timeline that no longer has repeated sequences, thereby avoiding caching problems.
- (Partly) figured out a way to improve data submission and avoid typo's, while getting more, more accurate and reliable data about the hardware.
- Increased testing times in the new timelines to reduce measurement errors and to better show the benefits of MPE by turning on MRQ.
There are a number of drawbacks from this approach. The download file size increases significantly from 122 MB to 755 MB (RED 4K and 422 MXF material require a lot of space). The testing time for the whole test increases significantly and is much longer than with PPBM5. That is why Bill and I have been discussing the possibility to use this extended test and a lite test with much shorter testing times, and while this looks like a solid idea, it has several drawbacks. How can you compare test 1 versus test 2? How do you store those data and query them again? How do you present them? My personal feeling is that we continue using PPBM5 for the 'quick-and-dirty' test and the new PPBM6 project for more elaborate testing. Bill and I still have to decide on this.
I have been unsatisfied with the response times of the 'Narrow Your Search' page, but have not been able to improve that, due to my lacking experience with DW, .html, .php and the various plug-ins used. So I first decided to design a new website using Joomla! and that was the first challenge. Next I wanted to introduce 'Subscriptions' as a way to recuperate some of the costs we make with hosting this website and compensate for the costs of the plug-ins and extensions we have to use in order to make it all work as we want to, as well as a slight compensation for the countless hours/days/weeks we spend on making this happen.
Well, the groundwork has been done, the foundations laid but we have a long way to go. When designing such a new site you run into difficulties that are due to lack of experience with webdesign and webshops, lead one to use extensions that in a later phase show to have been the wrong choice altogether and you need to change extensions again. Stuff like that. Please keep in mind that this is a 'One-man-band' effort.
The major hurdles to take are the following:
- Access to various pages based on subscription level.
- PayPal payment module.
- Speccy .xml parsing before submission and enter the results into the MySQL database.
- Explode Output.csv and enter it into the MySQL database.
- Design the queries and pop-up info screens with the relevant hardware info.
- Link to external price-databases to create up-to-date BFTB charts.
I expect this to take another two months to accomplish, unless I find experienced Joomla! and .php programmers to help me out. Please volunteer.
If you are really curious to see the groundwork and fundaments of the new site, go to http://ppbm7.com
Don't expect a fully working site, it is in a development stage and lacks a lot of functionality. But it does show where we want to go with this new site. I appreciate feedback on the site as it is now, welcome suggestions to improve it further and if you have experience with Joomla! or .php and are willing to help out, please let me know. I can use all the help I can get.
Note that there is no .vbs script included in the download file for the time measurement yet, so all you can do with the download file is have a look at the timelines and try them out. The AME preset files are included, so you can get a feel for the extended testing times following the instructions on the website. The instructions are no longer part of the download. The registration module works.
Sorry for taxing your patience so much, but I hope you understand the difficulties and challenges I'm facing and why it is taking so long..
PS. Have a look at the third paragraph on this page for some disappointing news.