28 Replies Latest reply on Apr 1, 2013 12:48 PM by Harm Millaard

    Interested in your opinion about PPBM7

    Harm Millaard Level 7

      Bill and I are working on a new site and a new test. It looks like it will not take much longer and is nearly ready for CS6.

       

      We have invested lots of time in designing the new website and adding functionality and there is still as long way to go, but we are making steady progress. Note that the subscription module in not operational yet, the download file is not up-to-date, the script is not included, etc. but we would like to get your opinion on our progress and hear your comments. Everything now is only based on our PPBM5 results.

       

      Please take a careful and critical look at PPBM6, take a look around the site and consider the navigation, layout, logic, design and - most importantly - how easy it is to find the info you are interested in and the way it is presented. Please be critical, so we can take your considerations and suggestions into account.

       

      After you have looked over this site, you would help us enormously by filling out this survey: Click here to take survey. Note we have only a basic subscription which allows for 100 responses only, so you may be out of luck if you wait too long.

        • 1. Re: Interested in your opinion about PPBM7
          Jim_Simon Level 9

          Looks pretty nice.  I like that the site doesn't seem to 'require' JS to function.

          • 2. Re: Interested in your opinion about PPBM7
            Harm Millaard Level 7

            Bump. Almost 120 reads, only a single reaction and no survey entries yet. No wishes, no changes desired, no additional info to be added?

            • 3. Re: Interested in your opinion about PPBM7
              Jim_Simon Level 9

              Perhaps you've just done the job very well.

              • 4. Re: Interested in your opinion about PPBM7
                Nicol Simard

                I have a couple of suggestions/questions. In the right column, you always add one option "All pages". I am not certain that it is necessary. And in the "Latest News" section, you've got the "Planning and building a NLE system" twice on the page. Once on the right side and once in the left column. Why twice? The one on the left does not follow your usual page layout.

                 

                To me, the "Planning and building" section is important enough to be a seperate section in the main menu. From there, you could continue with your usual logical arrangement.

                • 5. Re: Interested in your opinion about PPBM7
                  Harm Millaard Level 7

                  The 'All Pages' option is to circumvent the otherwise required top right hand navigation to go to the next page. The articles all have multiple page breaks in them, so you can just scroll down and then press 'Next >>' at the bottom of the page' to navigate to the next page, or use the 'scroll up' icon in the bottom right corner to go to the top of the page and select another page from the top right menu, or you can display all pages but with a lot of scrolling.

                   

                  "Planning and building" really does not belong on a benchmark site, but I considered it worthwhile enough to put it on this site. However, it turned out to be so long and covers so many pages (32 pages in total), that I made the bottom left hand menu to enable relatively quick navigation, instead of cluttering the main interface and menu.

                   

                  I appreciate your feedback and critical look at our site. We can only improve its worth by having feedback like yours and our acting upon it. I will have a serious look at how to improve that part of the site navigation. But maybe even more important for me, in comparison to the static information given on the 'Planning and building' part of the site, is how people perceive the dynamic content and especially the graphs used to visualize results.

                   

                  I'm trying out various graphs, gauges, Box-and-Whisker plots, error bars, multi-axis charts etc. but wonder if I'm not going overboard with statistical information that most people either do not understand, is too complex, or do not need. Look at http://ppbm7.com/index.php/latest-news?showall=&start=2 and consider if the message in the graph is instantly clear to you, or does it require more study than you are willing to spend?

                   

                  We have been lucky to have found a sponsor to help us with our charts, as shown in the right bottom of the footer area on each page, but we have to choose a format that appeals to most of our visitors and gives them the best 'bang-for-the-buck', so to speak.

                   

                  BTW, you did not fill out the survey yet.

                   

                  Anyway, thanks for your critical and constructive thoughts. I hope more people will follow your example.

                  • 6. Re: Interested in your opinion about PPBM7
                    Jim_Simon Level 9

                    wonder if I'm not going overboard with statistical information that most people either do not understand, is too complex, or do not need.

                     

                    I would say yes.  Meaningful benchmarks are run with as few variables as possible, ideally changing only one item at a time.  You have sooooo many variables with all user submissions that meaningful comparison simply can't be done.  The chart has little value other than bragging rights, so adding statistics seems a waste of time.

                     

                    Now, if Bill wants to make a separate chart for all the results he get's from swapping out hardware on the same machine, that's a chart worth seeing.

                    • 7. Re: Interested in your opinion about PPBM7
                      Nicol Simard Level 1

                      I just filled the survey. When I first looked at the site, I was in bed with my iPad and most of the graphs did not show up. I started going over your web page again and I like the graphs so far but will go through them more diligently. One suggestion comes to mind already: when you use a small graphic, allow people to download a more detailed version.

                       

                      On the page "Background Information: MPEG2-DVD Test", you use a graphic to show that adding more memory is important. But it is small and hard to read. I will go through all your graphs and get back to you in this thread.

                       

                      As for the VideoCard Performance chart, it is not too easy to read but not too hard to frustrate me. But that's just me. You need more feedback but people do not seem to be ready to give a proper critical look at your site. A critical look does not mean you do not like it, it means making suggestions that you think could help.

                       

                      Oh well.

                      • 8. Re: Interested in your opinion about PPBM7
                        Nicol Simard Level 1

                        Trying to reach the Contact page gives us an Error:404. I forgot to mention that earlier.

                        • 9. Re: Interested in your opinion about PPBM7
                          Harm Millaard Level 7

                          Please refresh the page. It should be solved now. Thanks for your survey entry.

                           

                          Nicol, you have touched on a problem we have with this site, or better said iPad and iPhone owners have with the lack of Flash support on their devices, and that is that the charts are created in Flash and that does not work on an iPad. I'm currently looking into replacing these charts with JS and HTML5.

                          • 12. Re: Interested in your opinion about PPBM7
                            Nicol Simard Level 1

                            I noticed something strange looking at the last two cards in the graph: the GTX550ti and the GT640. The last one has more cores and a greater bandwidth than the 550ti and yet gives much worse results.

                            • 13. Re: Interested in your opinion about PPBM7
                              Harm Millaard Level 7

                              Memory bandwidth of the GT 640 is 28.5 MB/s and is displayed as (250 - 28.5) = 22 MB/s

                              Memory bandwidth of the GTX 550 Ti is 98.4 MB/s and is displayed as (250 - 98.4) = 152 MB/s

                               

                              So interpreting this chart is not vey easy, is it?

                               

                              PS. There are three different GT 640 cards, but one uses only 144 cores, and then there are the two remaining ones with either DDR3 or DDR5 and in this case Bill only had the DDR3 model.

                              • 15. Re: Interested in your opinion about PPBM7
                                Harm Millaard Level 7

                                That is what I was afraid of, it really takes careful study in the current form and as Jim said, there is sooo much information to present that I'm really struggling with a way to present all these data in an easy and comprehensible way so you can all benefit from it. FYI, Bill was overwhelmed as well by the amount of data in these charts.

                                • 16. Re: Interested in your opinion about PPBM7
                                  Jim_Simon Level 9

                                  That is what we have here: Video Card Performance

                                   

                                  Nice.  Except, this I find confusing.

                                   

                                  "so the grey line is similar to time, lower is better in the display."

                                   

                                  Bandwidth is normally expressed such that higher is better.  I think you should keep it that way in the chart.  Post the actual bandwidth, not a difference.

                                   

                                  You can post multiple scales on the same side of the chart if need be, each in the color of the line or bar that the scale represents.  I think this would be far easier for most folks to read.

                                  • 17. Re: Interested in your opinion about PPBM7
                                    Harm Millaard Level 7

                                    Multi-axis charts is not the problem, it is relatively easy to create charts with multiple axis, even where you can toggle them on or off, but I have not yet figured out a way to use multi-axis in combination with inverted axis or logarithmic axis, where one axis starts at 0 at the bottom and the other axis starts at 0 at the top of the chart or one is linear and the other is logarithmic.

                                     

                                    I'm looking into Waterfall charts that may make it easier to understand. I'll keep you posted and may want to pick your brain on this. Thanks for the comment.

                                     

                                    I'm experimenting with this presentation form: Individual Performance

                                    It is only static and fictional at the moment, but is it readable and worthwhile. What do you think?

                                    • 18. Re: Interested in your opinion about PPBM7
                                      JEShort01 Level 4

                                      Harm and Bill,

                                      Thanks so much guys for continuing - and even ramping up - your PPBM efforts! I collected my thoughts and was ready to paste them into the survey page, but since Survey Monkey is not really set up for general comments, I'm pasting them here instead.

                                      • STRONG OPINION - Put Benchmark Results/Detailed Results (full table) by itself and label it "Benchmark Results Table" and move all other items in Benchmark Results section to "Benchmark Results Statistics"
                                      • Benchmark Results/Detailed Results table features don't work at all in Internet Explorer (I have IE v9); works fine in Firefox, but many Win7 users will be using IE
                                      • Would like to see new columns in Benchmark Results / Detailed results table: CPU die size (nm), # cpus (like in PPBM5), chipset (i.e. X79, C602, etc.), User Comments (i.e. so DeeDigital could explain why identical hardware went from score of 186 to 149), Disk Comments (i.e. many forum posts focus on where put scratch, projects, media, render outputs, etc.)
                                      • STRONG OPINION - Menu design should be streamlined; This is not that large a site and When "Latest News" "Planning and Building" is up, I see 4 separate menus (at page top, Main Menu on Left, Latest News...All Pages on right, and Planning & Building a NLE System at lower left.

                                        Suggestions to streamline:
                                        1) Only show "All Pages" for all areas; if you find you must have this level of sub-section, do it with a drag-down box at top and bottom of the section like many review sites do
                                        2) Have all menus at top; this is nicer too when someone wants to print something out
                                      • Need to clearly state what versions of CS the new benchmark is compatible with
                                      • Suggest adding a lifetime Platinum subscription option for say $100 or $150; that's the one I would select BTW if it were available
                                      • Also suggest dumping the Bronze and Gold options; there too many options for new subscribers
                                      • Update copyright to 2013 (i.e. bottom Methodology page)
                                      • "Buy a subscription" should be "Select a Subscription" since entry level is free
                                      • Remove the "We are volunteers!" banner to be consistent with new paid subscription fees; or, if the fees are purely going to internet fees, etc., explain that somewhere
                                      • Font size selector is broken on Buy a Subscription page
                                      • Suggest using www.ppbm6.com instead of www.ppbm7.com to reduce confusion (it seems that you also own this domain site)

                                      Keep up the great work, and thanks again!

                                      Regards,

                                      Jim

                                      • 19. Re: Interested in your opinion about PPBM7
                                        Harm Millaard Level 7

                                        Jim,

                                         

                                        Thank you for your very constructive thoughts. Taking all your remarks into consideration, I have started adjusting the page and the template. I'm not there yet, but I think the approach is about what you wanted and intended.

                                        • 20. Re: Interested in your opinion about PPBM7
                                          Jim_Simon Level 9

                                          I have not yet figured out a way to use multi-axis in combination with inverted axis or logarithmic axis

                                           

                                          What would the logarithmic axis be used for?  There are only three measurements - time, bandwidth and number of cores.  All three are linear, and only time needs 0 at the top.  So...cores and bandwidth on the left, time on the right (or vice versa).  Much easier to read.

                                           

                                          Also, make all three bars, not a combo of lines and bars.  So any given card will have three bars, one for time, one for memory bandwidth, one for number of cores, each bar a specific color relating to it's scale.

                                           

                                          Sort them on the chart according to time, as that is the most relevant stat.

                                           

                                           

                                          is it readable and worthwhile.

                                           

                                          Readable yes.  But worthwhile would depend on what you intend to use the gauges for?

                                          • 21. Re: Interested in your opinion about PPBM7
                                            Harm Millaard Level 7
                                            STRONG OPINION - Put Benchmark Results/Detailed Results (full table) by itself and label it "Benchmark Results Table" and move all other items in Benchmark Results section to "Benchmark Results Statistics"

                                             

                                            Done.

                                             

                                             

                                            Benchmark Results/Detailed Results table features don't work at all in Internet Explorer (I have IE v9); works fine in Firefox, but many Win7 users will be using IE

                                             

                                            Try with the new template, it may work now. Still some cosmetics to adjust though.

                                             

                                            Would like to see new columns in Benchmark Results / Detailed results table: CPU die size (nm), # cpus (like in PPBM5), chipset (i.e. X79, C602, etc.), User Comments (i.e. so DeeDigital could explain why identical hardware went from score of 186 to 149), Disk Comments (i.e. many forum posts focus on where put scratch, projects, media, render outputs, etc.)

                                            That may be feasible when I get the parsing of Speccy.xml completed.

                                             

                                             

                                            STRONG OPINION - Menu design should be streamlined; This is not that large a site and When "Latest News" "Planning and Building" is up, I see 4 separate menus (at page top, Main Menu on Left, Latest News...All Pages on right, and Planning & Building a NLE System at lower left.

                                             

                                            Done.

                                             

                                            Suggestions to streamline:
                                            1) Only show "All Pages" for all areas; if you find you must have this level of sub-section, do it with a drag-down box at top and bottom of the section like many review sites do
                                            2) Have all menus at top; this is nicer too when someone wants to print something out

                                            Done.

                                             

                                            Need to clearly state what versions of CS the new benchmark is compatible with

                                            Will be done.

                                             

                                            Suggest adding a lifetime Platinum subscription option for say $100 or $150; that's the one I would select BTW if it were available

                                            Also suggest dumping the Bronze and Gold options; there too many options for new subscribers

                                            I'm looking into that.

                                             

                                             

                                            • Update copyright to 2013 (i.e. bottom Methodology page)
                                            • "Buy a subscription" should be "Select a Subscription" since entry level is free
                                            • Remove the "We are volunteers!" banner to be consistent with new paid subscription fees; or, if the fees are purely going to internet fees, etc., explain that somewhere

                                            Done, but working on the template now for the footer.

                                             

                                            • Font size selector is broken on Buy a Subscription page
                                            • Suggest using www.ppbm6.com instead of www.ppbm7.com to reduce confusion (it seems that you also own this domain site)

                                            Will look into that.

                                            • 22. Re: Interested in your opinion about PPBM7
                                              JEShort01 Level 4

                                              Fantastic stuff Harm!

                                               

                                              You've sure been busy - thanks!!! I really like the changes, and I think that first timers to the site will be able to navigate it easier as well.

                                               

                                              Jim

                                              • 23. Re: Interested in your opinion about PPBM7
                                                Nicol Simard Level 1

                                                I just looked at the Benchmark results page. First problem: the page header says "PPBM6 RESLUTS" instead of "RESULTS".

                                                 

                                                Second problem: when doing an advanced search, the element section is not sorted by alphabetical order. Video Cards appears between "Clock Speed" and "Size OS Disk".

                                                • 24. Re: Interested in your opinion about PPBM7
                                                  Harm Millaard Level 7

                                                  Thanks Nicol. That typo has been corrected. The order in which elements appear in the advanced search section depend on the sequence in which they appear in the table view. Since clock speed is even more important than the video card, as long as they are CUDA capable, I have chosen to show clock speed before video card and that is the reason clock speed shows up before video card. I'm not sure fabrik allows me to sort elements in the advanced search by alphabet, but I'll look into it.

                                                   

                                                  PS. Non-published elements can be resorted, but published items can not, unless you also change the sequence in the table. The logic behind the order is not only impact, but also the fact that # Cores, # RAM and Clock Speed belong together, and Video card is a separate element.

                                                  • 25. Re: Interested in your opinion about PPBM7
                                                    Nicol Simard Level 1

                                                    I had noticed that the elements appeared in the same order as they are in the main list. However, it would be easier to find them just in an alphabetical order. But I know you will do your best, so I will not harp on that point.

                                                    • 26. Re: Interested in your opinion about PPBM7
                                                      Harm Millaard Level 7

                                                      With the step I made yesterday, inspecting the differences between systems is now a lot easier. Instead of looking at hard to read .xml files, several early adopters have submitted their Speccy snapshot files (instructions and submission form have been updated to reflect this new move) and hopefully the rest will follow suit shortly.

                                                       

                                                      The results page now looks like this:

                                                      Speccy results.png

                                                      After opening a snapshot file and moving the windows around a bit you can easily compare multiple systems in detail.

                                                      Speccy details.png

                                                      and with sufficient screen space even like this:

                                                      Speccy details 3.png

                                                      This should be a lot easier than those .xml files.

                                                      • 27. Re: Interested in your opinion about PPBM7
                                                        RjL190365 Level 4

                                                        Thanks, Harm. I had been guessing what the XML content was. Now, the .speccy files made it much easier to compare another system with the system I'm currently using.

                                                         

                                                        One major note: To avoid the risk of having unbalanced systems wrecking the rankings, the user should not even bother submitting PPBM6/7 results if he does not have a CUDA GPU: Such GPUs are locked to the MPE software only mode. The user should stick with the old PPBM5 test if he does not have a CUDA GPU.

                                                        • 28. Re: Interested in your opinion about PPBM7
                                                          Harm Millaard Level 7

                                                          Added that remark on the Home page and a rather obvious warning button on the submission form with a link to PPBM5. Thanks for the tip.