13 Replies Latest reply on Feb 6, 2011 4:06 PM by OneOfTheseWillBeAvailable

    DPX sequence performance

    OneOfTheseWillBeAvailable

      Hi,

       

      I'm running Premiere Pro 5.0.3 under Windows 7 64bit on a Core i7 950 CPU with nvidia GTX470 graphics and a RAID that benchmarks at over 300MB/s.

       

      Running R210 AVI files is not a problem, other than all the usual problems that exist with R210 AVI files.

       

      Running DPX sequences representing exactly the same image data goes at about 5fps.

       

      I'm not very surprised; still sequences in Premiere have been slow since time immemorial, but I'm a bit disappointed by this.

       

      Since DPX is a format with a lot of options and I'm in the process of writing some DPX handling code, I'll list the characteristics of the files in question:

       

      DPX 2.0 per SMPTE-268M-2003

      XPDS (least significant byte first ordering)

      Total image size is 8192K per frame, padded from 7.91MB with 92160 bytes EOI padding on image element 0

      Left-to-right, top-to-bottom ordering.

      1 image element at offset 2048

      Packing method value 1, "Filled to 32-bit words, method A"

      Reference low code value is 0

      Reference high code value is 1023 (this is not a "core" value per the DPX spec, but is required by apps such as Apple Color)

      No encoding (value 0)

      1920x1080

      10-bit RGB (Image Element Descriptor value 50d)

       

       

      In other words an absolutely basic standard 10-bit RGB DPX as produced and used by absolutely everyone.

       

      What gives?

       

      P

        • 1. Re: DPX sequence performance
          SteveHoeg Adobe Employee

          My RAID is around 450MB/s and I can play 2K DPX in realtime. Not sure if yours is just slightly too slow.

          • 2. Re: DPX sequence performance
            OneOfTheseWillBeAvailable Level 1

            It might be slightly too slow, but it's not five times too slow.

             

            And in any case, an R210 AVI is a very similar, if not actually more difficult, load.

             

            Also, the load works out at 200MB/s, which is less than it benchmarks.

             

            And Blackmagic Media Express will read and write DPX files to it.

             

            So no, it isn't too slow. Any other ideas?

             

            (By the way, I apologise for the ridiculous user ID - it was about the only thing I could find that wasn't in use)

             

            Best,

             

            Phil

            • 3. Re: DPX sequence performance
              Jim_Simon Level 8
              I apologise for the ridiculous user ID - it was about the only thing I could find that wasn't in use)

               

              You could have used your real name.  Many of us here do.

               

              By the way, what's an R210 AVI file?

              • 4. Re: DPX sequence performance
                OneOfTheseWillBeAvailable Level 1

                Actually I couldn't, That's the first thing I tried!

                 

                Anyway.

                 

                R210 is the 10-bit RGB uncompressed codec commonly used when working with video from things like Blackmagic and Aja boards. The Blackmagic implementation, which is by far the most common, has a glitch in the decoder which causes a rather complex luminance scaling behaviour and makes it unusable. It is actually quite widely used, because the behaviour is subtle and only readily detectable with test charts, but the thing is broken and horribly so. Blackmagic know about this and don't seem to care, which is why I wanted to go to DPX in the first place.

                 

                The point, as far as this discussion goes, is that an R210 AVI is, with audio, actually a larger disk loading than a DPX file. And Premiere Pro will read R210 AVI on my computer. But it won't read DPX.

                 

                For a long time it was fairly clear that Adobe didn't intend stills sequences to be used for realtime playback - since they were pushing DPX so heavily I rather hoped that had changed. To read stills sequences efficiently requires a little thought put into things like buffering and possibly multi-threaded reading, which is work I suspect Adobe have not done.

                 

                Best,

                 

                P

                • 5. Re: DPX sequence performance
                  Jim_Simon Level 8

                  Ah.  Got it.

                   

                  I knew the codec used by AJA is V210.  Didn't know BM had their own as well.

                  • 6. Re: DPX sequence performance
                    SteveHoeg Adobe Employee

                    The import is multi-threaded but requires relatively little processor power as it is uncompressed anyways. Work went into 5.0.2 to make 2K DPX playback realtime. If that isn't working for you then some part of your system isn't keeping up as lots of others have had success. As DPX is most sensitive to disk speeds that is the obvious answer but it is hard to say.

                    • 7. Re: DPX sequence performance
                      OneOfTheseWillBeAvailable Level 1

                      Jim,

                       

                      V210 is the uncompressed 10-bit YUV version. Aja I think have R10K which I believe avoids the luminance clipping issue endemic to Blackmagic, and a few other technical conveniences. Uncompressed 8-bit formats are well served by preexisting 8-bit fourCCs (HDYC and UYVY for YUV, usually RGBA for RGB).

                       

                      Steve,

                       

                      It is not disk speed as other software (including my own code) will happily read and write 1080p24 10-bit RGB DPX files on this system all day long, and at rates always in excess of 40fps. The problem is is not the disks, the problem the way Premiere is trying to read them.

                       

                      I don't think it's my DPX files either, as Premiere won't even read it's own output (which is not padded to any particular length) at anything other than a crawl.

                       

                      Could you please find out what if any assumptions are made in the DPX reader with regard to file size, alignment on disk and in memory, disk cluster and block size end-of-image padding and its placement on the various image elements? Also whether this was tested with anything other than 2K, because if the reader relies on alignment and buffering characteristics of a very precise file size, that would also cause breakage.

                       

                      Something is broke, and I'd like to either fix it my end, help you fix it, or at least have something worthwhile to say to other people when I get asked about this.

                       

                      Thanks,

                       

                      P

                      • 8. Re: DPX sequence performance
                        SteveHoeg Adobe Employee

                        2K is an approximate upper limit, nothing exclusive. Send me a single one of your DPX files, I will duplicate it and see how well it plays here.

                        • 9. Re: DPX sequence performance
                          OneOfTheseWillBeAvailable Level 1

                          Hi,

                           

                          Here you go:

                           

                          http://www.philrhodes.com/Test0000.dpx

                           

                          Feel free to try, but the speed issue is the same with DPX that Premiere itself creates, so if there is any problem with the file I don't think it's causing the overarching problem.

                           

                          P

                          • 10. Re: DPX sequence performance
                            SteveHoeg Adobe Employee

                            An image sequence of those will play without dropping frames for me. If you are confident your hardware is fast enough then I'm not really sure what else to suggest.

                            • 11. Re: DPX sequence performance
                              OneOfTheseWillBeAvailable Level 1

                              I tried putting the disks in RAID-0, just for the sake of a test. This 

                              provided over 1000MB/second of measured throughput, and Premiere did, 

                              finally, managed to play back a 1080p24 10-bit RGB DPX sequence in 

                              realtime.

                               

                              Remember that I found that other applications which would play DPX in 

                              realtime on the previous disk configuration, which measured about 

                              350-400MB/second. This makes sense, because a 1080p24 RGB 10-bit DPX file, 

                              even if padded to 8MB per frame, requires 192MB/second.

                               

                              This implies that Premiere's DPX reader is horribly inefficient. It 

                              requires at least twice, and very possibly five times, the amount of 

                              bandwidth that it really should. It reads, at two or three frames per 

                              second, a sequence that my own single-threaded and very naive development 

                              code will read and write at around 40. This is seriously crappy 

                              performance.

                               

                              On the basis of this I would like to suggest that whatever work was done 

                              on the DPX reader in recent versions is in urgent need of further 

                              refinement. I think users are entitled to assume that a RAID capable of 

                              350-400MB/second is capable of running a 1080p24 10-bit RGB DPX sequence, 

                              and if Adobe want to legitimately advertise that feature, they should 

                              ensure that it can.

                               

                              P

                              • 13. Re: DPX sequence performance
                                OneOfTheseWillBeAvailable Level 1

                                I filed it as a bug earlier.

                                 

                                I think I'm being reasonable. I mean, a 500MB/second RAID that won't run 

                                1080p DPX - but it will run 1080p R210 and R10K AVI files? That's... um... 

                                highly questionable?

                                 

                                P