5 Replies Latest reply on Mar 24, 2008 5:10 PM by Bayani Portier

    Chunked uploads

    flexsea
      Hi,

      I need to upload very large files to a remote service, which is accessible using HTTP.

      One option would be to sequentially send chunks of the large files using HTTP POST. While sending a file using a single HTTP POST ist standard Flex functionality, I wonder how to manage the chunked method.

      Since sending uses URLRequest and FileReference, an idea would be to subclass FileReference to only provide chunks of the original file. But how do I get e.g. efficiently chunks of a file?

      Any ideas, if this is a good strategy? Or any hints, how to manage this task somehow else?

      Kind regards,

      s

      PS: Using URLRequest/FileReference using very large files (300 MByte), makes the flex app hang for a long time somewhere in runtime. Even a registered progress control appears after minutes...
        • 1. Re: Chunked uploads
          Bayani Portier Level 1
          Hi there flexsea,

          I've been thinking about the same thing for quite a while. To avoid the large overhead, and to allow for other funky functionality like compression, I've been leaning towards manually chopping up the file into an array of raw bit streams of an arbitrary chunk size. The larger the chunks, the more compression you'd get, but the more data that would be lost between connections. I've been thinking about that algorithm too, but haven't had enough time to adequately crack it.

          The server would also take a reasonable hit decompressing and reconstructing the file on the mid-tier. Depending on the app, it may not be a good idea to do so, as unless you are processing the data on the server, it should remain in chunks, and be downloaded by the destination interface, meaning that inflation and reconstruction would occur client side.

          I would love to hear everyone's further thoughts on the matter.

          Regards,

          Bayani
          • 2. Chunked uploads
            flexsea Level 1
            Hi Bayani,

            quote:

            To avoid the large overhead, and to allow for other funky functionality like compression, I've been leaning towards manually chopping up the file into an array of raw bit streams of an arbitrary chunk size.
            .

            Would managing the file's byte stream manually be even a got strategy? Is the Flash VM fast enough to do a reasonable job? At least, the VM shouldn't get the big memory footprint, which I assume to be the reason for the 'hang'.

            Compression would be a nice mix-in later on. For now, a simple file transfer would be OK. Speed isn't a major issue here, since I'd transfer the big file to a central (LAN-based) server-part, which will in turn forward it to central infrastructure.

            quote:


            The larger the chunks, the more compression you'd get, but the more data that would be lost between connections. I've been thinking about that algorithm too, but haven't had enough time to adequately crack it.


            Kind regards,

            s

            PS: Sorry for this dump question, I started using Flex today ;-)

            • 3. Re: Chunked uploads
              Bayani Portier Level 1
              As to the memory hit, depends on how you implement it. If you have a pointer on the bit stream as to the current position of the file, and pull out only the chunk, compress and save to the local file system, your memory should be significantly lower. When sending, you would simply iterate through the file chunks one at a time.

              And yes, you are absolutely correct. If you flood flex with high variable usage, it does chunk because the windows Flex VM is not exactly efficient. (Check the task manager when you're profiling a significant application, minimize the screen and then maximise it again).
              • 4. Re: Chunked uploads
                flexsea Level 1
                quote:

                As to the memory hit, depends on how you implement it. If you have a pointer on the bit stream as to the current position of the file, and pull out only the chunk, compress and save to the local file system, your memory should be significantly lower. When sending, you would simply iterate through the file chunks one at a time.


                OK, I'll give this a try.

                quote:

                And yes, you are absolutely correct. If you flood flex with high variable usage, it does chunk because the windows Flex VM is not exactly efficient. (Check the task manager when you're profiling a significant application, minimize the screen and then maximise it again).


                OK, I'll do.

                BTW: Running in Safari 3.1 on OS X, in the described situation, the Flex app hangs for quite some time. Using Firefox, the delay is much slower.

                Thx!
                • 5. Re: Chunked uploads
                  Bayani Portier Level 1
                  The best advice I could give you is to run the profiler on the app, and see where the slowdown occurs.

                  And yes, WE NEED TRUE MULTITHREADING! (Just yelling so that the Adobe guys might hear.) I could go on and on as to the GUI issues that this would solve...