8 Replies Latest reply on Jun 21, 2008 9:58 PM by Oliver Goldman

    Air File I/O on large files

      I have an Air application that is downloading large (~5-15GB) files using custom chunked HTTP and writing to a FileStream. I've noticed that when the application runs, the memory usage on the system skyrockets. Checking the performance monitors shows that the air application itself is not using much memory, but it is generating a large number of page faults. The memory usage is high (>90% on 4GB system) until shortly after the application closes the FileStream, then it drops to like 20%.

      Under the covers, how is Air doing its file I/O? It appears like all free RAM is being used to buffer the file (memory-mapped I/O?). Is there any way to control this behavior? The system really grinds to a halt while the files are downloading.
        • 1. Re: Air File I/O on large files
          Oliver Goldman Adobe Employee
          Yes, AIR is pretty aggressive about reading as much as possible into memory. Which API are you using for your download?

          • 2. Re: Air File I/O on large files
            xwisdom Level 1
            it sometimes makes me nervous when I hear about how the poor performance and memory usage of AIR. I'm really wondering if an AIR can survive for long on the desktop. These memory issues and poor performance needs to be resolved real soon
            • 3. Re: Air File I/O on large files
              Oliver Goldman Adobe Employee
              When these problems lie in AIR itself, we're working hard to address them. For example, we made significant improvements to memory management in the 1.1 release.

              That said, many of these issues are in the application's hands. We can't keep memory usage down if an application allocates many objects; we can't keep CPU usage at zero if an application fires timers constantly. Performance is an end-to-end issue.

              • 4. Re: Air File I/O on large files
                billbrasky66 Level 1
                I'm using URLLoaders to fetch 4-16MB chunks of the file at a time.

                When the download starts, I open a FileStream to the output file. When the URLLoaders finish, I seek to the appropriate place in the file and write the loader's bytes, e.g.

                fs = new FileStream();
                fs.readAhead = 0; //we are not reading
                fs.addEventListener( IOErrorEvent.IO_ERROR, onFileError );
                fs.addEventListener( Event.COMPLETE, onIOComplete );
                fs.addEventListener( OutputProgressEvent.OUTPUT_PROGRESS, onOutputProgress );
                fs.openAsync( outputFile, FileMode.UPDATE );

                ... (later) ...
                fs.position = qw.position;
                fs.writeBytes( qw.data );

                The memory usage of the application itself is fine (like 120MB or so), so I don't think it's the URLLoader. System memory usage is high until I close the FileStream.

                • 5. Re: Air File I/O on large files
                  Oliver Goldman Adobe Employee
                  URLLoader stores all fetched data in memory until the download is complete. You might try using URLStream, which would allow you to write bytes out as they arrive. This minimizes memory use.
                  • 6. Re: Air File I/O on large files
                    billbrasky66 Level 1
                    I don't think the problem is the URLLoader. I think it's with the underlying code below FileStream.

                    I have a pause feature in the application that will allow the current URLLoaders to finish but doesn't start any new ones. After the pause is complete, there are zero URLLoaders active, but the FileStream is still open. In this case, the system is still showing heavy memory usage until I close the FileStream!

                    My guess is that FileStream is using memory-mapped I/O under the covers, mapping large amounts of file pages into memory. That's why it looks like the app isn't using much memory, but the system memory usage nears 100%.

                    • 7. Re: Air File I/O on large files
                      TheDude7563 Level 1
                      I use URLStream to download large files (2-3 GB) via AIR app, and I dump fetched data fo FileStream every 512kb using the progress event. My app uses about 30MB of memory on OSX/Windows XP. The CPU load on OS X is about 10-15%, and 3-5% on XP, but that is also due to the fact that I redraw some HTML every 500ms.
                      • 8. Re: Air File I/O on large files
                        Oliver Goldman Adobe Employee
                        If memory is being allocated by the system but not by the application then think the issue (if any) is with the operating system, not AIR. AIR does not memory-map files itself. However, it's still possible the OS is doing this in some cases.