12 Replies Latest reply: May 9, 2012 9:59 AM by Flex harUI RSS

    Memory related crashes in Flex application

    ron_bresler

      We have a relatively heavy flex application running in a browser.

       

      The Flash verison is 11.2 and Flex SDK is 4.1. Starting a while back we started having two kind of crashes often:

      1. the GCOD - a grey circle with white exclamation mark that means too much memory is being used, leading flash player to commit suicide to protect the browser.

      2. the browser becomes inactive for a while and then crashes, or recovers (but most often crashes).

       

      We know we have a lot of memory leaks, and in some cases the crash is justified by memory usage. But, in some cases we record crashes without a high memory usage. I cannot explain these crashes. But we also know that in some cases Flash anticipates a large memory allocation and crashes immediately: this is the cases where, for instance you try to assign a negative value to the length attribute of a vector (i suspect the vector attempts allocation of the length size after an unsafe type convertion into a uint - resulting in a huge vector that cannot be held in memory). I have extensively searched our code for such access to the length attribute of Vector and could not find any.

       

      Our measurements, assuming they are correct, lead to think that we have two kinds of different causes: one is plain memory leaks which lead us to the crash, and the other is an unknown cause that probably leads the flash player to trigger the GCOD with no real reason.

      I have two questions:

       

      1. How can I programatically anticipate the CGOD? is there a calculation I can apply to available memory metrics from within Flex applications so that the user can be warned to save all data before the crash in case the application leaks slowly?

      2. What other possible bugs or mechanisms can trigger the GCOD without good reason? Since the negative length bug is not the cause in my case, maybe there are other usages that may lead to abrupt and inadvertent memory allocation?

       

      The crashes are very frequent... Any help will be most welcome!

       

      Ron

        • 1. Re: Memory related crashes in Flex application
          drkstr_1 Community Member

          Have you used the profiler in Flash Builder at all?

           

          It is impossible (for me at least) to anticipate all possible bottlenecks ahead of time. I always use the profiler to identify and fix these issues, and as a result, I do not experience any memory related bugs in my apps. I can't imagine building a large scale application without the use of a profiler.

           

          If you are having trouble using the profiler to identify your bottlenecks, post back and I would be happy to give some suggestions.

           

           

          Cheers!

          • 2. Re: Memory related crashes in Flex application
            ron_bresler

            Thanks. I did mean to ask if we can anticipate the crash in runtime - that is - to detect the high consumption and alert the user just before so she can save her data.

             

            Also, this does not address the second issue - we are receivine the GCOD even when memory is relatively low. this cannot be a result of a memory leak, so a profiler probably will not help here.

             

            That said - We are trying to make use of the profiler but have had extremely bad experience with it. First, it crashes more often than not. this is something we're working on to resolve - even when profiling an empty project with close to no code at all in it - we still crash when trying to navigate the memory allocation stack and reference chains.

            Second, our application is way too large to be analyzed as is - I assume the best way to profile it is to cut it up into pieces - and that will also take some time. Lastly, we have had some success with the profiler in the past - but most of the time I just don't know how to use it.. I have read about different techniques, but most of the time I cannot find anything wrong with my classes or code. the main bulk of memory is hidden in classes of Function and BitmapData

             

            Function - could indicate abuse of Binding or event listeners, and the BitmapData - could indicate a leak of styles or background images in containers. I could not find a way to effectively use the Profiler to identify the source of these leaks. and the fact that it's usually not working at all is discouraging. Any tips or suggestions will be greatly appreciated.

             

            Let's start by saying that the application consumes too much memory. much more than it should. once we finish loading all code the application typically weighs about 400MB, which can jump up to 1GB without loading too much media (just some graphics occasionally) - as measured by task manager (including the browser and plugin)

             

            There are many "best practices" I've read about that are intended to help cope with memory cleanups - implementing IDistroyable to remove event listeners, renferences, styles, images and URLLoaders - numerous advice and warnings regarding ItemRenderers etc..  I've been avoiding these advice since it seems unreasonable that i should have so much code that does basically nothing except what is expected from the framework. dealing with these cleanups should be transparent and done by the framework/VM. it was my hope that the framework will just mature to handle these issues. I just keep the links and references in my code as simple and straightforward as possible hoping this should be enough - As I would in Java.

             

            Another issue is the depth of our UI tree. i was hoping it affects only CPU in event propagation and size calculations - but now I start suspecting that it also has memory implications (perhaps each container keeps BitmapData even though the bitmap cache is set to false by default?...).

             

            Any thoughts or tips would be appreciated. But I stress - even if we clear all memory leaks and reduce consumption - it appears that the GCOD appears sometimes when memory use is low..

             

            Thanks!..

            • 3. Re: Memory related crashes in Flex application
              Flex harUI Adobe Employee

              You can try using System.totalMemory to anticipate a crash.  It won’t reflect the memory shown by the task manager, but it will probably be proportional.

               

              That said, there is usually a price to be paid for not employing best practices.  If the profiler is barfing, that generally implies that you have simply created too many objects.  A well written large application is modular and doesn’t create things until just before it is needed.    The modules can be individually tested or at minimum, instantiated without instantiating every other module.  And a modular app is usually more efficient to write and maintain:  you don’t have to compile the whole thing for every change and different developers can work on different modules with fewer collisions.  It is probably worth it for you to refactor your app now.  Then the Profiler can help you find the other remaining problems.

               

              There is a “tutorial” on the Flex 3 profiler on my blog.  I haven’t updated it for Flex 4, but the principal is roughly the same.

               

              Due to the nature of Flash Player memory management, it is often better to use object pools instead of creating and destroying lots of objects.

               

              There are some questionable recommendations on how to manage memory better, but fundamentally, the issue is whether there are strong references to objects from the “GCRoots”.  It is probably worth your time getting a better understanding of that so you don’t waste time nulling out references that don’t matter.

               

              I’m sure there might be a bug or two involving memory leaks in the framework, but for the most part, the framework uses weak references correctly and does not leak memory despite what rumors may say.  But it is easy to use the framework in a way that causes your code to get stuck and that’s why understanding how the reference chain from GCRoots work is important to understanding what code you do need to clean up.

               

              The depth of the tree for most people doesn’t matter, but flatter is better.  If you are getting out past 30 levels, then that is likely to be a drag on performance.

               

              The most common way to crash the player besides consuming too much memory is to overflow the stack.  Sometimes you will get a stack overflow error, but quite often, the player disappears without any notification.  There is no control over maximum stack size and available stack can vary by player/browser.  There is no way to know how much available stack space there is.  I’ve only seen the stack overflow when I have a bug:  I have a function that eventually calls back into itself (i.e. Infinite recursion).  I’ve seen a few folks claim that they overflow in some complex sequence of calls that does have a cap.  That is certainly possible, and code that walks a deep and complex display object tree is going to be more prone to that condition.

               

              If an empty app crashes the profiler, make sure you restart Flash Builder before starting the profiler.  If you are almost out of heap in Flash Builder the profiler is more likely to fail.

               

              --

              Alex Harui

              Flex SDK Team

              Adobe Systems, Inc.

              http://blogs.adobe.com/aharui

              • 4. Re: Memory related crashes in Flex application
                drkstr_1 Community Member

                That's some good info from Alex there.

                 

                The only thing I would like to add...

                 

                There are many "best practices" I've read about that are intended to help cope with memory cleanups - implementing IDistroyable to remove event listeners, renferences, styles, images and URLLoaders - numerous advice and warnings regarding ItemRenderers etc..  I've been avoiding these advice since it seems unreasonable that i should have so much code that does basically nothing except what is expected from the framework. dealing with these cleanups should be transparent and done by the framework/VM. it was my hope that the framework will just mature to handle these issues. I just keep the links and references in my code as simple and straightforward as possible hoping this should be enough - As I would in Java.

                 

                That is way overkill for memory management. As Alex mentioned, the key to memory management is to encapsulate references into modular chunks that can be created and destroyed when needed. You only need to destroy the "root" object, and all the references encapsulated within it will follow. There are a few caveats to be careful of, like binding to a Singleton object for example, but for the most part, you should not need very much "clean up code" in your application. Implementing clean up on a class by class basis is fighting a losing battle. Better to design your code so you can snip off large chunks at a single point.

                • 5. Re: Memory related crashes in Flex application
                  ron_bresler

                  Thanks Alex, drksrt_1

                   

                  There are many advice here. Your tutorial on memory troubleshooting is also very helpful.. we have started working with the profiler despite bad experience and I'm hoping to have better results now. I am now convinced its essential for our success. and glad to hear you do not recommend excessive cleanup code.

                  I do have some questions still:

                   

                  1. Its very good if we can anticipate the crash based on polling the System.totalMemory. at the moment it seems that somewhere between 1.3 and 1.6 GB (measured from the outside) we crash. is there any better way to determine the deadly threshold more accurately other than to just run a crash test a few times and record the memory usage to determine the threshold? according to documentation it probably depends on other parameters such as physical memory...?

                   

                  2. the profiler in some cases blocks the reference chain, as in the case of binding - sometimes we see an Object with a reference named "site" or "host" holding up the class. is there any way to determine a) if this is an actual leak - I expect binding to keep a class in memory only if it is a "site" whose "host" is still in memory - since it is referenced for notification by the host. and this seems to be the case when I run naive testing. So I'm not sure I should be suspicious of a reference by "host" reference. But the profiler looks like the binding itself may cause the "host" object to remain in memory until the binding is undone. b) Who is the counterpart "host" or "site" for each such object - I cannot see who is keeping up the reference, and I do not have stack trace information on the GC root object holding up the reference either - to see where the binding took place.

                   

                  3. Unloading modules and the reloading them does not seem to flush out memory leaks. even when they are in a child application domains - I read in documentation that any existing reference to classes that were loaded in a module prevent the module from being unloaded. Our modules are not subapplications. but they are loaded into child app domain, if I got it right. Currently I have implemented unloading and on demand loading of each module and actually increased the leakage significantly by it. I am trying to understand how to improve our modularity so that the leaks may be contained and cleared by compartmentization into app domains or modules (as I've said before we are running many modules but have not bothered unloading them so far because the leaks aborts unloading). But I'm not sure if that is even possible. is there a strong way to free memory that was allocated in a certain context?

                   

                  Appreciate all your help in this...

                  Ron

                  • 6. Re: Memory related crashes in Flex application
                    drkstr_1 Community Member

                    1. Its very good if we can anticipate the crash based on polling the System.totalMemory. at the moment it seems that somewhere between 1.3 and 1.6 GB (measured from the outside) we crash. is there any better way to determine the deadly threshold more accurately other than to just run a crash test a few times and record the memory usage to determine the threshold? according to documentation it probably depends on other parameters such as physical memory...?

                     

                    What are you doing that requires this much memory? The memory footprint of the actual application code should be fairly small. Are you processing image/video data or something along those lines, or are those figures just the compounded result of a memory leak? If you are pushing a large amount of data into the front end, you may need to be thinking about memory caching techniques instead.

                     

                    Regardless, I think anticipating a crash is not the way to go about it. You can never really know when something is about to crash, and it is better to design around this assumption. IMHO, a better way is to set a target goal for peak memory you will allow for your application. This amount should be reasonable given the nature of the application, and your target market (are they running on mobile devices or high end workstations?). You would then profile the application and implement memory management techniques until your application runs without ever exceeding your peak memory target. For example, we will not release any mobile app that exceeds a peak memory of 12mb during testing.

                     

                    2. the profiler in some cases blocks the reference chain, as in the case of binding - sometimes we see an Object with a reference named "site" or "host" holding up the class. is there any way to determine a) if this is an actual leak - I expect binding to keep a class in memory only if it is a "site" whose "host" is still in memory - since it is referenced for notification by the host. and this seems to be the case when I run naive testing. So I'm not sure I should be suspicious of a reference by "host" reference. But the profiler looks like the binding itself may cause the "host" object to remain in memory until the binding is undone. b) Who is the counterpart "host" or "site" for each such object - I cannot see who is keeping up the reference, and I do not have stack trace information on the GC root object holding up the reference either - to see where the binding took place.

                     

                    This really depends on your code, and what you would expect it to do at the time. If you have a binding somewhere in your application that is referencing an object that should be cleaned up, then I would expect to run into trouble.

                     

                    One thing that may help is to profile individual chunks of your application at first, rather than the application as a whole. I usually create small tests where I can easily check for leakage. For example, if I have a component or a module, I will create a small test to pop it on and off the stage (or use it in some meaningful way). I then run the profiler, taking a memory snap shot before and after the test), and I know I have a leak in that specific component if anything stuck around after.

                     

                    3. Unloading modules and the reloading them does not seem to flush out memory leaks. even when they are in a child application domains - I read in documentation that any existing reference to classes that were loaded in a module prevent the module from being unloaded. Our modules are not subapplications. but they are loaded into child app domain, if I got it right. Currently I have implemented unloading and on demand loading of each module and actually increased the leakage significantly by it. I am trying to understand how to improve our modularity so that the leaks may be contained and cleared by compartmentization into app domains or modules (as I've said before we are running many modules but have not bothered unloading them so far because the leaks aborts unloading). But I'm not sure if that is even possible. is there a strong way to free memory that was allocated in a certain context?

                     

                    Again, my suggestion is to test in chunks. Solve each of the parts first, then start looking at how those parts are connected.

                     

                    It is hard to give any specific advise without an example that demonstrates the leakage.

                    • 7. Re: Memory related crashes in Flex application
                      ron_bresler

                      What are you doing that requires this much memory? The memory footprint of the actual application code should be fairly small. Are you processing image/video data or something along those lines, or are those figures just the compounded result of a memory leak?

                       

                      I can't explain this high memory use.

                       

                      Our application is an authoring tool for media rich presentations (in the education field). but we usually do not load the media into the main application - media is loaded only briefly when browsing in "preview" mode - graphics are loaded as thumbnails and then discarded. swf graphics are loaded as png thumbnails to avoid unknown code from running.. in some cases we load visual components into the application for "wyswig" composition - but mostly the application runs on forns that only display textual references to the media. we do a lot of XML processing as we constantly paltten our data tree to and from XML but this is usually very straighforward. I have never managed to explain where all this memory usage comes from.

                       

                      The largest documents we are loading are 3-5 MB. We are loading some static images/swf as part of skinning and sometimes in the application as well, but overall there is absolutely no apparant reason for these levels of memory consumption. This has been the case from day 1.

                       

                      I often think about changing the entire approach to download only the parts that are requried for presentation at any given time because of this problem - and dump the whole "smart client" approach. we currently perform processing on entire documents in the client side because we wanted extension modules to be pure flex. but even if we rebuild this as a regular web application - its too big a change... so right now I'm left with memory consumption I cannot explain.

                       

                      Even before anything starts to happen in the application - if all the module code is loaded (we now load modules in lazy mode) we start out at close to 0.5 GB in the task manager - without any data being loaded or displayed... during work, Leaks are generally contained since we recycle evey major UI unit. this recycling causes some "spending" of memory on components that are in the background, but it at least limits the number of "new" operations to a minimum. still the memory shoots up. we are also loading classes and modules from another large modular application (ours) - but these do not cause such issues in their native application.

                       

                       

                      We have other applications that do not get as bloated - and we use the same compilation and build processes for them. So far, profiling could not explain these levels either. We are not loading large media or large data - nothing that can explain this mem usage. Profiler registers most of that memory at BitmapData and Function classes its too low level to work with - I never managed to link these instances to actual high level objects... But I am am trying that again now with some new advice...

                       

                       

                       

                      I keep searching for the compiler option "dont use ridiculous amounts of memory"...

                       

                      BTW - anticipating the crash is just a "band-aid" becuse we have users that crash daily.

                      • 8. Re: Memory related crashes in Flex application
                        Flex harUI Adobe Employee

                        Run the profiler.  Take a memory snapshot.  View it.  Turn off all filters and don’t exclude globals.  Then take a screen snapshot of the profiler screens and post it.

                        • 9. Re: Memory related crashes in Flex application
                          ron_bresler

                          I ran the application and loaded some data into it. I have not activated any of the UI that deals with data manipulation - so this does not reflect the entire potential, but its a starting point. you can see in the snapshots that the profiler shows 115MB in use. this measures about 350 MB in the task bar when running IE.

                           

                          the files can be found here

                          http://dl.dropbox.com/u/78224889/memory%20snapshot%20after%20document%20loading.rar

                          (I sorted the classes by the package name)

                           

                          the top ranking classes are:

                          Function - 52%

                          Class - 19%

                          XML - 9%

                          Object - 4%

                          BitmapData - 3%

                          String - 2%

                          MovieClip - 2%

                          Array - 1%

                           

                          In addition, we have done some profiling  - actually fixed some minor leaks, but the work is tremendously slow and painful, and I have a few questions:

                           

                          1. The profiler shows many bogus reference chains which are not relevent - meaning - they cannot cause an object to be kept in memory. I cannot say if these are weak references or references that the framework knows how to break during other class cleanups. in any case, these paths make a lot of noise and make it very hard to distinguish between paths that can actually cause objects to leak and those who do not. these include:

                            a. mx.accessibility objects

                            b. host objects (in binding) - the host object cannot be held in memory as a result of binding, but it appears to be referenced.

                            c. site objects (in binding) that listen to notification from the same object or objects that have the same object higher up in their own reference chain (children of sorts)

                            d. event listeners that listen to notifications from the same object or objects that have the same object higher up in their own reference chain (children of sorts)

                           

                          2. The profiler breaks the reference chain in some cases for no apparent reason - showing only a GC root where in fact there is no GC root - perhaps its native code that references an object or some lower level framework code, but its definitely not a GC root. as in the example of event dispatchers or binding host objects. this makes debugging with the profiler very difficult because you can never inspect reference chain all the way up - you cannot see who are the dispatcher or the host objects.

                           

                          3. I've found when using the sortCompareFunction attribute in MXML (DataGrid) to activate (notice i stress activate in runtime - the declaration itself does not cause a leak) a sort-compare method of the same object (in the document scope class or super class) this causes a memory leak - only by declaring these sort-compare methods "static" we were able to GC the entire UI object from memory. this looks like a framework bug. But if there is a meaningful explanation to this behavior that can help me know what to look out for in other cases I'd love if you can explain here.

                           

                          Overall, my impression is that the profiler does not really help in showing us what causes a memory leak (in most cases - if it was a simple programming mistake that did not involve events or binding maybe it would) its mainly good for the create/destroy scenario and only in showing us the object count - its generally useless trying to find event listeners or bindings that cause a leak because the profiler gives no visibility to the full chain, and it shows chains that are bogus.

                           

                          here is an example of the chains (we closed all the mx.accessibility related chains to show only internal binding and event listener chains) of a class that leaked. for us. the fix was to change the sortCompareFunction reference to a static implementation. I suspect this will  cause the class itself to remain in memory if we even try to unload that module.. but the main point is that nothing in the profiler could hint at the direction of the leak. we had to manually remove chunks of code to zero in on the issue. is this the only approach?

                          here is the link:

                          http://dl.dropbox.com/u/78224889/reference%20chains%20to%20leaking%20dialog.rar

                          • 10. Re: Memory related crashes in Flex application
                            Flex harUI Adobe Employee

                            Ron,

                             

                            Sorting by name does not help.  Pretty much the important stuff in who the top memory consumers are.  I’m not going to rummage through 20+ images to build out that list.

                             

                            To answer your questions:

                             

                            1.  The profiler shows all reference chains.  It does not know how to calculate which ones are important.  And yes, that is a lot of noise.  For me, I know to ignore certain chains like ones that involve binding or EmbeddedFontRegistry or ones that are circular.

                            2.  The profiler marks the end of a chain with GCRoot.  Static variables are often seens as GCRoots because of it.  But again, ones related to binding or EmbeddedFontRegistry are probably not causing leaks.

                            3.  That is probably not a bug.  If you use an anonymous function it builds a scope chain at instantiation and can cause memory leaks.  If you use an instance function the function has a reference to its ‘this’ pointer.

                             

                            As explained in my blog post, I don’t use loitering objects too often.  I generally compare memory snapshots by hand and look for what instance counts are growing and usually start with the largest objects unless I know the code well and have suspicions.

                             

                            -Alex

                            • 11. Re: Memory related crashes in Flex application
                              ron_bresler

                              Hi Alex,

                               

                              I'm sorry. Here is a link to the list sorted by memory consumption (memory usage changed a little since the last measurement since we fixed a couple of leaks):

                              http://dl.dropbox.com/u/78224889/CGT_document_load_memory.rar

                               

                              I Agree regarding Static references - they should appear as GC root. But I was disappointed to see that the profiler cannot represent a link between an event dispatcher and a listener. The chain is cut so that I cannot see the dispatcher object that is holding the listener in place. the same thing with binding and the other examples. I understand this is a limitation.

                               

                              the answer #3 is very interesting. my MXML reads:

                               

                              <mx:DataGridColumn

                              dataField="productionID" sortCompareFunction="sortByProductionId"... />

                               

                              where sortByProductionId is a named method of the enclosing class (my MXML object). If I declare that function "static" there is not leak, but if i leave it a non-static method there is a leak (MXML remains the same in both cases - there are no annonymous functions - only the explicit reference to "this"). I would expect the column instance to keep an implicit reference to "this" but this reference is a circular reference since the column object depends only on my "this" object. I think there is some other mechanism I do not understand correctly that works here. is this what you mean by scope chain? It would be very interesting to understand this.

                               

                               

                              Thanks again for all your patience...

                              Ron

                              • 12. Re: Memory related crashes in Flex application
                                Flex harUI Adobe Employee

                                Which version of FlashBuilder are you using?  Recent ones should show in the chain from dispatcher to listener.  Maybe it got broken again.

                                 

                                If you are registering an instance function as the sort compare function, that will add an additional reference to the instance.  Even if it is circular, the question is whether there is a reference in the direction from a GCRoot to the instance.

                                 

                                When I look at the first screen of memory consumers I notice the following:

                                 

                                1.  There are 5114 MovieClips.  I can’t think of any way to fit that many on screen at once in any useful fashion.  You should find out why there are so many.  That will be a huge performance and memory drag.

                                2.  There are 128 BitmapData.  That is also a lot.  Make sure your source bitmap resolutions match up well with the screens.

                                3.  There are 134 TextFields.  I don’t often see screens crammed with that many text widgets, so again, reducing that should help.  Recycle instances or defer instantiation.

                                 

                                -Alex