10 Replies Latest reply on Jul 27, 2011 2:13 PM by Echysttas

    Allowing XML to be interpreted causes a 30 MB Memory Leak

    Echysttas

      Hello everybody ::- ). Please, if you are adept with E4X and XML in AS3, take a look at this CRAZINESS.

       

      I have a 420 KB XML. Fairly large, yes. Its structure is fairly simple:

       

      <Nodes>
        <Node ID="1">
          <Element Name="A" />
          <Element Name="B" />
        </Node>
      </Nodes>

       

      I have between 2 to 10 Elements in a Node and about 50 Nodes in total.

       

      I am storing all the individual "Element" collections into a Dictionary, where the Key is the value of the Node ID attribute. Don't ask why I did it like this, okay? I had a good reason for it.

       

      All right, so we got a Dictionary with about 50 keys, each of them containing values which are XMLLists of 2 to 10 Elements.

       

      So, the big XML is split into more XMLLists and stored into the Dictionary like this:

       

      for each (var obj: XML in objects)
        _SomeDictionary[int(obj.@ID)] = obj.Element;

      Now here comes the weird part:

       

      1. Run Application.
      2. Initialize some objects using 10 of the 50 Dictionary Values.
      3. Memory Consumption: 36 MB.

       

      ---

       

      1. Run Application.
      2. Initialize objects using ALL 50 of the Dictionary's Values.
      4. Memory Consumption: 47 MB.

       

      But... if I do it like this:

       

      for each (var obj: XML in objects)
        _SomeDictionary[int(obj.@ID)] = String(obj.Element);

       

      I get this:

       

      1. Run Application.
      2. Initialize objects using ALL 50 of the Dictionary's Values, and prior to initialization, cast the Dictionary's Value back to XMLList.
      4. Memory Consumption: 16 MB.

       

      So I shaved 30 MB of memory just by casting the XMLList to a String. Why does this happen? Eventually, inside the Initialization function which processes the Dictionary's Value, an XML List will ALWAYS be created.

       

      Second question: when NOT casting to String, WHY does the memory increase when I process the Dictionary's Values??? They're all XML Lists already! Why does the memory only grow from 36 MB to 47 MB when I actually pull out the XML List from the Dictionary and go through it with a function which initializes some objects based on it?

       

      P.S.: The function does not cause memory leaks: I already proved that I only get 16 MB when I am using the SAME function but instead of giving it an XML List pulled directly from the Dictionary, I give it an XML List cast out of a String which is pulled from the Dictionary.

        • 1. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
          UbuntuPenguin Level 4

          As for why the Dictionary eats more space when you store an XMLList vs a String, I would have to guess that an XMLList object has a larger memory footprint compared to the String.  It's like asking why can I stuff more 5th graders into a phone booth as opposed to NFL linemen.

           

          As for why there is a memory jump when you reprocess the XML I would have to see the code.

           

          Protip: Use weak keys so your values can later be gc'ed when you are done with them.

          1 person found this helpful
          • 2. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
            Echysttas Level 1

            Hello and thank you for answering! ::- ).

             

            I knew about the weak keys thing, but have not applied it. I didn't think it would be useful, and it might be, but I am still left with the question of why the memory increase ONLY WHEN PROCESSING one of the Dictionary's XMLs (going through its elements to initialize an object - and no, it's not because the object takes the space because the object is made up of primitive fields, and, more importantly, for demonstration's sake, I instantiated 100 more of the same object with NO MORE memory increase - so it just happens the first time).

            • 3. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
              Flex harUI Adobe Employee

              XML is parsed on-demand.  When you assign a string and convert it to XML,

              only the top-level node is actually fully created.  I don't know the details

              if sub-nodes have a partial representation or not.  Then as you start

              accessing sub-nodes, more XML objects are created.  When you convert back

              from XML to string and store string, the XML object can be garbage

              collected, reducing total memory.

              1 person found this helpful
              • 4. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
                Echysttas Level 1

                Hello ::- ). Thanks for clarifying!

                 

                I got 2 short confirmation sub-questions ::- ), just to wrap this up.

                 

                1. To make sure I really understood what happens: when I store XML Lists in the Dictionary, they are processed on first access (on demand practically) and will remain in memory for future access (that's why I can re-use them 100 times without further memory increase). But when I store Strings in the Dictionary and pull them out as XML in a function but not store them anywhere, they get GCed, obviously, and I get no memory penalty. Is this how it goes?

                 

                2. So what you're saying is that what I did is actually good practice, right?

                • 5. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
                  Flex harUI Adobe Employee

                  I'm not sure I completely understood you, but I think the answer to #1 is

                  yes.

                   

                  For #2, I don't see any upside to storing XML nodes in a Dictionary.  In

                  fact, I believe that XML keys in Dictionary is not officially supported.  If

                  you are looking to cache nodes, I would just cache the XMLList or an array

                  of nodes or in a Object keyed with strings if there is a convenient string

                  identifier.

                   

                  If you are truly looking for speed, you would convert XML to objects if

                  possible.

                  • 6. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
                    Echysttas Level 1

                    Thank you very much for your reply & time Flex harUI ::- ).

                     

                    BTW related to #1, if you care to say what you didn't understand of what I said, I will rephrase ::- ).

                    • 7. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
                      Flex harUI Adobe Employee

                      You were describing storing as a string and pulling out as an XML function

                      and not storing it.  It would probably be better to see what code you

                      actually were talking about.

                       

                      You could experience growth in the memory numbers depending on your code and

                      how the garbage collection runs.  You could end up allocating extra memory

                      pages until the collector kicks in.

                      • 8. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
                        Echysttas Level 1

                        Well, ok, here's the code ::- ). I will paste you the 3 functions involved to:

                         

                        1. Register XML data sets which are used to initialize Logicon Objects.

                        2. Demand for the initialization of an Object based on the earlier provided XML data.

                        3. Initialize the Object.

                         

                        Note: _ObjectInitializationXMLMappery = new Dictionary(); //not weak.

                        Note: This code is used by a library which I built for in-game arithmetical calculations. The library is called Logicon and works with ILogiconObjects.

                         

                            //Provides the Class with XML Data to use when initializing Logicon Objects.
                            //This Function may be called multiple times to add various XML data sets; however, conflicting Keys will be overwritten.
                            //PARAM data: XML data.
                            public function AddObjectInitializationXML (data: XML): void
                            {
                              var objects: XMLList = data.LO; //Retrieve & Process all individual Objects in the given Data.
                              //The Objects will be stored as String inside the Object Initialization XML Mapper because otherwise, tons of Memory will die.
                              for each (var obj: XML in objects)
                              { //This object has a Unique Object Name.
                                if (obj.@UON != undefined)
                                  //Create Dictionary Entry based on Unique Object Name. The Entry will contain all the Modifiers in this XML Node (M nodes).
                                  _ObjectInitializationXMLMapper["N:" + String(obj.@UON)] = String(obj.M);
                                else //No UON.
                                  _ObjectInitializationXMLMapper[int(obj.@ID)] = String(obj.M); //Create Dictionary Entry based on Object Type ID.
                              }
                            }

                         

                            //Initializes an object (from the earlier XML data).
                            //PARAM object: Logicon Object to initialize.
                            public function InitializeObject (object: ILogiconObject): void
                            {
                              var modifiers: String;
                              if (object.UniqueObjectName != null)
                                modifiers = _ObjectInitializationXMLMapper["N:" + object.UniqueObjectName];
                              else
                                modifiers = _ObjectInitializationXMLMapper[object.ObjectTypeID];
                              //No Modifiers found for this Object? Bye.
                              if (modifiers == null) return;
                              AddModifiersFromXMLToObject(modifiers, object); //Now add the Modifiers to the Object.
                            }


                            //Processes an XML List of Modifiers and adds them to a Logicon Object.
                            //PARAM modifiers: XML data to process.
                            //PARAM object: Logicon Object to which to add the Modifiers.
                            public function AddModifiersFromXMLToObject (modifiers: String, object: ILogiconObject): void
                            {
                              var noModifiers: Boolean = true; //TRUE: Throws an Error when no Modifiers are found in the XML.
                              //Process all Modifiers.
                              for each (var xm: XML in XMLList(modifiers))
                              {

                                //And here I'm doing some stuff based on xm and its attributes.

                                //Stuff like object.SomeProperty = xm.@SomeAttribute, various checks, blah blah.

                              }

                            }

                         

                        I really appreciate you taking an interest ::- D.

                        • 9. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
                          Flex harUI Adobe Employee

                          I think I see.  You are using the dictionary to cache XML or string

                          representation of the XML of objects yet to be converted.

                           

                          Seems like a lot of work to create the nodes in the XMLList, flatten to

                          strings and resurrect later.  But maybe that's the right thing for how you

                          consume the objects.  It appears there are lists of objects and usually,

                          those lists are managed and consumed in collections.  Some folks simply

                          store the XMLList, front it with an Array and on getItemAt, convert the XML

                          node to the object and cache in the array.

                          • 10. Re: Allowing XML to be interpreted causes a 30 MB Memory Leak
                            Echysttas Level 1

                            Oh, what I forgot to mention (big mistake!) is that this is the NON-LEAKY version. So it's the fixed version which yields 16 MB RAM consumption.

                             

                            I did that because the library supports adding multiple XMLs at run-time, at any given time, and, also, objects are initialized not very often. I would say 1 object / 3 seconds for the first minute of the game, then even less often, because I got an object pool and almost all intensively-created objects are pooled. (have experienced great problems in the past with Flash's GC because every time it gets invoked automatically, there is a slight stutter in the running application so I mitigated this by using object pools)

                             

                            Your Array solution seems to behave in a way I find familiar. Probably that's exactly what the Dictionary does! I was storing XML directly in it, NOT STRING (like in the above example). And when I first accessed a Dictionary Key, its associated (XML) Value would be for-each-ed and, apparently, cached by Flash as a fully parsed XML. I think this is the way the Dictionary worked and resulted in the 47 MB of memory consumption: just like the cached Array you proposed.

                             

                            The new version (pasted above) works with 16 MB because I keep Strings, process them and then immediately dump the processed XML.

                             

                            I admit that this definitely wastes CPU clocks but it keeps the application light. Given my usage case: rare use, I find this acceptable.