Skip navigation
Currently Being Moderated

Heavy Memory Leak in BlazeDS

Aug 22, 2010 9:34 PM

I have used BlazeDS in an application to push the data in realtime. However, we have been experiencing a heavy memory leak in the application. The application has a scheduler that pushes the data every one minute into the default queue. Without any browsers opened, the appication remain stable. And closing the browsers also cleans up the FlexClient object and other associated objects for that browsers. But the leak occurs when we open and keep browsers for a long period, say 2 to 3 days. The memory usage of application gradually increases and throws a "java.lang.OutOfMemoryError: Java heap space" Exception. On analyzing the HeapDump, I found that there are thousands of AsyncMessage Objects in the memory, not getting garbage-collected. This the channel definiton congured in services-config.xml file

 

 

        <channel-definition id="my-polling-amf" class="mx.messaging.channels.AMFChannel">
            <endpoint url="http://{server.name}:{server.port}/{context.root}/messagebroker/amfpolling" class="flex.messaging.endpoints.AMFEndpoint"/>
            <properties>
                <polling-enabled>true</polling-enabled>
                <polling-interval-seconds>4</polling-interval-seconds>
                <invalidate-session-on-disconnect>true</invalidate-session-on-disconn ect>
            </properties>
        </channel-definition>

Does anyone know the cause of this issue and how it can be overcome. Please help.

 
Replies
  • Currently Being Moderated
    Calculating status...
    Sep 6, 2010 11:46 AM   in reply to Mohd Fayaz

    I found the same bug in my application , I profiled it and found out that there are lots of instance of  PollCommandMessageResponder class in PollingChannel package , I obviously created one Subscriber and it continually create this instance ,

    Adobe is so slow to fix these kind of bugs so I think emulate subscriber service myself.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 10, 2010 1:37 PM   in reply to onuratamer

    You should certainly feel free to file a JIRA bug with a reproducible case.  We will take a look at it. Note that a reproducible case involves more than the config - we need to have the code that makes the problem happen.

     

    Or, since BlazeDS is open source, you could dig in to the code and provide a patch for the issue.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 15, 2010 3:42 AM   in reply to tomj

    Higly reproductible with BlazeDS4 :

     

    Simple application :

        2 consumers.

        Each consummer draws the result of the message (X/Y positionning from the message).

        Each destination is StreamingAMF / JMS Adapter with ActiveMQ.

     

    Launch clients (IE or FireFox) to consume the message.

     

    Send no more than 50 messages/s on one of the destination only.

     

    Press a few time F5.

     

    you won't wait too long to get OOM.

     

    Heapdump analysys shows :

    264 970 instances of class flex.messaging.messages.AsyncMessage

     

     

    The same test using BlazeDS 3.0 shows 0 instances of class flex.messaging.messages.AsyncMessage under the same load.

     

    pu-frontal2-monitor OOM.png

    This first image shows a number of threads and a memory consumption always increasing - BlazeDS 4 behaviour.

    (remember we press F5 to reload the application time to time)

     

    pu-frontal1-noOOM with Blazeds3.png

    This secondimage shows a number of threads aligned with the real number of clients - BlazeDS 3 behaviour.

    No problems with th memory consumption.

    (remember we press F5 to reload the application time to time)

     

     

    HTH

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 15, 2010 7:28 AM   in reply to quentin.buonanno

    Please file a bug with this information (and the repro code) in our bug tracking system - JIRA: http://bugs.adobe.com/blazeds/

     

    Thanks

    Tom

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 15, 2010 8:16 AM   in reply to onuratamer

    Hi.

     

    I think the other people on this thread are talking about memory consumption on the server while it looks like you are talking about memory consumption on the client. It's quite normal when using a polling channel to see many instances off the PollCommandMessageResponder class getting created. One of these should get created for every poll request. For example, if your polling interval was 2 seconds you would see one of these getting created every two seconds or so.

     

    Have you tried forcing garbage collection using the profiler? You should see these instances go away.

     

    Hope that helps.

     

    -Alex

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 15, 2010 8:30 AM   in reply to tomj

    Hi,

     

    BlazeDS doesn't destroy the clients session right after the connection closed.  The undeliverable messages are kept until the client timeout. The Flex Client can be restored if the same ID is used.  In order to prevent high usage of memory, it should make the flex client timeout short. After the Flex Client timeout, the messages(AsyncMessage) are discarded. For the memory management, we can also have custom OutBoundQueueProcessor to limit the number of messages in the queue. (for the blazeds posting)

    In short, the problem can be solved by setting the flex client timeout to 1 minute.

    This is on server only. We are investigating the browser leak issue. We did some MTBF test, one of the JMS test used polling amf. We didn't see any memory issue in 7 days run. Can you let me know which build did you find the memory leak? Mohd.

     

    William

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 22, 2010 1:12 AM   in reply to tomj

    BLZ-572 : created with some code/configuration to

    reproduce.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 28, 2010 1:49 PM   in reply to Mohd Fayaz

    I have tracked down (and fixed locally) a serious memory leak in PollingChannel.as. It turns out that PollCommandMessageResponder objects are getting created for every polling message but never getting released because they are getting held in memory by an event listener.

     

    I filed a bug: https://bugs.adobe.com/jira/browse/SDK-27883

     

    The fix is simple -- just cleanup the event listener. The nice thing about this bug is that you don't have to take my word for it -- please just look at PollingChannel.as code and notice at the bottom:

     

      public function PollCommandMessageResponder(agent:MessageAgent, msg:IMessage, channel:PollingChannel, log:ILogger)

        {

            super(agent, msg, channel);

            _log = log;

           

            // Track channel connected state.

            // If the channel disconnects while this poll is outstanding, suppress result/fault handling.

           channel.addEventListener(PropertyChangeEvent.PROPERTY_CHANGE, channelPropertyChangeHandler);

        }

     

    Now look (in vain) to see where that event listener is cleaned up -- it isn't.

     

    My fix is simple -- just release the event listener after it is no longer needed. I do it two places:

     

    Place 1:

     

    override protected function resultHandler(msg:IMessage):void

        {     

            var pollingChannel:PollingChannel = channel as PollingChannel;       

           

     

    //

    //VANTOS: This line was added to remove a serious memory leak

    channel.removeEventListener(PropertyChangeEvent.PROPERTY_CHANGE, channelPropertyChangeHandler);

     

    Place 2:

     

        override protected function statusHandler(msg:IMessage):void

        {       

    //

    //VANTOS: This line was added to remove a serious memory leak

    //

     

    channel.removeEventListener(PropertyChangeEvent.PROPERTY_CHANGE, channelPropertyChangeHandler);

     

     

    ************

    Please note that I've although I've verified that this does stop the memory leak (using the Flash Builder profiler), it is possible that there is a better way to fix this. But this fix seems to work for us.

     

     

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 29, 2010 6:36 AM   in reply to Mohd Fayaz

    I think there are two different issues here: possible memory leak on the server (BLZ-572) and the client (SDK-27883). 

     

    I haven't looked into this in too much detail but my guess is that BLZ-572 is probably not a bug. I think what's happening is that, the browser is closed (or the app is reloaded) with an HTTP based channel, messages accumulate until session times out which is expected. We've had similar bugs before where we explained people how to avoid it (i.e keep session timeout low, use disconnectAll trick etc.). I suggest you take a look the comments in this bug:

     

    http://bugs.adobe.com/jira/browse/BLZ-502

     

    Regardless, we'll take a look at BLZ-572 to see if it's different from BLZ-502.

     

    In SDK-27883, there's a valid a point that we should remove the event listener at some point but I don't think not removing it will necessarily cause a memory leak as indicated, need to verify it with profiler to make sure. My guess is it doesn't cause any memory leak but I agree that we should remove the event listener to be on the safe side.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 29, 2010 7:21 AM   in reply to Mete Atamel

    Yes, I agree these do appear to be two separate issues. Thank you for clearly separating them.

     

    There really is a memory leak on the client. If you have any doubts please run the Flash Profiler and watch the counts for PollCommandMessageResponder (you will need to set up a polling connection of course).

     

    Did you get a chance to look at the code that I pointed out in PollingChannel.as? You can see that for each poll a PollCommandMessageResponder is getting created. And each one of those registers an event listener in its constructor. That means Flash has to keep that PollCommandMessageResponder in memory. I don't see any way around that, but perhaps I'm missing something here? I'm asking that with honest curiosity.

     

    By the way, if I'm right it actually is worse than just a memory leak -- if we did happen to get the error event that all those PollCommandMessageResponder are listening for, all of them would suddenly have to be called which could be quite a large amount of work that might explain some of the random crashes we've seen.

     
    |
    Mark as:
  • Currently Being Moderated
    Sep 29, 2010 8:00 AM   in reply to OdysseusLevy

    You're right, and your fix looks correct as well. Thanks for uncovering and reporting this bug, we'll put in a fix soon.

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 25, 2011 8:40 AM   in reply to quentin.buonanno

    I found the same bug on my application, I use Flex 4.1, Blazeds 4, spring flex 1.5.2, spring 3.0.6., Weblogic 10, Websphere MQ 7.

    Does anyone know the cause of this issue and how it can be overcome. Please help.

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 2, 2012 9:41 PM   in reply to pretorian310

    Hi All,

     

    I am facing the same issue while using the BlazeDS and Spring flex Jars .I have tried all the possible solutions :-

     

    1) disconnect all  in SWF
    2) Invalidate session on disconnect  in serviceconfig.xml
    3) session time out in flex configuration file (serviceconfig.xml ) and web.xml --Reduce to 1 -2 minutes

     

    But still getting the growing number of instance of (flex.messaging.messages.AsyncMessage) and the objects which is a part of the body of the Asyn message when the browser is closed .

     

    It seems that there is no acknowledgment from the flex session to stop messages from the BlazeDS

    Flex session is alive which causes the server to  keep pushing the messages.

     

    I need help to overcome this issue

     

    Please also mention the version of spring -flex and Blaze DS to be used .Is that version is available in Maven Central repository .

     

     

    Thanks,

    Rahul S

     

     

     

     
    |
    Mark as:
  • Currently Being Moderated
    Jan 3, 2012 2:41 AM   in reply to TOC456

    Hi Rahul,

     

    Have you got a chance to follow the issues mentioned in the post above? I assume those did not help

     

    Please open a JIRA bug with the following information:

     

    1.       Environment details.

     

    2.       A standalone reproducible end-to-end test scenario that illustrates your problem.

     

    BlazeDS 4.6 requires Spring Flex 1.5.2. No, at this point, we don’t push the BlazeDS binaries in the Maven repository.

     

    Rohit

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points