FMIS restarting and memory issue

Report · May 22, 2011

I have Flash Media Interactive Server 3.5.5 on Windows 2008 server. I developed video chat application (many to many), it's only 1 FMS application with about 50-100 room instances and 1 lobby instance to which all users are connected. There are up to 600 - 700 users at a time which produce about 2000-2500 connections to FMS.
Problem I have is with steady raise of memory usage, which if goes little above 2GB will make FMS unresponsive and it would restart itself after about 1-2hours. To avoid that Im restarting FMS manualy every 3-4 days. Interesting thing I noticed is that if I restart FMS from admin console, used RAM memory won't fall at levels as if it would right after whole Windows server is restarted.

In my Application.xml there are following configs:
<Scope></Scope>
<Distribute numprocs="3">Instances</Distribute>

These are things from logs during FMS unresponsive period or right before it:

master.log
2011-05-21 22:01:11 996 (i)2581223 Core (3592) is no longer active. -
2011-05-21 22:01:16 996 (i)2581223 Core (1868) is no longer active. -

admin.log
2011-05-21 22:01:06 2832 (i)2581242 Core (3592) disconnected : _defaultRoot_:_defaultVHost_:::_0. -
2011-05-21 22:01:09 2832 (i)2581242 Core (1868) disconnected : _defaultRoot_:_defaultVHost_:::_1. -

edge.log
2011-05-21 14:22:23 840 (e)2661034 Connect failed ( , -1 ) : Connect failed: (10061) -

2011-05-21 14:23:05 840 (e)2661034 Connect failed ( , -1 ) : Connect failed: (10061) -

2011-05-21 22:01:06 840 (i)2581250 Edge disconnected from core (3592). -
2011-05-21 22:01:09 840 (i)2581250 Edge disconnected from core (1868). -

access.log
session    disconnect    2011-05-21    21:29:06    6632    68.191.113.251    29476    39743    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    72.198.193.1    4759    6837    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    83.70.249.91    11996    14797    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    108.8.209.2    57227    76111    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    212.183.140.17    3073    3595    -    -    -    -    200    -

Please help me with this issue.

Report · May 24, 2011

Hi,

Thanks for trying out FMS.

When you restart the FMS, one reason why the memory is still high is because of the existing users. They might as well attempt reconnection as soon as the FMS is up again, retaining your memory usage. There must be a sudden fall and rise if that is the case. Please let us know if this is correct.

You didn't mention the core logs, are there any error statements in the core log when it dies ?

I would also like to know the other configurations for FMS like FLV cache size , aggregate messages (enabled/disabled), queue enabled/disabled etc. Can you please provide those details as well ?

What is the system configuration of the machine on which the FMS is installed ?

Thank you !

Report · Jun 02, 2011

Hi,
Sorry for delay in reply.
After FMS restart, memory falls but not in levels where it's after Windows restart, even when there are no existing users who are attempting reconnection.

In core logs I can only find things like:
2011-06-01 19:17:12 5112 (e)2631029 Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: _defaultVHost_, IP: 75.205.202.118, App: chatcity/_definst_, Protocol: rtmp, Client: 7011883898884410480, Handle: 2265252077) : D5 D9 BE F8 ; 1F 82 p 3 , $ B8 8D S J ^ ) 94 FD S 0B 81 80 $ FF A9 > 83 CD 87 A8 L H F4 9D -

2011-06-01 21:34:25 5112 (e)2611392 Exception while processing message: Unexpected virtual message from client 6140488000333434699: DD 82 . 94 EB FC i 00 ( C5 E6

2011-05-28 11:26:59 3272 (i)2581368 Client 6080275479418588227 has been idle for 36012 seconds. -

In configurations I have these settings (just to repeat that application is video chat, so there are no FLV files involved, only live streaming):
<FLVCacheSize>10</FLVCacheSize>

<EnableAggMsgs>false</EnableAggMsgs>

Server specification is:
Intel Xeon X5450 @ 3.00GHz (2 processors)
16GB RAM

If there is anything more needed please tell me.

Report · Jun 02, 2011

Hi,

Thanks for the information.

When you said the memorny is still not less, you are talking of the memory consumed by FMS processes ?

Once the FMS service (and the FMS admin service) is restarted, the old processes are shutdown and new ones come up. Can you individually record the memory usage of these processes ? Probably there is something else that's blocking the memory to leak. Make sure the old Fms processes are properly shutdown during the restart (kill if not clean).

Regarding the bad network data messages in your core logs, they come when there are 'unwanted' requests coming to FMS. For example, if there is a http request which is corrput that is sent on to the FMS edge process (which is expecting rtmp and when http is disabled), it might reject this request and log a message something very similar to what you are seeing. I have found these messages often when the machine is sort of virus affected also. Please check for these things as well.

I hope the disabling of queue and aggre messages is to support your video chat application, the settings look fine to me and hence not a problem.

Report · Jul 11, 2011

Thanks for your answers, I'm replying with delay because I wanted to check these things some more.

When I said 'memory is not less', I meant that total memory in Windows after FMS restart doesn't return to memory levels when Windows is restarted, but I guess that's because of some Windows processes and not connected to FMS.

As for FMS processes, these are the things are notice, few hours after FMS restart I get 1 FMSCore.exe that is around 200-220MB, and 3-4 FMSCore's that are around 40-70MB. Size of that largest FMSCore.exe doesn't go down even when there are much less users connected to FMS. And like I said before, after 3 or more days of running FMS becomes unresponsive, and restarts itself. Right after FMS restart largest FMSCore.exe weights around 70MB but 15mins after it's already 120MB as users connect to video chat application.

As for logs, my core.log files have several lines like:

2011-07-04    03:59:24    3308    (e)2611392    Exception while processing message: Unexpected virtual message from client 6446421578071032403: 93 { # 92 " : t 00 00 00 02     -
2011-07-04    03:59:24    3308    (e)2611029    Bad network data; terminating connection : chunkstream error:message length 12086012 is longerthan max rtmp packet length    -
2011-07-04    03:59:24    3308    (e)2631029    Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: _defaultVHost_, IP: 94.4.57.121, App: chatcity/stream, Protocol: rtmp, Client: 6446421578071032403, Handle: 2497053121) : 88 1E & D7 FB / 8D 7 99 CB X ...

2011-07-02 14:09:57 1228 (e)2611363 Server detected a backward timestamp from 7990 to 7985 in file: C:\Program Files (x86)\Adobe\Flash Media Server 3.5\applications\chatcity\streams\_definst_\captured_16627033 -

While edge.log has:

2011-07-04 04:58:56 1384 (e)2661034 Connect failed ( , 8134 ) : Connect failed: (10061) -
2011-07-02 21:09:05 6864 (e)2661034 Connect failed ( , -1 ) : Connect failed: (10061) -

And master.log has:

2011-07-04 00:30:27 7128 (i)2581223 Core (6280) is no longer active. -
2011-07-04 00:30:27 7128 (i)2581223 Core (4352) is no longer active. -

Report · Jul 29, 2011

Latest problem that I have more often is that Windows 2008 where FMS 3.5 is stops working, so manual restart of Windows 2008 is required. In FMS logs I don't see any new items that I haven't wrote in previous posts here. Only thing I noticed is that in _definst_ logs there is massive disconnection of all users in 1-2 seconds when this happens. And this problem happens when there is big number of users connected, which is around 600 which have around 2000-2500 connections to FMS.

Thing that I suspect could be issue is that my Application Process Scope is set to default vhost, so whole application with all of it's instances is set in 1 Windows process which is sometimes 220-250MB large. Do you think I should set Application Process Scope to inst to avoid this? Although I have up to 60 application instances running at same time.

Please help me with this.

Report · Jul 29, 2011

Hi,

Yes changing the process scope can enhance the performance , but make sure your work flow is intact while distributing the traffic among different processes. These processes will not be able to share resoruces (read, media and objects) and hence is different.

Regarding the total load on the server, i think it would still have the same (more or less) memory usage, that is now distributed on many processes, right ?

Adobe Community

FMIS restarting and memory issue