Recent problem: clients are losing connection to AMS 5.0.1

[VIN_Nick]

For the past week, since Oct 14, we've been having issues with clients losing connection to our AMS application. We have online classes that include a chat room and sometimes an audio stream from the instructor. This application and its server has been running stable for months... until last week.

One of our staff was connected to the app, getting everything set up for a class, and she started complaining of getting disconnected. I looked in the application logs and noticed that the application.onDisconnect was being triggered. However, the client didn't register that it had been disconnected  for several seconds afterward. She let me know when she got the "Diconnected" message on her screen, which comes from the NetConnection.onStatus handler.

The only other sign of a problem is that in the server's core logs we started getting the following error when we checked bandwidth:

> Invalid command message data: unexpected bw response from client 000000001D670170

The client number changes frequently, so I'm guessing that this is first happening to one person and then another. We don't think that this bandwidth check is causing the disconnects because we rolled back to a previous version that didn't have it. The disconnects still happened after using the old version. What makes it frustrating is that it hasn't happened to anyone on the dev team, regardless of which network we've connected from. Also, the code hasn't changed on the client or the server, and to my knowledge, we're still using the same configs for months.

The AMS is v5.0.1, the client is written is AS2 and targets Flash Player 11.4.

The affected Flash Players reported are 15.0.0.152 and 15.0.0.189, in browsers IE, Firefox, and Chrome, but not all installations are affected.

Is there anything else I can look at to determine what the problem is? Does anybody have any insight as to what might be suddenly affecting our AMS app?

Many thanks,

Nick

[hparmar]

you seem to have said that you are seeing this even with older versions of AMS i.e AMS version < 5.

BTW, That unexpected bw response if because of bad network but AMS does "not" terminate connection on bad bw data.

If clients are getting terminated, it has to be some reason other than bad bw response.

We need to isolate when and after what changes did this problem started happening.

[VIN_Nick]

Yes, we came to the same conclusion about the unexpected bw. We realize that it is not causing the issue, but I included it because it started happening at the same time and thought that they might be caused by the same issue.

We have been trying to come up with what changed to cause this, and aren't coming up with anything.

• We have been on 5.0.1 for at least 5 months, no changes there.
• We haven't updated the main.asc since 9/16.
• The VM that runs our server is still the same, and AMS config files haven't been touched.
• The client swf had a change 2 weeks ago, but we've reverted the change and are now using the old swf again.
• The person who initially reported the problem was using the same computer that she's used many times to connect; she was going through her usual process to set up for the session.
• You mentioned bad network, and that is another thing that I have been looking into, but our IT department says that we haven't seen any huge increase in bandwidth usage, and that everything looks to be within normal operating ranges

If there are any other points that you can think of that I could check for changes, I'm open to suggestions. This problem has been ongoing for 6 days, and I have been searching for answers since it started.

[hparmar]

Do you have a support contract with Adobe...if yes, you can open an escalation can we can investigate this, but then we will have to ask you to enable additional settings in your configs generate more logging data and this may be more than few iterations back and forth.

[hparmar]

Are there simultaneous disconnects of multiple clients or is it some client getting disconnected randomly in between.

[VIN_Nick]

It's not simultaneous, but it is very frequent. When I was monitoring it on Tuesday (the day this started) there were several a minute, but not at the same time.

[hparmar]

1. Can you isolate it with one AMS server and one or two clients running.

2. Is it the publisher which is disconnecting or is it the subscriber which is disconnected.

[VIN_Nick]

1. The dev team isn't able to reproduce this behaviour, and we've been trying on different devices on different networks. It doesn't happen on computers at in the office or at home for me. This is something we're still trying to do, but all we can do right now is observe it, and usually we're trying to find ways to resolve it, because it's time critical (i.e. a scheduled session is coming up).

2. I'll need some clarification here on what's a publisher and subscriber. We're not sure if it's the server code or if it's being caused by the NetConnection. All I know for sure is that the server's event is being fired before the client has detected a change in connection status.

I've been looking into our support contract, but I recently took over this project, and was told that we may not have one. Just as an aside, how would I look up if we have one available?

[hparmar]

Publisher is one who is sending video/audio stream(from encoder/camera or such device) to AMS and subscriber is one who is receiving such a stream and playing it in his browser/swf/player. An individual may be both subscriber  and publisher but in that case he need to have two separate connections/streams for these tasks. A connection/stream at a time can either be a publisher or a subscriber but not both. So we have to find out if it is the publisher who is disconnecting or is it the subscriber. if a publisher disconnects it is possible that after some time all subscribers to that stream may also get disconnected.

[hparmar]

You can find out if you have any support contract by writing to entrsupp@adobe.com.

[VIN_Nick]

Ok, I understand now. No, this isn't a  problem with our audio streams, the entire connection is failing for the affected individuals. We have some sessions that are chat only but still lose participants.