Skip navigation
Davido22a
Currently Being Moderated

Live stream on FMS issues when reaching 1600 concurrent users

Apr 23, 2012 4:48 AM

Tags: #problem #live #server #4.5 #buffer #issues #buffering

Hey all,

 

 

 

We have an FMS 4.5.2 installation with 1x Origin and 2x Edge setup only for Live streaming. We are using FMLE latest version to stream to the Origin server and Edge servers are connecting to the Origin server. We are running four applications stream1, stream2, stream3 and stream4 all for live streaming (copied live) application. All servers have default configuration except edge/origin setup.

 

 

 

All servers have 4Gbit connections in bonding and network traffic is nicely distributed across all bonds and uplink is 10Gbit to the ISP.

 

 

 

Server is a 2 CPU / Quad Core HP DL 380 with 64Gbit of memory running Ubuntu 10.04 LTS

 

 

 

Now to the problem.

 

 

 

We are streaming live stream from FMLE with 700kbit/s to for example stream1 application and when we hit around 1600 concurrent users with 50% on edge1 and 50% on edge2 (800 concurrent users on each edge) in the player buffer drops and all people experience buffering issues at approx 30secs - 2minutes intervals. (network is not congested because if at that time do an SCP from the server outside network it copies 1Gbyte file within seconds.

 

 

 

Funny thing is that If I start a second FMLE and stream live stream to for example stream2 application at same time and open up second player on client the video runs great without any buffer issues from the same server at the same time.

 

 

 

edge1 server:

 

application: stream1 with 800 concurrent users, player has buffer issues

application: stream2 with 4 concurrent users, player has no issues

 

 

 

edge2 server

same as above

 

So my current conclusion is that it has to be something per application since other application does not have this issues when running simultaneously? We tried changing fmscore number settings and some buffer settings but nothing helped.

 

 

 

at the time when we get buffering issues this are the only problematic things that get in the log and its in edge.00.log on both EDGE servers but not at same times:

 

Edge1:

2012-04-23    12:50:21    29270    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

2012-04-23    12:55:30    29270    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

2012-04-23    12:56:42    29270    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

2012-04-23    12:56:42    29270    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

2012-04-23    13:14:40    29270    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

2012-04-23    13:20:30    29270    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

 

Edge2:

2012-04-23    12:56:32    9625    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

2012-04-23    13:02:23    9625    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

2012-04-23    13:08:03    9625    (e)2661034    Connect failed ( , 8134 ) : Connect failed: Connection refused (111)    -

 

there is no packet loss between edge and origin servers and latency is at 0.2ms and nothing in the logs of the origin server

 

We even tried to deploy Wowza Medis Servers with edge / origin setup and were able to handle around 4000 (2000 on one and 2000 on second edge) concurrent users without any issues.

 

Anyone has any ideas or at least what are our next options to do and what settings to change on the FMS? or how to debug and what to check when buffering issue appears? any more debugging we can enable on the FMSs? too se if we hit some kind of limit somewhere?

 

 

thanks

 
Replies
  • Currently Being Moderated
    Apr 23, 2012 1:27 PM   in reply to Davido22a

    Looks very similar to my problem ( i posted some time ago ) but with a VOD application. In my case limit on edge was around 200 users . I solved it by not using origin / edge solution and using only origins.

     
    |
    Mark as:
  • Currently Being Moderated
    Apr 29, 2012 10:39 PM   in reply to Davido22a

    In my tests I got the same problem. I got buffering arround 1300 concurrent users. Changing the bitrate or or FPS didn't changed much.

     
    |
    Mark as:
  • Currently Being Moderated
    Nov 12, 2012 3:14 AM   in reply to Davido22a

    hey, i got the similar problem, logging like this

    2012-11-1218:50:1223434(e)2661034Connect failed ( , 5779212 ) : Connect failed: Connection refused (111)-
    2012-11-1218:50:5423434(e)2661034Connect failed ( , 5779212 ) : Connect failed: Connection refused (111)-
    2012-11-1218:51:3623434(e)2661034Connect failed ( , 1166880400 ) : Connect failed: Connection refused (111)-
    2012-11-1218:54:1423434(e)2661034Connect failed ( , 1175301776 ) : Connect failed: Connection refused (111)-
    2012-11-1218:54:5523434(e)2661034Connect failed ( , 1164775056 ) : Connect failed: Connection refused (111)-
    2012-11-1218:55:3723434(e)2661034Connect failed ( , 16 ) : Connect failed: Connection refused (111)-
    2012-11-1219:13:0823434(e)2661034Connect failed ( , 1158459024 ) : Connect failed: Connection refused (111)

     

    it seems that the port number is invalid, but we never use such ports.

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points