Hey all,
We have an FMS 4.5.2 installation with 1x Origin and 2x Edge setup only for Live streaming. We are using FMLE latest version to stream to the Origin server and Edge servers are connecting to the Origin server. We are running four applications stream1, stream2, stream3 and stream4 all for live streaming (copied live) application. All servers have default configuration except edge/origin setup.
All servers have 4Gbit connections in bonding and network traffic is nicely distributed across all bonds and uplink is 10Gbit to the ISP.
Server is a 2 CPU / Quad Core HP DL 380 with 64Gbit of memory running Ubuntu 10.04 LTS
Now to the problem.
We are streaming live stream from FMLE with 700kbit/s to for example stream1 application and when we hit around 1600 concurrent users with 50% on edge1 and 50% on edge2 (800 concurrent users on each edge) in the player buffer drops and all people experience buffering issues at approx 30secs - 2minutes intervals. (network is not congested because if at that time do an SCP from the server outside network it copies 1Gbyte file within seconds.
Funny thing is that If I start a second FMLE and stream live stream to for example stream2 application at same time and open up second player on client the video runs great without any buffer issues from the same server at the same time.
edge1 server:
application: stream1 with 800 concurrent users, player has buffer issues
application: stream2 with 4 concurrent users, player has no issues
edge2 server
same as above
So my current conclusion is that it has to be something per application since other application does not have this issues when running simultaneously? We tried changing fmscore number settings and some buffer settings but nothing helped.
at the time when we get buffering issues this are the only problematic things that get in the log and its in edge.00.log on both EDGE servers but not at same times:
Edge1:
2012-04-23 12:50:21 29270 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
2012-04-23 12:55:30 29270 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
2012-04-23 12:56:42 29270 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
2012-04-23 12:56:42 29270 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
2012-04-23 13:14:40 29270 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
2012-04-23 13:20:30 29270 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
Edge2:
2012-04-23 12:56:32 9625 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
2012-04-23 13:02:23 9625 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
2012-04-23 13:08:03 9625 (e)2661034 Connect failed ( , 8134 ) : Connect failed: Connection refused (111) -
there is no packet loss between edge and origin servers and latency is at 0.2ms and nothing in the logs of the origin server
We even tried to deploy Wowza Medis Servers with edge / origin setup and were able to handle around 4000 (2000 on one and 2000 on second edge) concurrent users without any issues.
Anyone has any ideas or at least what are our next options to do and what settings to change on the FMS? or how to debug and what to check when buffering issue appears? any more debugging we can enable on the FMSs? too se if we hit some kind of limit somewhere?
thanks
3
Replies
AdChoices