Live stream problem with FMS 4.5 + FMLE

[Xrtili7]

Hi,

I've got a live stream (and DVR) setup using FMS 4.5 and FMLE.

The hardware for the FMS server we use is 2x Xeon E5645 2.4GHz, 32GB RAM, OCZ RevoDrive R3 X2 PCI-e SSD. Installed OS is CentOS 5.8.

We have 6 workstation PCs, each equiped with 4 BlackMagic Intensity PRO capture cards and FMLE 3.2. Using these PCs we stream Live video to FMS server.

Video stream is grabbed from HDMI cable, to BlackMagic Cards, which are encoding by FMLE using H.264 format, 700kbps bitrate per stream.

Video stream is published to server through RTMP stream, which is later saved for DVR in RAW format. We only keep 3 days of recording and delete the old ones.

Server and Encoder PCs are in the same network, connected by gigabit managed switch.

The problem I'm having is that after 10-15 hours FMLE starts to drop frames, because video buffer is increased. What I observed is that this happens immediately when Server CPU load increases above 60%.

Based on the above observation I decreased the number of channels streamed by the server to 10, which reduced CPU load. But the problem still persists.

Whenever I restart FMS and delete all DVR data, the CPU load (when streaming 10 live channels) is only 1%, but after 2 days CPU load increases to 50-60%.

Whenever I restart FMS and don't delete DVR data, the CPU load is 5-10%, and after 2 days it still increases to 60-70%.

Another thing I observed is that there is only single fmscore process running, but it has lots of threads which are switched on and off in split seconds. These threads are launched on different CPU cores, but at any given point in time the distribution of the load isn't equal among CPU cores. This leads certain CPU cores being loaded by more than 60% and frame drops start to occur.

For the moment there are just 10 users watching this service, so I don't think this load accounts for the problem.

Has anybody had similar problem, or know how can I optimize or finetune the system to run without problem? I would appreciate any suggestions.

Another thing I noted through last couple of days:

When I was restarting FMS the CPU load was reduced to 30%, but after 1 week past when I restart it CPU load only goes down to 75%. Everything is the same, nothing has been changed and there is no disk IO issues involved.

P.S. I've modified application.xml using these values:

<Scope>vhost</Scope>

<Distribute numprocs="5">app</Distribute>

<LifeTime>

<RollOver></RollOver>

<MaxCores></MaxCores>

</LifeTime>

[Guest]

I've experienced similar issues on Red Hat 5.0 as described above. Initially FMS restart was helping to decude CPU load, but when the problem wasn't resolved I moved to Wowza.

[Xrtili7]

One thing I discovered through further research is that when I changed video format from H264 to VP6 the problem resolved for 1 week. during this time with 4 days of DVR for 21 Live streams together CPU load only reaches steady 30%. After 1 week the CPU load gradually increases and with the same amount of DVR content CPU load is more than 90%!!!

I've found out guys who had similar problems a year ago, but they couldn't find any solution, even from Adobe.

Can anybody suggest any ideas?

[Xrtili7]

I finally nailed down the source of the problem. As it appears it is the dvr Index file that causes all this trouble.

The thing is that this index file is constantly expanding in size, no matter that we are deleting old video segments. Once this index files become more than 3mb in size, FMS starts to cause problems.

Has anyone got idea how to work with these index files manually? I mean to manually remove the unnecessary data from them?

[KarishmaB]

Hi

How many channels are you publishing?

If there are too many channels, it is recommended to have one  FMSCore process start for each of them. To do so, you will have to change the scope to app.

---snippet---

<Application>

<Process>

    <Scope>app</Scope>

   <Distribute numprocs="3">inst</Distribute>

.

.

</Process>

</Application>

Also, to delete older content, you will have to enable disk management. Refer http://help.adobe.com/en_US/flashmediaserver/devguide/WSeb6b7485f9649bf23d103e5512e08f3a338-8000.htm... for more info.

[Xrtili7]

Hi Karishma,

Thanks for the reply. You are the only one out there to have any sort of answer to my problem.

The thing is that we use RTMP streaming for the moment, not HTTP so I couldn't find similar disk management functionality there.

I've manually configured so that each channel connects to different FMSAdapter, so there is separate FMSCore launched for each one of them.

For the moment we stream 20 live streams and save 3 days of DVR.

The problem that I finally nailed down is in the proprietary INDEX files that FMS is creating for each DVR stream. As the time goes old content is deleted, but this INDEX file isn't cleaned up and grows in size, which gradually slows down whole FMS.

I couldn't find any method of manually altering data that is written in these INDEX files.

[KarishmaB]

Hi

Which "index" files are you actually pointing at? Are these the *.f4x files created?

These files are created as a part of HDS content saved on disk. Configuring disk management should help you control the size of HDS content on disk.

I am assuming you are streaming to livepkgr application. If you don't want HTTP streaming (i.e. HDS) then you should stream to live application instead.

-Karishma

[Xrtili7]

We are not using HDS, we are using RTMP live streaming options, thus there is no Disk Management functionality available here.

Index files are very same *.f4x files created for DVR content saved in RAW files, but the problem is these index files are ever-expanding and once they pass certain size they crash FMS.

[KarishmaB]

These *.f4x files are index files for HDS content saved on disk.

You have 2 options:

1) Publish to live application - you lose the HDS functionality and there won't be any HDS content saved on disk

2) If you wnat both RTMP and HDS, configure diskmanagement duration for HDS content so that you just have the latest content on disk

[Xrtili7]

I've finally decided to move everything from RTMP streaming to HTTP using HDS.

I'll be migrating my system for the next few days and will let you know the results. I'll consider all the suggestions you provided Karishma.

Could you also tell me best practices for architecture?

I mean, because there is no Edge/Origin setup in HDS case, I have to swap Edge servers with some proxy/cache solutions.

Should I go for nginx in front of Origin Server or should I use plain Varnish instead?

And also can cache content aware Load balancing be done effectively with Varnish compared to nginx?

The idea is for the Load Balancer (which will be in front of multiple proxy/cache servers) to know which content is available in cache of which proxy servers and redirect traffic accordingly, not just random traffic based selection.

[KarishmaB]

Hi

I may not be able to provide you inputs on Varnish v/s nginx. It depends on your architecture and kind of load balancing you want to achieve.

But if you need help in configuring Varnish, you can refer http://www.adobe.com/devnet/adobe-media-server/articles/varnish-sample-for-failover.html

Let us know if you face any issues in HDS setup/playback.

-Karishma

[Xrtili7]

Hi Karishma,

I've redeployed the whole project on HDS version. Everything is working fine and disk management function is also automatically handeling all of the old content itself.

I've got 1 question though, I have set up DVR window to 72 hours, which makes *.bootstrap files on average 800KB in size. The problem is that in HDS streaming fragment duration is 4 seconds, so I had to set HttpStreamingBootstrapMaxAge to 2 seconds, which therefore means that client player request this bootstrap file once every 2 seconds.

As an end result to play 1mbps bitrate stream client needs at least 4mbps bandwidth, plus 3/4 of the traffic is taken by the bootstrap file.

Is there any way to get around this issue, because it seems really unoptimized solution.

[KarishmaB]

Hi

If you want DVR for 72 hours then there is no better way. But are you sure bootstrap is requested after every 2 seconds?

Setting HttpStreamingBootstrapMaxAge to 2, implies its TTL is 2. It doens't imply the interval after which player requests the bootstrap. Ideally it should be based on the duration of fragments created on disk, and should be downloaded every 4 seconds. But, note it's the duration of actual fragment created and not the duration you configure that determines this frequency. Both may differ if the keyframes are not aligned according to the fragment duration.

-Karishma