Skip navigation
Xrtili7
Currently Being Moderated

Live stream problem with FMS 4.5 + FMLE

Feb 12, 2013 6:00 AM

Tags: #problem #adobe #video #live #server #fms #h264 #streaming #rtmp #stream #live_streaming #fms_4.5

Hi,

 

I've got a live stream (and DVR) setup using FMS 4.5 and FMLE.

The hardware for the FMS server we use is 2x Xeon E5645 2.4GHz, 32GB RAM, OCZ RevoDrive R3 X2 PCI-e SSD. Installed OS is CentOS 5.8.

We have 6 workstation PCs, each equiped with 4 BlackMagic Intensity PRO capture cards and FMLE 3.2. Using these PCs we stream Live video to FMS server.

Video stream is grabbed from HDMI cable, to BlackMagic Cards, which are encoding by FMLE using H.264 format, 700kbps bitrate per stream.

Video stream is published to server through RTMP stream, which is later saved for DVR in RAW format. We only keep 3 days of recording and delete the old ones.

 

Server and Encoder PCs are in the same network, connected by gigabit managed switch.

 

The problem I'm having is that after 10-15 hours FMLE starts to drop frames, because video buffer is increased. What I observed is that this happens immediately when Server CPU load increases above 60%.

Based on the above observation I decreased the number of channels streamed by the server to 10, which reduced CPU load. But the problem still persists.

 

Whenever I restart FMS and delete all DVR data, the CPU load (when streaming 10 live channels) is only 1%, but after 2 days CPU load increases to 50-60%.

Whenever I restart FMS and don't delete DVR data, the CPU load is 5-10%, and after 2 days it still increases to 60-70%.

Another thing I observed is that there is only single fmscore process running, but it has lots of threads which are switched on and off in split seconds. These threads are launched on different CPU cores, but at any given point in time the distribution of the load isn't equal among CPU cores. This leads certain CPU cores being loaded by more than 60% and frame drops start to occur.

 

For the moment there are just 10 users watching this service, so I don't think this load accounts for the problem.

 

Has anybody had similar problem, or know how can I optimize or finetune the system to run without problem? I would appreciate any suggestions.

 

Another thing I noted through last couple of days:

When I was restarting FMS the CPU load was reduced to 30%, but after 1 week past when I restart it CPU load only goes down to 75%. Everything is the same, nothing has been changed and there is no disk IO issues involved.

 

 

P.S. I've modified application.xml using these values:

 

<Scope>vhost</Scope>

<Distribute numprocs="5">app</Distribute>

<LifeTime>

<RollOver></RollOver>

<MaxCores></MaxCores>

</LifeTime>

 
Replies
  • Currently Being Moderated
    Feb 12, 2013 6:04 AM   in reply to Xrtili7

    I've experienced similar issues on Red Hat 5.0 as described above. Initially FMS restart was helping to decude CPU load, but when the problem wasn't resolved I moved to Wowza.

     
    |
    Mark as:
  • Currently Being Moderated
    Feb 25, 2013 5:53 AM   in reply to Xrtili7

    Hi

    How many channels are you publishing?

    If there are too many channels, it is recommended to have one  FMSCore process start for each of them. To do so, you will have to change the scope to app.

    ---snippet---

     

    <Application>

    <Process>

        <Scope>app</Scope>

       <Distribute numprocs="3">inst</Distribute>

    .

    .

    </Process>

    </Application>

     

    Also, to delete older content, you will have to enable disk management. Refer http://help.adobe.com/en_US/flashmediaserver/devguide/WSeb6b7485f9649b f23d103e5512e08f3a338-8000.html#WSec225f632fa00875-23954b6f1300b641158 -8000 for more info.

     


     


     
    |
    Mark as:
  • Currently Being Moderated
    Mar 4, 2013 2:17 AM   in reply to Xrtili7

    Hi

    Which "index" files are you actually pointing at? Are these the *.f4x files created?

    These files are created as a part of HDS content saved on disk. Configuring disk management should help you control the size of HDS content on disk.

    I am assuming you are streaming to livepkgr application. If you don't want HTTP streaming (i.e. HDS) then you should stream to live application instead.

     

    -Karishma

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 5, 2013 3:42 AM   in reply to Xrtili7

    These *.f4x files are index files for HDS content saved on disk.

    You have 2 options:

    1) Publish to live application - you lose the HDS functionality and there won't be any HDS content saved on disk

    2) If you wnat both RTMP and HDS, configure diskmanagement duration for HDS content so that you just have the latest content on disk

     
    |
    Mark as:
  • Currently Being Moderated
    Mar 5, 2013 11:29 PM   in reply to Xrtili7

    Hi

    I may not be able to provide you inputs on Varnish v/s nginx. It depends on your architecture and kind of load balancing you want to achieve.

    But if you need help in configuring Varnish, you can refer http://www.adobe.com/devnet/adobe-media-server/articles/varnish-sample -for-failover.html

    Let us know if you face any issues in HDS setup/playback.

    -Karishma

     
    |
    Mark as:
  • Currently Being Moderated
    May 27, 2013 12:21 AM   in reply to Xrtili7

    Hi

    If you want DVR for 72 hours then there is no better way. But are you sure bootstrap is requested after every 2 seconds?

    Setting HttpStreamingBootstrapMaxAge to 2, implies its TTL is 2. It doens't imply the interval after which player requests the bootstrap. Ideally it should be based on the duration of fragments created on disk, and should be downloaded every 4 seconds. But, note it's the duration of actual fragment created and not the duration you configure that determines this frequency. Both may differ if the keyframes are not aligned according to the fragment duration.

     

    -Karishma

     
    |
    Mark as:

More Like This

  • Retrieving data ...

Bookmarked By (0)

Answers + Points = Status

  • 10 points awarded for Correct Answers
  • 5 points awarded for Helpful Answers
  • 10,000+ points
  • 1,001-10,000 points
  • 501-1,000 points
  • 5-500 points