12 Replies Latest reply: Nov 10, 2014 10:17 PM by loguetr53 RSS

    FMS 4.5.x: The FLV segment cache is full

    nebbb Community Member

      We just migrated our service to FMS 4.5.  Since our new server is in production, we started having the following error message in the core.log files:

       

      The FLV segment cache is full

       

      There is very little information about this problem on the web.  The only information found is to increase the value of parameter SERVER.FLVCACHE_MAXSIZE in the fms.ini file.  We tried this:

       

      SERVER.FLVCACHE_MAXSIZE=2000

       

      Increased the cache size only delayed the problem from a few hours to 1 day.  It seems that FMS 4.5 is filling up the cache without cleaning anything.  Note that we are doing a very high number of video recordings.  The only way to solve the issue is to restart the FMS service.

       

      One week ago, we were running FMS 3.5.7 on Windows 2003 and we never had this issue.  FMS was running with 500Mb of cache and was working fine for years.  This bugs has appeared since we moved to the FMS 4.5 on a Windows 2008 server.

       

      Coincidentally, Adobe released the 4.5.3 patch that includes fix for memory problems one week ago.  But even this version has the issue.

       

      I saw a similar thread here: http://forums.adobe.com/thread/913107

       

      Anyone can point to a solution?  This is a major bug.

        • 1. Re: FMS 4.5.x: The FLV segment cache is full
          ShivenK Adobe Employee

          Hi,

           

          Please go through this piece of information and configure the parameters accordingly:

          http://help.adobe.com/en_US/flashmediaserver/configadmin/WS5b3ccc516d4fbf351e63e3d119f2925 e64-7ff0.html#WS5b3ccc516d4fbf351e63e3d119f2925e64-7fee

           

          Also, let me know:

               1.     How many streams are being published onto the server?

               2.     What is the container and codec for the published streams?

           

          Thanks,

          Shiven

          • 2. Re: FMS 4.5.x: The FLV segment cache is full
            nebbb Community Member

            Hello Shiven,

             

            thank you for the reply.  A few days back, we increased SERVER.FLVCACHE_MAXSIZE to 2500, and also found that we can clean up the memory faster by setting FLVCachePurge to 1.  We also figured that the old FMS 3.5.7 probably would have had the same issue.  In fact, we saw that the same week we changed to our new server, we also had an increase in usage.

             

            However, this still doesn't explain that the memory fills up so quickly.  It seems that recording videos is a very heavy memory thing for FMS.  Here are the answers to your questions:

             

            1) we had about 35,000 streams on a day that the cache filled up

            2) we use standard H.263 in FLV container with NellyMoser mono.

             

            We actually were able to reproduce the problem on any FMS version by setting SERVER.FLVCACHE_MAXSIZE to 1 and doing a few recordings.  After about 5 recordings (of 30 seconds each), the cache problem appears.  By setting the FLVCachePurge to 1, it's harder to reproduce as the cache is cleaned faster.  However, I would expect a cache system to clean up the oldest stream recorded no matter if the caching period is over or not.  There is certainly a major issue to look at here for your team, as we can't expect to just increase the cache size each time the problem re-appear.

             

            Currently, in addition to the two changes that we did in the configuration, we also do a reboot of the FMS service every night.  This is a temporary measure and we need a real solution to this issue.  We can't be rebooting FMS as a final solution.

             

            Ask your team to test and reproduce the cache problem following these steps:

             

            a) set SERVER.FLVCACHE_MAXSIZE=1 in fms.ini

            b) record a few stream to the FMS, each of about 30 seconds

            c) notice that the cache generate the error and that it's impossible to record another stream until the FMS service is rebooted

             

            EXPECTED behavior: clear the oldest stream, no matter the time set in FLVCachePurge

             

            Please follow-up on this thread with more explaination.

             

            Thank you.

            • 3. Re: FMS 4.5.x: The FLV segment cache is full
              Jorge Maiquez [DS]

              Hi Nebbb,

               

              Did you manage to get this problem under control? We're seeing exactly the same problem after migrating to 4.5, and tweaking those settings has not made it go away.

               

              Best regards,

              Jorge

              • 4. Re: FMS 4.5.x: The FLV segment cache is full
                nebbb Community Member

                No we did not.  We are rebooting the service daily as the solution.

                • 5. Re: FMS 4.5.x: The FLV segment cache is full
                  Jorge Maiquez [DS] Community Member

                  What a PITA. Good job on supporting your clients Adobe!

                  • 6. Re: FMS 4.5.x: The FLV segment cache is full
                    jim_houseman Community Member

                    I'm having this problem with v5.0.3

                    • 7. Re: FMS 4.5.x: The FLV segment cache is full
                      nebbb Community Member

                      Hello Shiven,

                       

                      we reported very specific repro steps more than 2 years ago about this issue.  Others are also getting it in pretty much all version of FMS/AMS.  Can you tell us what is causing this?

                      • 8. Re: FMS 4.5.x: The FLV segment cache is full
                        jim_houseman Community Member

                        It's now coming on the end of October. Any answer? @ShivenK?

                        • 9. Re: FMS 4.5.x: The FLV segment cache is full
                          stoem

                          My friend Jorge asked me to post this reply as a potential solution.

                           

                          This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:

                           

                          1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).

                          2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):

                           

                          CONFIG FILE Server.xml

                          line 251:    <AutoCloseIdleClients enable="true">

                          line 264:    <FLVCachePurge>10</FLVCachePurge>

                          line 265:    <FLVCache retryOnCacheFull="false"> (add attribute to tag)

                          line 317:    <ApplicationGC>1</ApplicationGC>

                          line 319:    <FLVCacheSize>25</FLVCacheSize>

                          line 594:    <CoreGC>30</CoreGC>

                           

                          CONFIG FILE fms.ini     (renamed to ams.ini in version 5 of FMS)

                          line 74:        SERVER.FLVCACHE_MAXSIZE=3000

                          NOTE: This value (in MB) should be no more than 25% of total RAM.

                           

                          3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:

                           

                          sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

                           

                          Bottom line: allow a core to go idle so that it unloads and releases memory.

                           

                           

                          Hope that helps.

                          • 10. Re: FMS 4.5.x: The FLV segment cache is full
                            stoem Community Member

                            My friend Jorge asked me to post this reply as a potential solution.

                             

                            This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:

                             

                            1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).

                            2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):

                             

                            CONFIG FILE Server.xml

                            line 251:    <AutoCloseIdleClients enable="true">

                            line 264:    <FLVCachePurge>10</FLVCachePurge>

                            line 265:    <FLVCache retryOnCacheFull="false"> (add attribute to tag)

                            line 317:    <ApplicationGC>1</ApplicationGC>

                            line 319:    <FLVCacheSize>25</FLVCacheSize>

                            line 594:    <CoreGC>30</CoreGC>

                             

                            CONFIG FILE fms.ini     (renamed to ams.ini in version 5 of FMS)

                            line 74:        SERVER.FLVCACHE_MAXSIZE=3000

                            NOTE: This value (in MB) should be no more than 25% of total RAM.

                             

                            3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:

                             

                            sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

                             

                            Bottom line: allow a core to go idle so that it unloads and releases memory.

                             

                             

                            Hope that helps.

                            • 11. Re: FMS 4.5.x: The FLV segment cache is full
                              hparmar Adobe Employee

                              Also you can configure Rollover for cores in Application.xml...See

                                  <LifeTime>

                                    <RollOver></RollOver>

                                    <MaxCores></MaxCores>

                                  </LifeTime>

                              in Application.xml

                               

                              Also http://www.adobe.com/devnet/adobe-media-server/articles/hardening_guide.html

                              • 12. Re: FMS 4.5.x: The FLV segment cache is full
                                loguetr53

                                The sync; echo 3 | sudo tee /proc/sys/vm/drop_caches is great for linux host.  Any recommendations for Windows 2008 based machines?   The only solution I foresee would be to have some script to reboot the service daily.  Which is not an option for us with multiple clients globally hitting the system.  

                                 

                                Would load balancing the solution in order to allow idle time for one server?