We are running FMS 4.5.7 with standard livepkgr application for live RTMP and HLS streaming (HDS module disabled). For some reason after about 30 minutes httpd process starts using more and more system memory and in about a minute or so it crashes the server because it drains memory completely. It doesn't matter how many users are connected to the stream. What I noticed is that if there is no HLS users (or HLS is disabled) everything is fine, however it is enough to connect at least one HLS user to the server after 30 minutes and the memory usage starts rising. If the user connects into 0-30 minute window it works.
I have tried to play with stream recording options in Event.xml for that stream but it didn't help. I have tried to record into one file and into segments.
Our server has 32GB of RAM.
We have modified Apache configuration file to Sandbox HLS streams at instance level, but it should not be a problem, shouldn't it?
Options -Indexes FollowSymLinks
P.S. We also run vod application in parallel for vod RTMP and HLS streaming. Both HLS and RTMP for VOD works just fine.
After two weeks I finally solved this.
To anyone struggling with similar issue make sure you have the licence serial loaded AND the server fully restarted. At least on Linux restarting the server through Flash Media Administrator Console leave some zombie processes running which blocks loading the licence correctly. Restart whole OS instead.