We are using Adobe Media Server 5.0.1 in an edge-origin setup. Our servers have 16 GB RAM and Windows 2008 R2. We use the streaming servers for RTMP, VOD, multicast, and live streaming. The servers mostly run fine, but once in a while the memory usage of the AMSCore process on the origin server starts growing and takes up all of the physical memory and page file. Event viewer shows many warnings related to AMS Core with the message "System memory load (99) is high.".
Has anyone seen this behavior and have a solution or workaround? It seems to have started happening after we enabled live streaming, but it happens randomly and is hard to recreate. Please let me know if I need to provide more details.