Software: FMS 3.5 interactive license
OS: Fedora Core 4
CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
DISK: 8 2T drives with software raid 10
APP: Standard VOD app
FMS Connections: Max 5.10k Avg 2.01k
Bandwidth Out: Max 248M Avg 85M
CPU 100% all the time. Server becomes unresponsive. I have 4 other servers with the same specs with less connections and have the same result. What can I do to bring the load of the server down?? With performance like this I will need 10 servers to serve such a small amount of traffic. I also have 2 servers running Debian 5.0.5 with twice the cpu as the one listed above and it didnt help very much at all.
We are definitely able to saturate a 1G NIC before maxing out cpu. However, our tests are run on
4 CPU’s [Intel(R) Xenon(R) CPU E5430 @2.66 GHz]
9 GB RAM
For RTMP/RTMPE, we see 1G NIC max out at about 30%. This goes down with more RTMPT/RTMPTE connections.
We also tested on RHEL 4, our officially supported platform, though I suspect the issue is simply that you are hitting the limits of what can be handled by a single cpu machine.
One suggestion I have is to try and spread the load out across multiple fmscore processes so you can make better use of all that RAM. That might allow you to cache more content in memory, reducing disk i/o, which might improve your throughput.
I dont understand your meaning about me reaching the limit of a single cpu machine. The cpu you listed only has a slight performance diff than mine. My proc is listed as 8 cores in linux. Just as I am sure your does.
I also have FMS running on 2 Intel(R) Xeon(R) CPU E5520 @ 2.27GHz with Debian 5.0.5 showing 16 cores. With the same amount of ram in this server has very little performance increase. Server is being killed between 80% and 90% cpu usage with no more than 300M out of a 4 port 1G bond on the nic.
I would love to see your FMS configs that you use to produce this kind of traffic with so little load. I am running 10 FMS servers right now trying to balance out load and looks like I need to add 10 more with this kind of crappy performance.
I see in many places with you all say that you can max out a 1G nic bla bla bla and so on but I have yet to see any proof of this. Do you have some sort of secret config that only you are allowed to have?? For what I paid for my 10 interactive license I would think that I would be able to produce 20 times the traffic that I am doing now.
8 cores? Hmm, than I am the one that is confused. You didn't mention 8 core, so I assumed single core. Our tests were done on a 4 core machine. But as I said, we're able to max out a 1G NIC at 30% cpu.
Couple things to note, more RAM doesn't necessarily help if your process can't use it. With 32-bit processes, you're obviously limited by how much RAM can be used, which is why I suggested distributing load across more fmscore processes, so each process can use more of that 62GB of RAM that you have. Also, I'm not sure if you need PAE kernel to use that much RAM?
The other thing to note is that our testing has been done on RHEL 4, which is our officially supported platform. Any chance you can try your test on an RHEL 4 machine?
Our configs would be default configs. But the main config is the size of the file cache, in fms.ini, SERVER.FLVCACHE_MAXSIZE = 500 (MB).
Another thing to check is how much of this traffic is RTMPT vs. RTMP? RTMPT would also affect performance.
I am not running into a ram problem. I can distribute load across more
fmscore processes when the ones that are running now are pegging my system
at 100% cpu. Its completely unusable. FMS is up no more than 30 min before
the server is completely hammered.
On a side note I am in no way new to serving moves and performance of a
server. I server muli Gb/s out of these same server setups using just
lighttpd. Even the few installs of Darwin I have perform better than this
I understand that your "officially" supported platform is RHLE 4 but its
very hard for me to believe that it would perform all that much diff on any
other distro of linux. Are you really trying to tell me that every person
that runs FMS uses RHLE 4? Its very strange that every search I have ever
done on performance tuning has your same response. Its like your reading out
of a book and no one is getting there problems resolved.
No Troy. I'm not saying everyone uses RHEL4. In fact, i know that they don't. But I also haven't heard of this before either. But depending on kernel version, glibc versions, etc. there could definitely be varying degrees of performance. For example, on Windows, a while ago, Microsoft released a service pack that had some tweaks to tcp offloading that caused major performance degradation. But at a high level, there is nothing about FMS that is Linux distro specific. But as we have done our testing on RHEL4, I'm curious to know if you run into the same problem running on RHEL4.
Europe, Middle East and Africa