We are definitely able to saturate a 1G NIC before maxing out cpu.  However, our tests are run on

4 CPU’s [Intel(R) Xenon(R) CPU E5430 @2.66 GHz]

9 GB RAM

For RTMP/RTMPE, we see 1G NIC max out at about 30%.  This goes down with more RTMPT/RTMPTE connections.

We also tested on RHEL 4, our officially supported platform, though I suspect the issue is simply that you are hitting the limits of what can be handled by a single cpu machine.

One suggestion I have is to try and spread the load out across multiple fmscore processes so you can make better use of all that RAM.  That might allow you to cache more content in memory, reducing disk i/o, which might improve your throughput.

8 cores?  Hmm, than I am the one that is confused.  You didn't mention 8 core, so I assumed single core.  Our tests were done on a 4 core machine.  But as I said, we're able to max out a 1G NIC at 30% cpu.

Couple things to note, more RAM doesn't necessarily help if your process can't use it.  With 32-bit processes, you're obviously limited by how much RAM can be used, which is why I suggested distributing load across more fmscore processes, so each process can use more of that 62GB of RAM that you have.  Also, I'm not sure if you need PAE kernel to use that much RAM?

The other thing to note is that our testing has been done on RHEL 4, which is our officially supported platform.  Any chance you can try your test on an RHEL 4 machine?

Our configs would be default configs.  But the main config is the size of the file cache, in fms.ini, SERVER.FLVCACHE_MAXSIZE = 500 (MB).

Another thing to check is how much of this traffic is RTMPT vs. RTMP?  RTMPT would also affect performance.

No Troy.  I'm not saying everyone uses RHEL4.  In fact, i know that they don't.  But I also haven't heard of this before either.  But depending on kernel version, glibc versions, etc. there could definitely be varying degrees of performance.  For example, on Windows, a while ago, Microsoft released a service pack that had some tweaks to tcp offloading that caused major performance degradation.  But at a high level, there is nothing about FMS that is Linux distro specific.  But as we have done our testing on RHEL4, I'm curious to know if you run into the same problem running on RHEL4.