aSiNe wrote: Hi, and much appreciate the response. 1. As it appears to be happening at random, I can't say exactly *when* the server stops serving requests. When the server does stop serving requests, you cannot connect to RTMP instances, and the FMS HTTP server doesn't serve anything either ( in fact it never times out, just get a blank screen), so both appear to be hung, and don't get restarted. |
The edge going down--even if the cores and apache were running--would cause exactly that. There's nothing wrong with apache, so it doesn't get restarted; there's nothing wrong with the cores, so they don't get restarted (although eventually they will quit because all of their apps are unused--which you may be seeing in the logs); the only problem is that they're not getting connections proxied or migrated to them because there's no (working) edge to give them any.
The most useful thing would be to verify that this is what's happening. If you can leave a client connected for a long time, ideally playing some stream on repeating loop, then the next time FMS stops accepting new connections, you can see if the existing client is still playing. If so, that means the edge is down but the core is still running; if not, that means something else is wrong.
Here are some more things to try next time it happens:
- nc localhost 80, type a few lines of random garbage, and see whether it fails to connect, hangs and ignores everything you type, disconnects you (immediately, after the first line, or otherwise), or sends back random garbage of its own.
- ps -C fmsedge, to see if the edge is still running.
- killall -6 fmsedge, which should kill the (apparently-hung) edge and get a core dump that someone on the FMS team can look at. (You may have to put "ulimit -c unlimited" in the server script, or otherwise change your settings.
Also, after killing the edge, see if it comes back up automatically. (There should also be stuff in the logs about that happening.)
To me, nothing looks like its crashing, but then I might just be reading it wrong. |
No, that looks accurate. The only thing you're seeing is cores quitting normally, either because of inactivity or because their edges have gone down or stopped responding. (Unfortunately, there's no way to distinguish between these cases from the core log.)
6. Not as far as I know, however is there a minimum swap space required? The sevrer has 8GB ram installed. |
There's no specific minimum; you just have to have "enough" that you don't run out. On one extreme, I have FMS running on a VM with 512MB total RAM+swap; it can serve one or two clients without any problems. On the other extreme, someone reported using 128GB of RAM+swap to run 100 core processes doing who knows what.
With default settings and a moderate load, I think 8GB and minimal swap should be more than enough. But if you want to make sure, run "cat /proc/meminfo" (or your favorite tool for the same purpose) and see how much MemFree + SwapFree is. If it never gets anywhere near 0, you don't have to worry.
As for restarting the httpd service, I was mistaken. My troubleshooting took me in that direction as this problem only arised *after* I had reinstalled FMS with Apache (previously I had opted to not install and everything ran fine). So I assumed that because Apache was crashing, this was affecting FMS somehow. However FMS is definitely not crashing - this is what is confusing me the most. Even after the streams are inaccessible, I can still connect to the server using the admin console! |
Well, the admin is handled by another process (fmsadmin), which runs almost completely independently of the rest of FMS (so it can be used to detect problems, start and stop apps, etc.). So, I wouldn't expect that to stop working.
Unfortunately, the admin is mainly there to administer the core processes (which do all the work) and the master (which is used to control the whole set of processes); the edge is supposed to be a lightweight and simple proxy that doesn't need much administration, so the admin server barely talks to it. Except, of course, when the edge isn't working, as seems to be happening for you.
I realize this is confusing. Let me try to break it down:
- RTMP on 1935 or 80: Connects to the edge, which then finds a core process to hand the connection to and then gets out of the way.
- RTMPT on 80: Connects to the edge, which then finds a core process and tunnels the connection for you.
- HTTP on 80: Connects to the edge, which then finds a webserver and proxies the connection for you.
- Admin on 1111: Connects to the admin, which does everything internally.
So, a hung edge would mean no new RTMP connections, but existing ones work fine; RTMPT and HTTP are hosed; admin is unaffected.
Anyway, thanks for gathering all of this information; it'll be very helpful in tracking down the bug and fixing it.
36
Replies
AdChoices