Yes, you are right that subscribers are only served by the Apache and FMS process has no role in that. So FMS's Auth/File module won't work for it.
Apache provides wide possiblities to write your own authentication scheme. You may try using the Apache's mod_auth module or write your own.
For HDS-Vod, yes you don't require running FMS. Only apache is required. But for HDS-live, you need FMS for publisher to send rtmp data to FMS and record it on the server.
thank you for your response.
After playing around with 4.5.0 and trying to bench it a bit, resulting in total crash of the edges, we found that 4.5.2 was released. Unfortunately the FMIS dev download page links to 4.5.0, one has to manually find another page with the updates. This should probably be fixed..
Anyhow, as of 4.5.1 it seems that the traffic is sent directly to Apache, confirming our guesses, and what you wrote.
Regardin authentication, we've looking in to a few solutions similar to what you suggested.