I am not sure what you are trying to protect - whether someone ripping your stream or want to protect someone streaming to you.
Now if you want to protect anyone streaming from your server, please use SWFVerification. Now this will block your FMS connection too. So you need to allow it in as exception.
To protect anyone publishing, you can either use Access Adaptor or Server-side AS to write logic to allow only connections from valid publishers.
For example - navigate to <yourinstallationAMS_Directory>/samples/applications/live - you will see main.asc. Open it in Notepad and you will logic on how to allow clients - you will see that encoders like FMLE or FMS as publisher are not checked for authentication - you can that check by writing your own code.