Is there a supported way for clients to authenticate the identity of an AMS/FMS? Or is it necessary to write a custom challenge response upon connection to do this?
It's hard to guess what you are trying to achieve but Client properties on the server-side will get you started - http://help.adobe.com/en_US/adobemediaserver/ssaslr/WS5b3ccc516d4fbf35 1e63e3d11a11afc95e-7ec3SSASLR.2.3.html.
Assume the client knows in advance which server it wants to connect to. It should have some way of verifying the identity of this server, so that the client knows the server is not an imposter.
So is there anything existing in AMS/FMS that helps with this, or is it necessary to write the entire procedure custom?
Maybe few of below would work:
1) Use SWF Verification: If it is fake AMS, ideally it should not have the same SWF with itself and hence SWF verification should fail.
2) Use Server-side methods to return which client is expecting or use Access or Auth plug-ins.
Thanks for the response, but this is not what I am looking for. SWF verification is performed at the server. It doesn't matter if the fake AMS doesn't know what a legitimate SWF hash is; it can simply let connections in, and then we have a client connected to a fake server. The same is true of the other access and authentication plugins. These basically let us construct rules about when a client should be allowed to connect to the server, based upon how many other clients are connected, how much network traffic they are generating, or some other server state. There is nothing to prevent a fake server from simply accepting clients.
I am looking for something where verification is performed by the client. If I were to write this custom, I would expect to use a standard challenge-response algorithm, probably based on digital signatures. I have not, however, yet located a crypto library that I can get to work on the media server.
Can you point me to a library that does work for digital signatures? Preferably I can use the same library on the media server and client.