c:\Program Files\Adobe\Flash Media Server 4.5\conf\_defaultRoot_\_defaultVHost_\Vhost.xml
...
<ResourceLimits>
<!-- Max number of clients that can connect to this vhost when running as local. -->
<!-- enforced by License key -->
<MaxConnections>-1</MaxConnections>
...
</ResourceLimits>
or if you have multiple vhosts, then adjust the path. I think this is what you are looking for. You can also control this number with server-side code:
http://help.adobe.com/en_US/flashmediaserver/ssaslr/WS5b3ccc516d4fbf35 1e63e3d11a11afc95e-7edfSSASLR.html. Look for entries application.clients.and/or application.rejectConnection.
