The application.clients object available in the server side action script maintains a list of all client connected to the instance at that moment.
You can traverse the array and compare the Client.ip property to determine the max connections coming from an ip and accordingly accept/reject connections.
It can be as simple as this in the application.onConnect,
application.onConnect = function(clientObj)
var totalConnections = 0;
for (i = 0; i < application.clients.length; i++)
If (application.clients[i][ip] == clientObj[ip])
If(totalConnections > 10)
Above piece of code is untested so you might be required to make some changes to make it work but your solution should look something like that.