are you using Flash Media Server? if so, you must update to FMS 4.0.3 or 4.5 (or above). Flash Player 11.1 changes the communication protocol with the server over the group "server control channel", FMS 4.0.2 and below don't understand the new handshake protocol.
the Cirrus service understands all versions of the group server control channel handshake. if you're using Cirrus, i'll have to look into this, but last time i checked (which was a little before the 11.1 release) P2P multicast still worked between all versions of Flash Player.
i just tried the following combinations:
1. publish from 11,1,102,55 subscribe on 10,3,183,5 and 11,0,1,152
2. publish from 10,3,183,5 subscribe on 11,0,1,152 and 11,1,102,55
3. publish from 11,0,1,152 subscribe on 10,3,183,5 and 11,1,102,55
(all using a Cirrus server, a simple MXML app [not OSMF], and a simple groupspec and P2P multicast)
all three combinations worked for me, although i had to reload the SWF to get access to the camera after it had been used by another process in a couple of instances which i didn't fully catalog.
i confirm works fine with cirrus but not with fms 4.5