one NetConnection is better than two, because congestion state is shared within the NetConnection but not between them (in your usage). as far as what's happening on the wire, there should be no difference between using two NetStreams and using four.
i can see one potential problem, though: P2P data (NetStream.send()) is sent with "Normal" priority. P2P audio is sent with "Higher" priority, but P2P video is sent with "Lower" priority. also, video is sent with full reliability, so if you are sending a LOT of NetStream.send data, the video could become delayed, since it is sent with full reliability. the audio should stay current, though, as it has the highest priority of what you're sending.
one thing that's possible is that certain networks may have lots of buffering in them (wireless networks are notorious for this), so if you are sending a lot of data over a link, it may be buffered by the network, which must necessarily drive up latency. a test you could do to see if this is what's happening is to do a command-line "ping" from one of the peers to the other while you're experiencing the high a/v latency. if the ping time is really high (in line with the observed latency) then there's little that can be done about that, except to send less data.
in the future, selecting partial reliability for video will be possible, which might also help mitigate this in highly congested situations.
and the differents of the P2P autdio,video data and P2P data(with NetStream.send()) ? If the P2P data(with NetStream.send()) can be lost (or NetStream would retransmit it when it loss) ?