use adobe's rtmf protocol.
this is from, http://www.amazon.com/gp/product/1435460200/ref=s9_psimh_gw_p14_d6_i1?pf_rd_m=ATVPDKIKX0DE R&pf_rd_s=center-2&pf_rd_r=1Z09P04WYEXNYW66ZHD3&pf_rd_t=101&pf_rd_p=1389517282&pf_rd_i=507 846 :
Since Flash Player 10, you can create multiplayer games without the need of an intermediary server to facilitate player communication. The Flash Player can use a protocol (Adobe's Real-Time Media Flow Protocol) that allows direct peer-to-peer communication.
Instead of using server-side code to handle the game logic and coordinate game state among players, each peer in the network handles their own game logic and game state and communicates that directly to their peers and each peer updates their game state based on the data received from others.
To use peer-to-peer networking each peer must connect with an Adobe server. Peer-to-peer communication does not go through that server (or it that would not be peer-to-peer) but peers must stay connected with the Adobe server in order to communicate with each other.
To communicate with the Adobe server you should use your own server URL and developer key. That URL and key can be obtained at http://www.adobe.com/cfusion/entitlement/index.cfm?e=cirrus.
Below is a simple tic-tac-toe game that uses Adobe's peer-to-peer networking to pair-up players. The NetConnection class is used to establish a connection to the Adobe server while the NetGroup class is used for peer-to-peer communication.
"I have a project that require me to use flash(HTML5 also) to build a online video chat system"
kglad's solution will not work for html5.
There is currently no standard rtmp/rtmfp support for html5.
You could try to implement RtmpClient http://code.google.com/p/php-rtmp-client/
but that would not work for video/audio.
You would need to transcode the video to deliver it to html5 video.
This solution will work for flash based system although
some users may not be able to connect via UDP connections and may need to fall back to rtmp connections.
kglad's suggestion also is dependent on a currently "Free" Adobe server/service to do the initial connection handshake.
I would advise againt using Adobe Cirrus/Stratus because they will surely drop support it just as they have for several other products.
You would be better served to run a RTMP MediaServer ( Wowza, Red5, FMS/AMS ) yourself or use a paid for solution like influxis, Amazon or similar service.
Red5 is open source so if $$ is an issue I would use that.