the dataReliable flag doesn't control whether you're using TCP or UDP. if you're using RTMP, you're using TCP because that's what RTMP goes over. if you're using RTMFP, you're using UDP because that's what RTMFP goes over.
if you're using RTMP, the videoReliable, audioReliable, and dataReliable flags have no effect because TCP is fully reliable all the time.
those flags only have meaning for RTMFP connections. when video/audio/dataReliable is set to true, RTMFP sends the corresponding message with full reliability, retransmitting it if necessary. if a reliable flag is false, RTMFP will not retransmit the corresponding message, and the message will only remain in the transmission queue for 1 second (so depending on network congestion/throughput, the message might not be transmitted at all).
messages of different types are sent with different priority in RTMFP. audio is highest priority, then NetStream.send() (data), then video. all RTMFP Groups modes (multicast, posting, directed routing, object replication) are lower priority than video. priority matters when bandwidth is tight.