RTMP is only used as a transport protocol in LCDS (there's no NetConnection API on the server), but you can definitely write chat-style applications with LCDS.
Take a look at the various samples contained in the samples.war that the installer lays down.
To display the LiveCycle Data Services ES samples home page, open a browser to the following URL once you're up and running:
http://localhost:portnumber/samples (portnumber is the HTTP port number; for example, http://localhost:8080/samples)
The samples app contains a variety of samples (including chat and data feed style examples).