I am developing an application where I need musicians to play together and I want to have an audience listen to it. I have created my publisher clients and they can create the streams, but now I want to take those multiple streams and combine them so that you can hear the guitar, singer, etc. all together.
Is this possible, do I need to tell the server side script to combine them and create a new stream? Any help would be appreciated!
In this use case, each musician would publish a stream, and subscribing clients would subscribe to each of the streams separately. FMS has no means of mixing multiple streams into a single stream.
That said, I don't think the experience is going to be very good. In order for the music to be cohesive, you'd need to have zero latency on every stream, which will be nearly impossible to achieve. Latency of only 50-100ms (which is typical when everyone has a fast connection) on each client would put the musicians so far out of sync that it would be a bad listening experience.
Not with FMS. On the server side, you can only attach a stream to another stream... you can't attach two streams to one stream.
No amount of buffering will save you from transmission latency. The use case you've described really requires no buffer and no latency (each stream would need to bounce off the server and back to each musician with no latency in order for them to hear each other), which is a pipe dream when it comes to FMS.
So, you have several points of latency to consider:
1. Publishing latency between each musician's location and the server
2. Latency on the streams going back to the musicians
3. Latency on the streams going out to the listening clients.
In the case of item 3, you could use a buffering strategy to bring the streams into sync, but for items 1 and 2, I think you'd be fighting a losing battle. If you introduce a buffering strategy to sync what each musician hears, each musician will be hearing the streams at a different offset from "real time" (that offset being determined by the buffer time and latency combined). With that in mind, none of the musicians will be able to keep time accurately. Even if there is a flux of only 200ms or so (which is difficult to achieve consistently even with only one stream), having musicians playing 200ms out of sync would make for a very sloppy performance (and would likely make the musicians crazy in the process).
Thanks for the input, the theory we are discussing here is what I was always concerned about. Looking over several other sites that appear to offer this "No Latency" jamming mode, Im just wondering how they can do this? Have a look at:
Do you think that they are using some other means besides a media server to connect these streams, I would imagine they could use P2P but there would still be the latency right?
What Really made me want to get into using the FMS was the fact that I could have others connect to this stream and hear the final output.
So with all the variables does this goal even seem possible, through any means, FMS or not?
I haven't used it myself, but the info on the ejamming site indicates that they're using a P2P protocol, and I suspect it's UDP based.UDP is more suitable for a use case such as this, as UDP is a lossy protocol. Additionally, the P2P nature of ejamming's application takes the trip to the server out of the picture, as the data doesn't need to bounce off the server. This reduces the time it takes to get the data from one client to another.
FMS, on the other hand, uses TCP/IP, which is a lossless protocol. Any data that doesn't make it to the server on the first trip (and in the proper sequence) has to be re-sent, which adds to latency on the stream. That, combined with encoding and transmission latency really removes reliability from an application that depends on consistently low latency. That's not to say it isn't possible, it just isn't reliable.
All that said, you might consider testing with RTMFP. RTMFP is a UDP based P2P protocol supported by Flashplayer 10 (http://labs.adobe.com/technologies/stratus/). RTMFP requires that you use a rendezvous service, and currently, the only such service is Adobe's stratus service. The word is that future versions of FMS will support RTMFP rendezvous services, but there is no official word on whether "future versions" means the next version, or a version further down the road. Of course, all of this introduces a few new considerations:
1. Each musician will need a lot more upload bandwidth, as each subscriber would need to connect to the publisher to receive the stream. For example, let's say you have 5 musicians. That means each musician needs to serve his/her stream to each of the other 4 musicians (RTMFP does not support multicast/swarming)
2. This approach doesn't satisfy the need to play the streams to a listening audience (RTMFP is not suitable for serving streams to a large audience). You'd still need to publish a stream from each musician to FMS over RTMP, come up with some sort of timecode logic to track latency, and then try to sync those streams on the playback client.
I tend to shy away from the term "impossible" (that word tends to lead to foot-in-mouth), but in this case, it seems to me that an FMS-centric solution would be less than reliable.
Hey Jay, thanks for all the info, I think this project might have to require the musicians to all be together and transmit through a pre-mixed channel. I really hate to abandon the idea completely (multiple player in different locations + mass audience), but I think even if I create a P2P program via some of Java's great music librarys it would still require a centralized server to combine all these streams and push them out to a large audience base.
What do you think?
Well, I don't have the Java chops to offer a worthwhile opinion, but if java can handle mixing the streams, perhaps you can use java on the server side. There are a number of java rtmp server implementations out there, and if you can push the published streams over UDP (or some other lossy transport) to the java server app with controllable latency, you might be able to do the whole thing on the server side with Java.
I would like to hear more about the way you want to use FMS. It would help us understand the new interesting senarios and your expectations from future media platforms.
PS: sent PM which contains the email address which can be used for communications.