You cannot play mp4 container directly. If you look into documentation of appendBytes you will see that this method understands FLV headers only. This means that you need to create an FLV header at runtime and append mp4's payload after that.
In any case, this is a very tedious task and you are better off sticking with OSMF framework that does that (it is open source).
is it possible to refine what you mean by "mp4's payload".
writing a standard flv-header (0x46, 0x4C, 0x56, 0x01, 0x05, 0x00, 0x00, 0x00, 0x09) is no problem at all, but what to write next?
which mp4-boxes are required? i guess "mdat" in some way, obviously.
any additional information / links?
"is it possible to refine what you mean by "mp4's payload"."
by payload I mean bytes that mp4 codec specific bytes that follow header.
Adobe has documentation (PDF) that describes video file specs. I don't have a link to it ready but you can find it on their site. Again, reverse-engineering OSMF code provides deeper insight in how bytearray may be handled.