FP 11+ Audio Encoding Issue With H.264 Video
I'm not sure if this issue is with the Flash Player when encoding the stream or with the Flash Media Server that is saving the stream so I'll be posting in both forums.
The problem is when publishing an audio and video stream from Flash Player 11 (Windows, FF) to the Flash Server 4.5 using the H.264 video codec. The audio portion of the recorded stream doesn't appear to be playable by anything and FFMPEG cannot process the audio stream. Here are the differences in the stream:
When using Sorenson as the video codec and recording the stream (flv) from the Flash server, the information about the audio stream from FFMPEG is this:
Stream #0.0: Audio: nellymoser, 22050 Hz, mono, s16
This clip can be played and processed by various players and FFMPEG with no problem.
When using H.264 as the video codec and recording the stream (f4v) from the Flash server, the information about the audio stream from FFMPEG is this:
Stream #0.1(eng): Audio: nmos / 0x736F6D6E, 22050 Hz, 1 channels
This clip does not play the audio and cannot be understood by FFMPEG.
I tried using the Speex codec as well and I have a similar issue as above (it doesn't play the audio stream and FFMPEG can't understand it).
Note the version of FFMPEG I'm using supports both Nellymoser and Speex so there should not be an issue.
1) Any idea why the codec appears different and won't play.
2) How to achieve the goal: send H.264 video stream with audio from the Flash Player and have a recorded stream with playable audio.
After you record the stream to Flash Media Server, process it with the F4V Post Processor tool to play it back on the desktop or using progressive download (you can stream it without using the Post Processor):
Even after running it through the post processor, the audio does not play. Read this thread (the posts in the middle discuss the audio problem, specifically no audio after running through the post processor): http://forums.adobe.com/message/3677448#3677448
It does not play using progessive download, won't play in any players, and can't be recognized by FFMPEG.
I'm not sure if this is an issue with the Flash Player when encoding it, or the Flash Media Server when saving it, but the fact you can see the same audio codec appears differently to FFMPEG so something is going on. This is a major problem!
My FP11 H264 recording content plays fine, when live and as vod content via FMS. However nothing recognises it like, media info, mp4info, mpui or FFmpeg when inspected and even VLC Player can not play it. FLVCheck says its ok. MP4UI did say the atom size was too small, so I can only think FP11 is not creating the container within the MP4 spec.
I would like this raised as a bug! Point me to the correct url please.
Not sure about FMS, but I find that when using Wowza, if I publish h.264 / speex from the flashplayer, and record to an FLV container, FFMPEG is able to process the file.
Keep in mind that neither Nellymoser Asao nor Speex are MP4 compliant, so I would expect most applications to fail to read those files.
So did you do what Jay said and it now can be processed? So Jay you are saying instead of saving it as MP4, I should tell FMS to save the file as an .flv, then run it through FFMPEG to convert it to a standard MP4? Sounds strange, but if it works I'll try anything because as of now there is no way to play those files without sending them through the server.
One more bit of advice...
The AAC encoder in FFMPEG (FAAC) is terrible, and tends to result in poor quality at low bitrates. If you're transcoding to AAC, you'll likely want to use 128kbps or higher. If you workflow and endpoint targets allow for it, I'd go with MP3 audio.
Thanks for letting me know. I'm already using that combination (H.264/MP3) with the clips generated by the Flash Media Live Encoder so I'm good there. The problem of course is there is no way to encode MP3 audio in FP11 along with the H.264 video. For regular voice, do you find Nellymoser or Speex preferable to start with (knowing it's going to end up as MP3 in the end)? I've always just used Nellymoser. Obviously if MP3 was available, this wouldn't be an issue.
From a quality perspective, I prefer Nellymoser. Speex encoding in FP is limited to 16kHz, where Nellymoser is capable of 44.1khz. For voice, the difference isn't all that noticeable, but if there's any music involved, you'll find that the sounds is a bit dull when using speex, as the higher frequencies aren't represented.
Also, if your application involves multiway communication and you want to take advantage of AEC, your only option is Speex, as AEC doesn't work with Nellymoser.
From a licensing standpoint, Speex is the clear winner. To be completely legal, you'd need a license from Nellymoser to decode the audio on the server side using FFMPEG, and that license is $7500 per year per server the last time I checked. Speex on the other hand is not patent encumbered, and (AFAIK) requires no licensing.
That said, if you're in a position where codec licensing is an issue, you'll want to look into an MP3 encoding license for your implementation as well.
Jay, unfortunately that didn't work.
When I send H.264 video and Speex or Nellymoser audio and try to record it on the server like this:
mystream=Stream.get("mp4:output.mp4") = I get the video, but the audio is not recognized by FFMPEG
mystream=Stream.get("mp4:output.flv") = I get the video, but the audio is not recognized by FFMPEG
mystream=Stream.get("output.flv") = I get the audio properly, but the video stream does not record (which looks like it follows what it says in the documentation)
Is there any special tags I need to use with FFMPEG to be able to read it or a certain way on the server to specify to record it?
None that I'm aware of.
As I mentioned before, I've been using Wowza lately (and h.264/speex to flv works fine there), so perhaps this is just something that FMS won't do. Maybe someone from the FMS engineering team will have suggestions about how to record a file that anything other than FMS can read.
So from your experience you wouldn't say it's an issue with how the player is encoding it, but rather how FMS is saving it on the back end? At least determining that would help isolate where to go for help (although I don't see any other suggestions coming from employees in this thread yet).
Adobe - any ideas?
Using my latest H264 recording application I was able to use FFmpeg to transcode to AAC. I understand you need to use the flvpackager to make the recorded mp4 playable direct from the desktop.
JC I suggest posting your application code (bits you are using for h264 recording) for forum users to debug.
I know this thread has moved beyond the F4V Post Processor (thanks for all the info, Jay), but I wanted to let you know that the audio bug mentioned in the previous thread has been fixed. Unfortunately, I didn't grab the fix build number and I don't have access to the bug base anymore, but maybe someone from Adobe could grab it for us.
I'm confused now.
1) Are you saying that recorded files will now have the audio playing/recognized properly outside of being served by FMS?
2) What was "fixed"? Was it the Flash Player encoder, the F4V Post Processor, or Flash Media Server?
I think what Jody was getting a was the inclusion of ulaw and alaw (pcm) audio in the flashplayer. The trouble there is, unless you're going to use a really low sample rate, the bitrate required for pcm is going to be very high, and (if I'm not mistaken), the flashplayer will not perform AEC for pcm.
If you check, you'll see in my other thread http://forums.adobe.com/thread/921501 an employee claims this was fixed with the latest version of FP11. However, I tested it and the problem still persists.
He said if it didn't work to submit a bug, so I did here: https://bugbase.adobe.com/index.cfm?event=bug&id=3044308 If you want it fixed, please go vote for it.
I have the feeling this is an FMS issue. When I play the H.264/speex FLV recorded through Wowza in flashplayer 22.214.171.124, it plays back just fine for me, both streaming from Wowza, and via progressive download.
I see your bug report has been closed without resolution. I raised the same problem in this thread:
And got the reply:
Thank you for the info. Apparently this is a bug in the Post Processor tool. The responsible team is aware of it and they are working on a fix. Of course I can't tell you when it will be fixed or released but it is in the works.
that was in August and nothing since. It seems a fairly major bug that you can't have audio in your saved video. I can't find anywhere to contact that team for an update, and the problem seems to have been forgotten or ignored. If anyone from Adobe is looking in, an update would be appreciated.
Yes you are correct that they closed my bug with no resolution claiming the issue is not in the Flash Player.
Adobe - there is no means in the current Post Processor to convert the audio to anything playable. How can we not have audio in our recorded clips?
Obviously this is a serious issue that many people are having. Can you give us an update on this or what to do now that you have closed the bug I submitted above?
When encoding live in H264 with the FP 11 then playing it live on another client (through FMS r) the video sometimes (1/3 of times) has major skips or jerks. It's almost as if the video goes back to some older keyframes and then returns to live. Same experience with non-h264 produces smooth video.
If the server (FMS ) records the video, the replay looks smooth, which is strange because I would have assumed the FP was not encoding well with H264.
Has anyone had this issue? please reply
@arnold.25: yes i am also facing similar issues. when i googled for it, found out that its because of either the buffer or bandwidth causing the issue. so need to set some buffertime on your playback client end or the bandwidth should be sufficeint enough to play the stream. but when i tried the first method it intriduced a lot of delay and the second method is expensive. so i am in search of some other solutions for this.
I tried this out, ffmpeg is rendering the file fine at my end. I used ffplay to try and play this, I am getting video and audio.
I have tried h264 with both nellymoser and speex.
If still you are having this problem please pass me your file so that I can investigate further.
Thanks for your time. I could resolve this at my end. It was nothing but the swf version mismatch on my compiler-settings. I was using the Flash player 11.2 and the compilet option was setting -swf-version to 14. So H264 encoding had this jerking issue. But once give the appropriate version # it got resolved. I got the version detatils from here