I'm trying to play a MP4-Stream. The stream is send from my android phone. The problem is, that the moov atom, which is needed to play the mp4, is only written if the phone is done with the recording progress. So at the moment I'm only to play the streamed data, after the recording has finished.
My idea was to write the ftyp and moov atom by myself, so that the streamed data can be played, while the phone is recording.
I tried to use the moov atom from another videofile but this didn't work. I also have read, that normally it's impossible to build a moov atom, if only the mdat atom is given.
But in my case I know the recording conditions, like framerate, etc..
So my question is, is it possible to generate a valid/useable moov atom for the incoming stream if I know the recording parameters?
It's possible. I've done it 4 years ago to implement "live streaming" to original iPhone. Just fill STSZ and STCO atoms with constant size frames, then pad each frame with zeros. Yeah, size will be huge, but you'll get real live streaming :-)
It seems to be rather impossible to stream not yet finished mp4 file because player would need special tables with chunks and offsets to locate every data sample. You can fake FTYP, MOOV and other atoms, but you can't generate all tables without having file complete. Better strategy would be to generate many short mp4 files and send them file by file...
Related
Boss handed me a bit of a challenge that is a bit out of my usual ballpark and I am having trouble identifying which technologies/projects I should use. (I don't mind, I asked for something 'new' :)
Job: Build a .NET server-side process that can pick up a bitmap from a buffer 10 times per second and produce/serve a 10fps video stream for display in a modern HTML5 enabled browser.
What Lego blocks should I be looking for here?
Dave
You'll want to use FFmpeg. Here's the basic flow:
Your App -> FFmpeg STDIN -> VP8 or VP9 video wrapped in WebM
If you're streaming in these images, probably the easiest thing to do is decode the bitmap into a raw RGB or RGBA bitmap, and then write each frame to FFmpeg's STDIN. You will have to read the first bitmap first to determine the size and color information, then execute the FFmpeg child process with the correct parameters. When you're done, close the pipe and FFmpeg will finish up your output file. If you want, you can even redirect FFmpeg's STDOUT to somewhere like blob storage on S3 or something.
If all the images are uploaded at once and then you create the video, it's even easier. Just make a list of the files in-order and execute FFmpeg. When FFmpeg is done, you should have a video.
One additional bit of information that will help you understand how to build an FFmpeg command line: WebM is a container format. It doesn't do anything but keep track of how many video streams, how many audio streams, what codecs to use for those streams, subtitle streams, metadata (like thumbnail images), etc. WebM is basically Matroska (.mkv), but with some features disabled to make adopting the WebM standard easier for browser makers. Inside WebM, you'll want at least one video stream. VP8 and VP9 are very compatible codecs. If you want to add audio, Opus is a standard codec you can use.
Some resources to get you started:
FFmpeg Documentation (https://ffmpeg.org/documentation.html)
Converting raw images to video (https://superuser.com/a/469517/48624)
VP8 Encoding (http://trac.ffmpeg.org/wiki/Encode/VP8)
FFmpeg Binaries for Windows (https://ffmpeg.zeranoe.com/builds/)
I've been digging through the Media Source Extension examples on the internet and haven't quite figured out a way to adapt them to my needs.
I'm looking to take a locally cached MP4/WebM video (w/ 100% keyframes and 1:1 ratio of clusters/atoms to keyframes) and decode/display them non-sequentially (ie. frame 10, 400, 2, 100, etc.) and to be able to render these non-sequential frames on demand at rates from 0-60fps. The simple non-MSE approach using the currentTime property fails due to the latency in setting this property and getting a frame displayed.
I realize this is totally outside normal expectations for video playback, but my application requires this type of non-sequential high speed playback. Ideally I can do this with h264 for GPU acceleration but I realize there could be some platform specific GPU buffers to contend with, though it seems that a zero frame buffer should be possible (see here). I am hoping that MSE can accomplish this non-sequential high framerate low latency playback, but I know I'm asking for a lot.
Questions:
Will appendBuffer accept a single WebM cluster / MP4 Atom made up of a single keyframe, and also be able to decode at a high frequency (60fps)?
Do you think what I'm trying to do is possible in the browser?
Any help, insight, or code suggestions/examples would be much appreciated.
Thanks!
Update 4/5/16
I was able to get MSE mostly working with single frame MP4 fragments in Firefox, Edge, and Chrome. However, Chrome seems to be running into the frame buffer issue linked above and I haven't found a way to pre-process a MP4 to invoke this "low delay" mode. Anyone have any clues if it's possible to create such a file with an existing tool like MP4Box?
Firefox and Edge decode/display the individual frames immediately with very little latency, but of course something breaks once I load this video into a Three.js WebGL project (no video output, no errors). I'm ignoring this for now as I'd much rather have things working on Chrome as I'll be targeting Android as well.
I was able to get this working pretty well. The key was getting Chrome to enter its "low delay" mode by muxing a specially crafted MP4 file using modified mp4box sources. I added one line in movie_fragments.c so it read:
if (movie->moov->mvex->mehd && movie->moov->mvex->mehd->fragment_duration) {
trex->track->Header->duration = 0;
Media_SetDuration(trex->track);
movie->moov->mvex->mehd->fragment_duration = 0;
}
Now every MP4 created will have the MEHD fragment duration set to 0 which causes Chrome to process it as a live stream.
I still have one remaining issue related to the timestampOffset property which in combination with the FPS set in the media fragments control the playback speed. Since I'm looking to control the FPS directly I don't want any added delay from the MSE playback engine. I'll post a separate question here to address that.
Thanks,
Dustin
I have mp4 video file,which i need to load on my page,i am using MSE for that,but i don't know how can i get my video in segments with .m4s extensions,with header.m4s as parent segment with all information about my video file stored in it?Please help.
I believe that if a video is embedded on the website, it can be downloaded.
The only thing you could do is make it difficult for download.
This might be helpful. It says using a flash video is a good option to make downloading videos a bit difficult. Never used it but you could give it a try.
To protect the video, you should probably not try to artificially obfuscate the video loading. MPEG DASH supports encrypted MP4 video and common encryption (CENC), that could be a thing you can look into.
I am trying to record audio from a microphone/iSight camera from Mac to a NSData object.
I have tried to do it using QTKit, but I found out that you could only save it as a .mov file.
But the fact is that I want to recode the audio into a FLAC file. Is that posible, or I'll need to use another framework?.
Thanks.
Grab the source for VLC (if you can deal w/GPL -- it has limitations on use that many find onerous) and have a read. It does transcoding, amongst other things.
Beyond that, one dead simple approach is to save as AIFF and then use a command line tool (via NSTask) to do the conversion.
Or you could just go with Apple Lossless -- it is open source these days.
Of course, this also begs the question; why do you need lossless compression when recording voice [low bandwidth in the first place] via a relatively sub-par microphone?
I have a multimedia application that among other things converts video using FFMpeg. Video conversion being the pain that it is, I have in my test suits some tests that check our ability to convert various video formats, with emphasis on sample videos known not to work.
A common problem we've noticed from users is that some videos end up with their audio desynched after being processed, and I am looking for a way to check this in my tests.
Extracting the audio portion of the resulting videos is not a problem.
My best idea so far would be to check the offset of the first non-silence at both the beginning and end and compare each between the two videos, but I'm hoping someone smart has a better idea.
The application language/environment is Java, but since this is for testing, I'm free to use any toolset.
The basic problem is likely that the video and audio are different lengths. Extract the audio and test its length vs. the video length. If they are significantly different (more than maybe .05 sec, I'm not really sure what is detectable as "off"), then there's a problem.
To fix it, re-encode the audio to match the video length, and then put the audio and video back into a container format.