I am developing a Mac app which needs to provide a HTTP Live Stream (just the last 2 seconds or so) for the main screen (Desktop).
I was thinking of the following process:
Create a AVCaptureSession with a AVCaptureScreenInput as input (sessionPreset = AVCaptureSessionPresetPhoto)
Add a AVCaptureVideoDataOutput output to the session
Capture the frames (in kCVPixelFormatType_32BGRA format) in captureOutput:didDropSampleBuffer:fromConnection: and write these to a ffmpeg process for segmenting (using a pipe or something) that creates the MPEG-TS and playlist files.
Use an embedded HTTP server to server up the segmented files and playlist file.
Is this the best approach and is there is no way to circumvent the ffmpeg part for encoding and segmenting the video stream?
What is the best way to pipe the raw frames to ffmpeg?
It sounds like a good approach. You can use have ffmpeg output to a stream and use the segmenting tools from Apple to segment it. I believe that the Apple tools have slightly better mux rate, but it might not matter for your use case.
Related
My setup is as follows:
OBS Studio to create the video feed
Ant Media Server to distribute the stream
Now I'm building an app that will display this stream and I'm currently using ExoPlayer, however I'm having a hard time getting it to work for both RTMP and HLS, I read some where that I could embed a webplayer in my app would that be easier? Here is my code for ExoPlayer:
//RTMP Url
String url = "rtmp://192.168.1.244/WebRTCApp/379358104902020985845622";
BandwidthMeter bandwidthMeter = new DefaultBandwidthMeter();
TrackSelection.Factory videoTrackSelectionFactory =
new AdaptiveTrackSelection.Factory();
TrackSelector trackSelector =
new DefaultTrackSelector(videoTrackSelectionFactory);
SimpleExoPlayer player = ExoPlayerFactory.newSimpleInstance(this, trackSelector);
PlayerView playerView = findViewById(R.id.simple_player);
playerView.setPlayer(player);
/*
Create RTMP Data Source
*/
RtmpDataSourceFactory rtmpDataSourceFactory = new RtmpDataSourceFactory();
MediaSource videoSource = new ExtractorMediaSource.Factory(rtmpDataSourceFactory)
.createMediaSource(Uri.parse(url));
player.prepare(videoSource);
player.setPlayWhenReady(true);
Any help on this would be much appreciated.
Most online video streaming use Adaptive Bit Rate streaming (ABR) protocols to deliver the video, mainly HSL and DASH this days.
Most Media players, like ExoPlayer, support these protocols well, although they are complex and evolving protocols so there are always edge cases.
Many video conferencing applications use WebRTC which is a real time optimised protocol - the usual approach is to use a WebRTC client for this type of stream.
The difference between the two approaches from a streaming latency point of view, at a very high level, is:
ABR protocols prioritise quality and avoiding interruptions and buffer enough of the video to try to gaurantee uninterrupted playback. They are usually aimed at movie and live video streaming services. Even for low latency implementation the latency is measured in multiple seconds and more.
WebRTC prioritises latency and sacrifices quality if necessary. It is aimed typically at real time sensitive applications like video conferencing where it is important not to fall behind the discussion even if it means a temporary video glitch or even brief interruption in video. Latency is usually measured in sub seconds.
Any Media Server comes from the WebRTC side although recent versions support HLS /CMAF and Low Latency DASH (these are still higher latency than WebRTC generally as noted above).
For your service, if you are able to use a DASH or HLS stream you may find that it is an easier path with ExoPlayer. If you look at the demo app for example you will see DASH and HLS streams but no RTMP ones. You can easily extend or modify the demo app to play your own HLS or DASH stream and this is often an easy way to start - look at the sample material in the assets/media.exolist.json and add your own URL:
https://github.com/google/ExoPlayer/blob/aeb306a164911aa1491b46c2db4da0d329c83c65/docs/demo-application.md
However, ExoPlayer should also support RTMP via an extension if this is your preferred route - there is a specific extension that allows this:
https://github.com/google/ExoPlayer/blob/0ba317b1337eaa789f05dd6c5241246478a3d1e5/extensions/rtmp/README.md
In theory you simply need to add this dependency to your application:
if your application is using DefaultDataSource or DefaultDataSourceFactory, adding support for RTMP streams is as simple as adding a dependency to the RTMP extension
It would be worth checking the issues list in this repository for any recent issues and/or workarounds.
I'm trying to take two videos and transform them with ffmpeg into a single video. It works great if you take the two videos, run them through ffmpeg and then serve that file up via an API. Unfortunately the upper range for these videos is ~20 minutes, and this method takes too long to create the full video (~30 seconds w/ ultrafast).
I had an idea to stream the output of the ffmpeg command to the client which would eliminate the need to wait for ffmpeg to create the whole video. I've tried to proof this out myself and haven't had much success. It could be my inexperience with streams, or this could be impossible.
Does anyone know if my idea to stream the in-progress output of ffmpeg is possible / feasible?
you should check hangfire. I used this for running the process on the background, and if it needs a notification, signalR will help you
What do you mean by "streaming" ? Serving the result of your command to an http client on the fly ? Or your client is some video player that play the video (like a VLC player receiving a tcp stream of 4 IP cameras) ?
Dealing with video isn't a simple task, and you need to choose your protocols, tools and even hardware carefully.
Based on the command that you send as an example, you probably need some jobs that convert your videos.
Here's a complete article on how to use Azure Batch to process using ffmeg. You can use any batching solution if you want (another answer suggests Hangfire and it's ok too)
I'm trying to build a system for real-time analysis on server for video streamed from the client using WebRTC.
Here is what I currently have in mind. I would capture the webcam video stream from the client and send it (compressed using H.264?) to my server.
On my server, I would receive the stream and every raw frame to my C++ library for analysis.
The output of the analysis (box coordinates to draw) would then be sent back to the client via WebRTC or a separate WebSocket connection.
I've been looking online and found open-source media server like Kurento and Mediasoup but, since I only need to read the stream (no dispatch to other clients), do I really need to use an existing server? Or could I build it myself and if so, where to start?
I'm fairly new to the WebRTC and video streaming world in general so I was wondering, does this whole thing sound right to you?
That depends on how real-time your requirements are. If you want 30-60fps and near-realtime, getting the images to the server via RTP is the best solution. And then you'll need things like a jitter buffer, depacketization etc, video decoders, etc.
If you require only one image per second, grabbing it from the canvas and sending it via Websockets or HTTP POST is easier. https://webrtchacks.com/webrtc-cv-tensorflow/ shows how to do that in Python.
I have a client/server audio synthesizer where the server (java) dynamically generates an audio stream (Ogg/Vorbis) to be rendered by the client using an HTML5 audio element. Users can tweak various parameters and the server immediately alters the output accordingly. Unfortunately the audio element buffers (prefetches) very aggressively so changes made by the user won't be heard until minutes later, literally.
Trying to disable preload has no effect, and apparently this setting is only 'advisory' so there's no guarantee that it's behavior would be consistent across browsers.
I've been reading everything that I can find on WebRTC and the evolving WebAudio API and it seems like all of the pieces I need are there but I don't know if it's possible to connect them up the way I'd like to.
I looked at RTCPeerConnection, it does provide low latency but it brings in a lot of baggage that I don't want or need (STUN, ICE, offer/answer, etc) and currently it seems to only support a limited set of codecs, mostly geared towards voice. Also since the server side is in java I think I'd have to do a lot of work to teach it to 'speak' the various protocols and formats involved.
AudioContext.decodeAudioData works great for a static sample, but not for a stream since it doesn't process the incoming data until it's consumed the entire stream.
What I want is the exact functionality of the audio tag (i.e. HTMLAudioElement) without any buffering. If I could somehow create a MediaStream object that uses the server URL for its input then I could create a MediaStreamAudioSourceNode and send that output to context.destination. This is not very different than what AudioContext.decodeAudioData already does, except that method creates a static buffer, not a stream.
I would like to keep the Ogg/Vorbis compression and eventually use other codecs, but one thing that I may try next is to send raw PCM and build audio buffers on the fly, just as if they were being generated programatically by javascript code. But again, I think all of the parts already exist, and if there's any way to leverage that I would be most thrilled to know about it!
Thanks in advance,
Joe
How are you getting on ? Did you resolve this question ? I am solving a similar challenge. On the browser side I'm using web audio API which has nice ways to render streaming input audio data, and nodejs on the server side using web sockets as the middleware to send the browser streaming PCM buffers.
I need a solution to play a segment of an mp3. I have a few 1,000 audio files which are currently stored on Amazon S3, and would like to allow users to play them, however I would like to limit the play length to 30 seconds or so in the middle of the recording.
I'm not sure if I need to create an entirely new file (snippet) such as I would for a thumbnail if it were an image, or if it's possible using some player/steam to safely limit it that way so they cannot access the whole song.
I'm coming from a Rails environment and using Paperclip to handle the files and JPlayer to play them if it matters.
Any pointers or best practices?
This is possible by using the HTTP "Content-range" header. This header says 'please just give me the bytes from here to here and ignore the rest'. If the web server is set up to handle them (Apache is for instance), then you get a 206 response with a body of just those bytes.
You must create a small proxy application that effectively acts as a gateway between the listener and Amazon.
To see if your host will respond try this from the command line:
curl -v -I http://www.mfiles.co.uk/mp3-downloads/01-Tartaros%20of%20light.mp3
Where the url is one of yours. If you are lucky you will see:
Accept-Ranges: bytes
Content-Length: 5284483
This means that the server does accept the Content-range header and the full length of the file is 5284483 bytes long.
Let's request the first third of the file:
curl -H'Range: bytes=0-1761494' http://www.mfiles.co.uk/mp3-downloads/01-Tartaros%20of%20light.mp3 > /tmp/test1.mp3
You should now be able to play /tmp/test1.mp3 and hear the first third of the track.
The next step is to create a proxy application. A good approach would be to use https://github.com/aniero/rack-streaming-proxy but you would probably need to fork the project to send the 'Range: bytes=0-1761494' header. Alternatively have a look at Sinatra.
A bonus here is that because you are proxying the remote server, you could obfuscate the actual URL of the file by having a simple database table with an ID for each file. I would suggest writing a small script that also stores the byte length of each file, so that you don't have to calculate the range for each request.
Thus a GET to "/preview/12345" would proxy "http://amazon.com/my_long_url" and give you just the first third of the file.
On top of that, you could put Varnish in front of your own server, which would cache these partial MP3 files and mean that you are not having to constantly go back to Amazon to get the files.
Unfortunately, you'll need to make new snippets - there isn't really a way to tell a user's browser "download this entire mp3 file, but only play and allow access to the middle 30 seconds".
i think it is simplier to solve the problem in the client side.
Are you using flash to play the audio files?
If yes, I have done something similar (but with videos) using JWPlayer (it also supports audio files).
You can develop a custom plugin to control the snippet you want to play and then stop the audio file and show a message or something like that.
This solution combined with signed urls or/and rtmp streaming with CloudFront can be very safe.
Due to the mp3 format limitation, you cannot seek to the arbitrary frame in the middle of the song and start transmission from that point.
So, there are basically three options:
1. Create new files offline. Very easy, but space consuming.
2. Transcode files on the fly. CPU consuming, degrades quality.
3. Limit playback with first X seconds: just peek into the song' header, get its bitrate and calculate size of the byte chunk to serve
And don't ever transmit more than you need: people will manage to intercept the stream and save it to disk (business side); save your users' traffic (good karma).