I have a transfer of a live video stream from a server to the javascript function of a client browser:
server: gstreamer x264enc-hardware ! whatever-I-want ! appsink
=== transfer of data stream with a proprietary protocol ===>
HTML5 browser client: javascript function receives data sent by the appsink
In other words, I'm trying to display a h264 live stream created on a server with a proprietary transfer protocol, the data re-appearing in a javascript function inside an HTML5 client browser.
I was thinking of using MediaSource MSE in the browser to decode h264 and display the image.
Note that the video stream settings (video only, resolution, bandwidth) are fixed and known on both sides. So, everything can be hard-coded and the purpose is not to implement a generic solution.
What could I use on the server side (replacement of the "whatever-I-want" gstreamer plugin) so that the work in the HTML5 browser is not too complicated?
One solution would be to do nothing on the server side and use the broadway.js library to decode NALU h264 in javascript but it obviously doesn't leverage video MediaSource and the decoding capability of the browser.
Could I use Gstreamer avmux_dash and hope that MediaSource can input the transmitted data?
Alternatively, how could I create "MP4 fragments" and could MediaSource read them "easily"?
One approach, that has been used by some major players in the past to translate from one streaming protocol to another, is to receive in your proprietary transfer protocol and then re-package into HLS or DASH on a local server on the device.
You can then stream from that local host to a regular HLS or DASh player on the device.
It sounds inefficient (it is inefficient) but it works, even on mobile devices which their lower processing and power capabilities.
Related
I know how to access audio input devices via getUserMedia() and route them to the WebAudio API. This works fine for microphones and such. But in my use-case, I would rather like to hook into the audio stream of an output device. The use case is that I want to create a spectrum analyser for audio coming from a digital audio workstation (DAW) running on the same PC.
I tried to enumerate the devices and call getUserMedia() with the device id of an audio device, but the stream returned only showed silence data. The only solution I found so far is to install an audio loopback device (like Soundflower on Macs) to route the DAW's output to and then use this as an input device for getUserMedia(). But this will require the user to install 3rd party software.
Is there any way to hook directly into the audio stream of an output device instead, before it is actually sent to the physical device (speaker or external soundcard)?
This can be achieved using the desktop capture APIs (chrome.desktopCapture.chooseDesktopMedia). An example for chrome is included here
I am using kurento's master git to make a WebRTC to RTP bridge.
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
HttpGetEndpoint httpEndpoint=new HttpGetEndpoint.Builder(pipeline).build();
org.kurento.client.Fraction fr= new org.kurento.client.Fraction(1, 30);
VideoCaps vc= new VideoCaps(VideoCodec.H264,fr);
httpEndpoint.setVideoFormat(vc);
AudioCaps ac= new AudioCaps(AudioCodec.PCMU, 65536);
httpEndpoint.setAudioFormat(ac);
webRtcEndpoint.connect(httpEndpoint);
However inspite of this the output video playing is encoded to webm . I have tried various other approaches as well ( using RTP ENdpoint , using Gstream filter , using VLC HTTP to RTP streamer ) . however no method gives me a video playable on safari and IE ie H264 encoded . Requesting media developers and kurento team for help .
Safari and IE do not support RTP/H.264. From you code, I understand that you are trying to create a WebRTC to tag bridge. In that case, the HttpGetEndpoint will provide media through HTTP pseudostreaming. However, Kurento only provides that type of live HTTP pseudostreaming in WebM format. To be best of my knowledge, neither Safari nor IE support WebM, hence what you want to do will not work independenlty on the caps you force to the HttpGetEndpoint. You will be only able to see it working on Chrome, Fireforx or other browsers with WebM support.
The only solution for you could be the HttpGetEndpoint providing media in MP4 format (or any other format supported by IE and Safari), but creating the live stream in that format is very tricky and we (the Kurento team) did not had the time for implementing that and this feature is not in our short term roadmap.
However, we have many users integrating WebRTC with IE and Safari using RTMP. In that case, you need to integrate Kurento with an RTMP capable media server (this can be done in different ways) and later let the RTMP media server to serve media to the browsers.
I want to let WebRTC encoded and play h264(NAL) stream(local file).
In the WebRTC tutorial, getUserMedia is use for get local camera connecting to the system, I don`t know if the getUserMedia function support
capture the local stream file like h264 stream.
If it doesn't work that way, may be I should modify WebRTC source code(I'm studying it).
Here is the question, If i change WebRTC code, how can i integration the new code into browser? Made it a plugin?
Firefox supports an extension to the <video> element that you can use to do this.
First, set the source of a video element:
v1.src = "file:///...";
Then you can call the (currently prefixed) mozCaptureStream or mozCaptureStreamUntilEnded function to get a MediaStream.
stream = v1.mozCaptureStream();
The proposed specification.
Note however that you need to ensure that the file is same origin with respect to the page. The same origin rules for file:/// are probably going to cause issues. Otherwise your MediaStream isn't going to be accessible to you. One way to ensure that is not to set the location directly, but to load the file using an <input type="file"> element.
As noted in other answers, Firefox currently only supports the baseline profile of H.264.
First, you are right getusermedia will not work for you. However, there are a couple of options.
Hack a stream together using RTCDataChannel. Breaking up the media stream and delivering each packet and then handling it on the client side.
Take a look at this demo for precorded media streams. I do not believe that H264 is addressed but it could help you on your way(probably for Firefox only)
Use some sort of webrtc breaker/endpoint that is native to stream the file. I know specifically that others(including myself) have streamed H264 to Firefox through the Janus-Gateway
Couple of asides:
Firefox only supports Baseline profiles in streaming h264 for a webrtc peerconnection
Chrome does not support h264 for webrtc at all
Are you trying to have getUserMedia return a h.264 encoded stream?
In which case, today it will only be possible with Firefox today, under some specific environment (cisco 264 plugin installed) and only for the base profile.
Chrome promised in november to add this capacity, but there is no timeline that I know of Expect at least Q2 2015.
Using our (temasys) commercial plugin you will soon be able to do that in IE and Safari.
Those are the only options on client side I can think of. On server side you can use whatever you want to transcode, including janus, kurento, powermedia, licode/lynkia, ....
Note: using other means like Datachannel or WebSocket are ok to transfer files, but would greatly reduce the user experience as you would not have all the added recovery (and security) mechanisms included in SRTP, DTLS, and would also not have specific mistreated media enhancements that are in webRTC like jitter, buffers, netQ, ect ...
So I want to be able to send a normal video from a video file (AVI or any other) through WebRTC, can that be done? The only examples I see of WebRTC are video chats, so I feel as if its only geared towards webcam and chats.
So my question is, technically can sending normal video from a video file (not webcam) over WebRTC be done?
Try: "Pre-recorded media streaming" --- Documentation and Source Code.
This experiment uses MediaSource API to render Blobs in <video> element. This experiment has some issues need to be fixed e.g. it can't send longer WebM videos.
You can try this experiment as well.
The codecs typically used in AVI are not directly supported by WebRTC clients, but if you are writing your own standalone client then of course it could read an AVI or other video file and transcode it to VP8 video and Opus audio (or whatever other codecs you were able to negotiate), and transmit it via RTP. If you are trying to do video transcoding in JavaScript in a browser then that will be very slow.
I am new to WebRTC and trying to figure out how to create a program outside a browser which receives a WebRTC audio stream and outputs it on speakers.
Are there any WebRTC libraries for Java or C#?
That receiver will be running on a linux machine.
--
I've been thinking about using getUserMedia() to access the microphone. But then:
In what format will such a stream be transmitted?
Let's say I use WebRTC2SIP and build a Java endpoint using JSIP;
or I just use a socket and send the stream over http.
What audio format will I get on the receiver side? So far I have read WebRTC does compress the stream somehow.
I guess there are two ways for you:
build the whole WebRTC voice engine for android/iOS or Mac etc., and just use the API provide by VOE.
build standalone NS/VAD/AECM/AGC modules and using it in your project. for example, you build standalone NS module for android mobile, you use AudioRecord(java layer, android things) to record sound from MIC, and do the noise suppression process on these data(jni layer, WebRTC things), and finally playback the processed data by using AudioTrack(java layer, android things).
EDIT:
for the 2nd situation, the format is PCM raw data.
Check out the working Audio demo and code at demo.easyrtc.com
The code is all open source and can be checked out at https://github.com/priologic/easyrtc
You can look for any known issues around easyRTC at our forum at
https://groups.google.com/forum/#!forum/easyrtc
Also check out our main site at easyrtc.com