I want to have only audio stream communication between peers , I changed the parts of kurento.utils.js to get only audio stream via getusermedia
but it's not working
I used this example node-hello-world example
WebRtcPeer.prototype.userMediaConstraints = {
audio : true,
video : {
mandatory : {
maxWidth : 640,
maxFrameRate : 15,
minFrameRate : 15
}
}
};
to
WebRtcPeer.prototype.userMediaConstraints = {
audio : true,
video : false
};
is it possible use kurento service for only audio stream?
This is indeed possible with Kurento. There are two ways of doing this, depending on the desired scope of the modification:
Per webrtc endpoint: when you process the SDP offer sent by the client, you get an SDP answer from KMS that you have to send back. After invoking the processOffer method call, you can tamper the SDP to remove all video parts. That way, your client will send back only audio.
Globally: You can edit /etc/kurento/sdp_pattern.txt file removing all video related parts, this will force SdpEndpoints (parent class of WebrtcEndpoint) to only use audio.
EDIT 1
The file sdp_pattern.txt is deprecated in KMS 6.1.0, so method 2 shouldn't be used.
EDIT 2
There was an issue with the kurento-utils library, and the client was not correctly setting the OfferToReceiveAudio. It was fixed some time ago, and you shouldn't need tampering the SDPs now.
git origin: https://github.com/Kurento/kurento-tutorial-js.git
git branch: 6.6.0
My solution is only changing var offerVideo = true; to var offerVideo = false; in generateOffer function of kurento-utils.js file.
My approach is to modify the options that you pass to the WebRtcPeer.
var options = {
onicecandidate: onIceCandidate,
iceServers: iceServers,
mediaConstraints: {
audio:true,
video:false
}
}
Besides, in the kurento-utils.js, the mediaContraints is overidden by this line:
constraints.unshift(MEDIA_CONSTRAINTS);
So comment it.
Related
Hi I have been successfully recording an Agora audio call, where one person speaks in a broadcast role, and during the call mixes in a number of audio files.
All the audio was being recorded until we upgraded to flutter 2 and associated upgraded packages.
Now all that is recorded is the broadcaster voice, and no mixed in audio.
The broadcaster and audience members can all hear the mixed in audio within the call without issue.
The code (Flutter) is similar to this:
Mix in Audio into a valid RTC session, with default settings
final playing = await session.playAudioFile(path, (){
state = MessagePlayerState.STOPPED;
if (!disposing) {
whenFinished();
}
});
The recording options are as follows (My UID is a hardcoded string, that is not the same as any participant UIDs)
http.Response response = await http.post(
Uri.https(AGORA_REST_URL, '$AGORA_REST_API_VERSION/$appId/cloud_recording/resourceid/$resourceId/mode/mix/start'),
headers: <String, String>{
HttpHeaders.authorizationHeader: 'Basic $basicAuth',
HttpHeaders.contentTypeHeader: 'application/json; charset=UTF-8',
},
body: jsonEncode(<String, dynamic>{
'cname': channelName,
'uid': uid,
'clientRequest': {
'recordingConfig':{
'channelType':0,
'streamTypes':2, // TODO: Should be a streamTypes of 0 (audio only), but get failures.
'audioProfile':1,
'videoStreamType':0,
'maxIdleTime':120,
'transcodingConfig':{
'width':360,
'height':640,
'fps':30,
'bitrate':600,
'maxResolutionUid':'1',
'mixedVideoLayout':1
},
'recordingFileConfig':{
'avFileType': ['hls','mp4']
}
},
'storageConfig':{
'vendor':1,
'region':3,
'bucket':AWS_RECORDING_BUCKET, // TODO: Env Var
'accessKey':AWS_BUCKET_ACCESS_KEY,
'secretKey':AWS_BUCKET_SECRET_KEY,
}
},
}),
);
The m3u8 and ts files are present in the S3 bucket.
Adjusting the metadata tags in S3 results in a file that plays fine in Safari, but no mixed in audio is heard.
Converting the file to aac with ffmpeg shows this error
[hls # 0x7fd6cc808200] Opening '2838cfc6254e9fec2e3088976f39d7ce_bip_20210618014151427.ts' for reading
[mpegts # 0x7fd6cc00a600] Packet corrupt (stream = 0, dts = 1437390).
size= 480kB time=00:00:30.69 bitrate= 128.1kbits/s speed=1.49e+03x
video:0kB audio:470kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.093976%
And the result is the same as from the S3 bucket.
Any help or hints appreciated.
This can be closed/ignored. Turns out we had an edge condition that did not show when the app was used normally, but if, for instance, you wanted a very stage managed recording to show off to others it broke.
User started with DataChannel only.
AudioChannel is added through renegotiation later.
var mLocalAudio;
navigator.getUserMedia({ video: false, audio: true },
function (myStream) {
mLocalAudio = myStream;
mConn.addStream(myStream);
}, function (e) { });
On the remote peer, ontrack will be triggered and we add the stream on to an <audio> element.
But since this is a many to many connection, there will have multiple peers trying to swith on / off their audio channel from time to time.
My problem is, how can I identify which audio track is belongs to which user?
Try to archive:
I try to use the HTML5 EventSourcing API https://developer.mozilla.org/de/docs/Web/API/EventSource to push events to my client application (javascript).
working example code with plain node http:
With a plain example node implementation it works perfectly and as expected. Example code: https://www.html5rocks.com/en/tutorials/eventsource/basics/
Problem:
When i try to integrate EventSourcing (or SSE) into my API endpoint which is based on hapi (currently using latest - 18.1.0) it does not work.
My route handler code mixed with some code i found:
const Stream = require('stream');
class ResponseStream extends Stream.PassThrough {
setCompressor (compressor) {
this._compressor = compressor;
}
}
const stream = new ResponseStream();
let data = 0;
setInterval(() => {
data++;
stream.write('event: message\n');
stream.write('data:' + data + '\n\n');
console.log('write data...', data);
// stream.end();
}, 1000);
return h
.response(stream)
.type('text/event-stream')
.header('Connection', 'keep-alive')
.header('Cache-Control', 'no-cache')
Findings:
I already searched and it seems since hapi 17.x there they exposed the flush method for the compressor < https://github.com/hapijs/hapi/issues/3658 >, section features.
But it still does not working.
They only way it sends a message is to uncomment the stream.end() line after sending the data. The problem obviously is that i cant send further data if i close the stream :/.
If i kill the server (with stream.end() line commented) the data gets transmitted to the client in a "single transmission". I think the problem is is still somewhere with the gzip buffering even when flushing the stream.
There are some code examples in the hapi github but i got none working with hapi 17 or 18 (all exmaples where hapi =< 16) :/
Someone know how to solve the problem or has a working EventSource example with latest hapi? I would kindly appreciate any help or suggestions.
Edit - Solution
The solution from the post below does work but i had also an nginx reverse proxy in front of my api endpoint it seems the main problem was not my code it was the nginx which had also buffered the eventsource messages.
To avoid this sort of problem add in your hapi: X-Accel-Buffering: no; and it works flawless
Well I just tested with Hapi 18.1.0 and managed to create a working example.
This is my handler code:
handler: async (request, h) => {
class ResponseStream extends Stream.PassThrough {
setCompressor(compressor) {
this._compressor = compressor;
}
}
const stream = new ResponseStream();
let data = 0;
setInterval(() => {
data++;
stream.write('event: message\n');
stream.write('data:' + data + '\n\n');
console.log('write data...', data);
stream._compressor.flush();
}, 1000);
return h.response(stream)
.type('text/event-stream')
}
and this is client code just to test
var evtSource = new EventSource("http://localhost/");
evtSource.onmessage = function(e) {
console.log("Data", + e.data);
};
evtSource.onerror = function(e) {
console.log("EventSource failed.", e);
};
These are the resources that where I found my way to working example
https://github.com/hapijs/hapi/blob/70f777bd2fbe6e2462847f05ee10a7206571e280/test/transmit.js#L1816
https://github.com/hapijs/hapi/issues/3599#issuecomment-485190525
In this document, it uses URL.createObjectURL to set the video source. (This is the code to answer a call).
var offer = getOfferFromFriend();
navigator.getUserMedia({video: true}, function(stream) {
pc.onaddstream = e => video.src = URL.createObjectURL(e.stream);
pc.addStream(stream);
pc.setRemoteDescription(new RTCSessionDescription(offer), function() {
pc.createAnswer(function(answer) {
pc.setLocalDescription(answer, function() {
// send the answer to a server to be forwarded back to the caller (you)
}, error);
}, error);
}, error);
});
I expected video.src to be the address to retrieve the remote video. So it should be fixed and given by the other side of the connection (whoever initiated the call). But the value of URL.createObjectURL is generated on the answerer's side, and it event depends on when the function is called. How it can be used to get the remote video stream?
Edit:
The result of URL.createObjectURL looks like blob:http://some.site.com/xxxx-the-token-xxxx. With this string, how does the video component know where to load the remote stream? Is there a hashmap of {url:stream} stored somewhere? If so, how does the video component access the hashmap?
A stream object does store a token string, which you can get with stream.toURL. But it is different from the result of URL.createObjectURL. The value of URL.createObjectURL depends on time. If you call it twice in a row, you get different values.
URL.createObjectURL(stream) is a hack. Stop using it. Efforts are underway to remove it.
Use video.srcObject = stream directly instead. It is standard and well-implemented.
This assignment of a local resource should never have been a URL in the first place, and is a red herring to understanding how WebRTC works.
WebRTC is a transmission API, sending data directly from one peer to another. No content URLs are involved. The remote stream you get from onaddstream is a local object receiver side, and is the live streaming result of the transmission, ready to be played.
The documentation you read is old and outdated. Thanks for pointing it out, I'll fix it. It has other problems: you should call setRemoteDescription immediately, not wait for the receiver to share their camera, otherwise incoming candidates are missed. Instead of the code you show, do this:
pc.onaddstream = e => video.srcObject = e.stream;
function getOfferFromFriend(offer) {
return pc.setRemoteDescription(new RTCSessionDescription(offer))
.then(() => navigator.getUserMedia({video: true}))
.then(stream => {
pc.addStream(stream);
return pc.createAnswer();
})
.then(answer => pc.setLocalDescription(answer))
.then(() => {
// send the answer to a server to be forwarded back to the caller (you)
})
.catch(error);
}
It uses srcObject, avoids the deprecated callback API, and won't cause intermittent ICE failures.
Because a WebRTC connection involves several steps and what you get from such a connection is a stream. But the src property of the video tag does not accept a stream, but a URL. And this is the way to "convert" a stream to a URL.
I am looking into WebRTC but I feel like I'm not understanding the full picture. I'm looking at this demo project in particular: https://github.com/oney/RCTWebRTCDemo/blob/master/main.js
I'm having trouble understanding how I can match 2 clients so that Client A can see Client B's video stream and vice versa.
This is in the demo:
function getLocalStream(isFront, callback) {
MediaStreamTrack.getSources(sourceInfos => {
console.log(sourceInfos);
let videoSourceId;
for (const i = 0; i < sourceInfos.length; i++) {
const sourceInfo = sourceInfos[i];
if(sourceInfo.kind == "video" && sourceInfo.facing == (isFront ? "front" : "back")) {
videoSourceId = sourceInfo.id;
}
}
getUserMedia({
audio: true,
video: {
mandatory: {
minWidth: 500, // Provide your own width, height and frame rate here
minHeight: 300,
minFrameRate: 30
},
facingMode: (isFront ? "user" : "environment"),
optional: [{ sourceId: sourceInfos.id }]
}
}, function (stream) {
console.log('dddd', stream);
callback(stream);
}, logError);
});
}
and then it's used like this:
socket.on('connect', function(data) {
console.log('connect');
getLocalStream(true, function(stream) {
localStream = stream;
container.setState({selfViewSrc: stream.toURL()});
container.setState({status: 'ready', info: 'Please enter or create room ID'});
});
});
Questions:
What exactly is MediaStreamTrack.getSources doing? Is this because devices can have multiple video sources (e.g. 3 webcams)?
Doesn't getUserMedia just turn on the client's camera? In the code above isn't the client just viewing a video of himself?
I'd like to know how I can pass client A's URL of some sort to client B so that client B streams the video coming from client A. How do I do this? I'm imagining something like this:
Client A enters, joins room "abc123". Waits for another client to join
Client B enters, also joins room "abc123".
Client A is signaled that Client B has entered the room, so he makes a connection with Client B
Client A and Client B start streaming from their webcam. Client A can see Client B, and Client B can see Client A.
How would I do it using the WebRTC library (you can just assume that the backend server for room matching is created)
The process you are looking for is called JSEP (JavaScript Session Establishment Protocol) and it can be divided in the 3 steps I describe below. These steps start once both clients are in the room and can comunicate through WebSockets, I will use ws as an imaginary WebSocket API for communication between the client and the server and other clients:
1. Invite
During this step, one desinged caller creates and offer and sends it through the server to the other client (callee):
// This is only in Chrome
var pc = new webkitRTCPeerConnection({iceServers:[{url:"stun:stun.l.google.com:19302"}]}, {optional: [{RtpDataChannels: true}]});
// Someone must be chosen to be the caller
// (it can be either latest person who joins the room or the people in it)
ws.on('joined', function() {
var offer = pc.createOffer(function (offer) {
pc.setLocalDescription(offer);
ws.send('offer', offer);
});
});
// The callee receives offer and returns an answer
ws.on('offer', function (offer) {
pc.setRemoteDescription(new RTCSessionDescription(offer));
pc.createAnswer(function(answer) {
pc.setLocalDescription(answer);
ws.send('answer', answer);
}, err => console.log('error'), {});
});
// The caller receives the answer
ws.on('answer', function (answer) {
pc.setRemoteDescription(new RTCSessionDescription(answer));
});
Now both sides are have exchanged SDP packets and are ready to connect to each other.
2. Negotiation (ICE)
ICE candidates are created by each side to find a way to connect to each other, they are pretty much IP addresses where they can be found: localhost, local area network address (192.168.x.x) and external public IP Address (ISP). They are generated automatically by the PC object.
// Both processing them on each end:
ws.on('ICE', candidate => pc.addIceCandidate(new RTCIceCandidate(data)));
// Both sending them:
pc.onicecandidate = candidate => ws.send('ICE', candidate);
After the ICE negotiation, the conexion gets estabished unless you try to connect clients through firewalls on both sides of the connection, p2p communications are NAT traversal but won't work on some scenarios.
3. Data streaming
// Once the connection is established we can start to transfer video,
// audio or data
navigator.getUserMedia(function (stream) {
pc.addStream(stream);
}, err => console.log('Error getting User Media'));
It is a good option to have the stream before making the call and adding it at earlier steps, before creating the offer for the caller and right after receiving the call for the callee, so you don't have to deal with renegotiations. This was a pain a few years ago but it may be better implemented now in WebRTC.
Feel free to check my WebRTC project in GitHub where I create p2p connections in rooms for many participants, it is in GitHub and has a live demo.
MediaStreamTrack.getSources is used to get video devices connected. It seems to be deprecated now. See this stack-overflow question and documentation. Also refer MediaStreamTrack.getSources demo and code.
Yes. getUserMedia is just turning on camera. You can see the demo and code here.
Please refer to this peer connection sample & code here to stream audio and video between users.
Also look at this on Real time communication with WebRTC.