Why is WebRTC remote video source generated by URL.createObjectURL - webrtc

In this document, it uses URL.createObjectURL to set the video source. (This is the code to answer a call).
var offer = getOfferFromFriend();
navigator.getUserMedia({video: true}, function(stream) {
pc.onaddstream = e => video.src = URL.createObjectURL(e.stream);
pc.addStream(stream);
pc.setRemoteDescription(new RTCSessionDescription(offer), function() {
pc.createAnswer(function(answer) {
pc.setLocalDescription(answer, function() {
// send the answer to a server to be forwarded back to the caller (you)
}, error);
}, error);
}, error);
});
I expected video.src to be the address to retrieve the remote video. So it should be fixed and given by the other side of the connection (whoever initiated the call). But the value of URL.createObjectURL is generated on the answerer's side, and it event depends on when the function is called. How it can be used to get the remote video stream?
Edit:
The result of URL.createObjectURL looks like blob:http://some.site.com/xxxx-the-token-xxxx. With this string, how does the video component know where to load the remote stream? Is there a hashmap of {url:stream} stored somewhere? If so, how does the video component access the hashmap?
A stream object does store a token string, which you can get with stream.toURL. But it is different from the result of URL.createObjectURL. The value of URL.createObjectURL depends on time. If you call it twice in a row, you get different values.

URL.createObjectURL(stream) is a hack. Stop using it. Efforts are underway to remove it.
Use video.srcObject = stream directly instead. It is standard and well-implemented.
This assignment of a local resource should never have been a URL in the first place, and is a red herring to understanding how WebRTC works.
WebRTC is a transmission API, sending data directly from one peer to another. No content URLs are involved. The remote stream you get from onaddstream is a local object receiver side, and is the live streaming result of the transmission, ready to be played.
The documentation you read is old and outdated. Thanks for pointing it out, I'll fix it. It has other problems: you should call setRemoteDescription immediately, not wait for the receiver to share their camera, otherwise incoming candidates are missed. Instead of the code you show, do this:
pc.onaddstream = e => video.srcObject = e.stream;
function getOfferFromFriend(offer) {
return pc.setRemoteDescription(new RTCSessionDescription(offer))
.then(() => navigator.getUserMedia({video: true}))
.then(stream => {
pc.addStream(stream);
return pc.createAnswer();
})
.then(answer => pc.setLocalDescription(answer))
.then(() => {
// send the answer to a server to be forwarded back to the caller (you)
})
.catch(error);
}
It uses srcObject, avoids the deprecated callback API, and won't cause intermittent ICE failures.

Because a WebRTC connection involves several steps and what you get from such a connection is a stream. But the src property of the video tag does not accept a stream, but a URL. And this is the way to "convert" a stream to a URL.

Related

calculate packet send/receive time

i am using vue.js for webrtc client. My system has multi servers so i need to pick one and connect. I am getting server list from my webservice. Everything is ok till here, i want user connect to best server so decided to make custom ping (its called rtt, i think).
to sum up my aim is send a udp packet to server(sendTime in long), then server receives this packet and now server sends my a packet(receiveTime in long) => this servers ping is receiveTime - sendTime
So i will do this for every server and choose best server.Is it posible to send and receive udp packet or anyone got better idea?
In my opinion the simplest way would be to make a get request to every url, then do a promise race on the requests, which one returns the fastest is the server you would use.
const urls = ['https://www.google.com/', 'https://www.facebook.com/', 'https://www.twitter.com/', 'https://www.instagram.com/'];
const serverPromises = [];
urls.forEach((url) => {
serverPromises.push(
new Promise((resolve, reject) => {
fetch(url, {
mode: 'no-cors'
}).then((response) => {
if ([400, 500, 404].includes(response.status)) {
return reject(null);
}
return resolve(url);
})
})
);
});
Promise.race(serverPromises).then((url) => {
if (url) {
console.log('URL to use: ', url);
}
});
Check it out, see if it suits your needs!
To me this seems to be the easiest, and quickest way, without any calculations, plus you get the result as soon as the first request resolves, this way you don't have to wait for every request to resolve.

Watch for changes to calendar, when to make request

When watching for changes to a collection of events on a given calendar, how often do I need to make a watch request?
Where would I put my code to make a watch request? Does it only need to be done once?
My code below gets an access token and makes a post to create a watch channel, however I'm not sure where to host the code or how often I need to run it:
let { google } = require("googleapis");
let functions = require("firebase-functions");
let privatekey = require("./config.json");
let axios = require("axios");
let jwt = new google.auth.JWT(
privatekey.client_email,
null,
privatekey.private_key,
["https://www.googleapis.com/auth/calendar"]
);
const token = await jwt.authorize();
let headers = {
"Access-Control-Allow-Origin": "*",
"Content-Type": "application/json;charset=UTF-8",
Authorization: token.token_type + " " + token.access_token
};
let data = {
id: randomId,
type: "web_hook",
address: "https://rguc-calendars.firebaseapp.com/notifications",
params: {
ttl: 3600
}
};
axios
.post(
"https://www.googleapis.com/calendar/v3/calendars/thirdyear#rguc.co.uk/events/watch",
data,
{ headers }
)
.then(function(response) {
// success
})
.catch(function(error) {
// error
});
push notifications
The Google Calendar API provides push notifications that let you watch
for changes to resources. You can use this feature to improve the
performance of your application. It allows you to eliminate the extra
network and compute costs involved with polling resources to determine
if they have changed. Whenever a watched resource changes, the Google
Calendar API notifies your application.
Register the domain of your receiving URL.
For example, if you plan to use https://example.com/notifications as your receiving URL, you need to register https://example.com.
Set up your receiving URL, or "Webhook" callback receiver.
This is an HTTPS server that handles the API notification messages that are triggered when a resource changes.
Set up a notification channel for each resource endpoint you want to watch.
A channel specifies routing information for notification messages. As part of the channel setup, you identify the specific URL where you want to receive notifications. Whenever a channel's resource changes, the Google Calendar API sends a notification message as a POST request to that URL.
Once you have set up the watch google will notify you when ever there is a change you wont have to call it again.

How to read from AzureIOT only messages from one device

I have an Azure IOT solution where data from 2 devices go to the same IOT hub. From my computer I need to read the messages only from one of the devices. I implemented the ReadDeviceToCloudMessages.js in https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-node-node-getstarted
var client = EventHubClient.fromConnectionString(connectionString);
client.open()
.then(client.getPartitionIds.bind(client))
.then(function (partitionIds) {
return partitionIds.map(function (partitionId) {
return client.createReceiver('todevice', partitionId, { 'startAfterTime' : Date.now()}).then(function(receiver) {
console.log('Created partition receiver: ' + partitionId)
receiver.on('errorReceived', printError);
receiver.on('message', printMessage);
});
});
})
.catch(printError);
But I am getting all the messages in the IOThub. How do I get messages only from one device.
You can route the expected device message to build-in endpoint: events. Then you can only receive the selected device message from your above code.
Create the route:
Turn "Device messages which do not match any rules will be written to the 'Events (messages/events)' endpoint." to off and make sure the route is enabled.

WebRTC: How do I stream Client A's video to Client B?

I am looking into WebRTC but I feel like I'm not understanding the full picture. I'm looking at this demo project in particular: https://github.com/oney/RCTWebRTCDemo/blob/master/main.js
I'm having trouble understanding how I can match 2 clients so that Client A can see Client B's video stream and vice versa.
This is in the demo:
function getLocalStream(isFront, callback) {
MediaStreamTrack.getSources(sourceInfos => {
console.log(sourceInfos);
let videoSourceId;
for (const i = 0; i < sourceInfos.length; i++) {
const sourceInfo = sourceInfos[i];
if(sourceInfo.kind == "video" && sourceInfo.facing == (isFront ? "front" : "back")) {
videoSourceId = sourceInfo.id;
}
}
getUserMedia({
audio: true,
video: {
mandatory: {
minWidth: 500, // Provide your own width, height and frame rate here
minHeight: 300,
minFrameRate: 30
},
facingMode: (isFront ? "user" : "environment"),
optional: [{ sourceId: sourceInfos.id }]
}
}, function (stream) {
console.log('dddd', stream);
callback(stream);
}, logError);
});
}
and then it's used like this:
socket.on('connect', function(data) {
console.log('connect');
getLocalStream(true, function(stream) {
localStream = stream;
container.setState({selfViewSrc: stream.toURL()});
container.setState({status: 'ready', info: 'Please enter or create room ID'});
});
});
Questions:
What exactly is MediaStreamTrack.getSources doing? Is this because devices can have multiple video sources (e.g. 3 webcams)?
Doesn't getUserMedia just turn on the client's camera? In the code above isn't the client just viewing a video of himself?
I'd like to know how I can pass client A's URL of some sort to client B so that client B streams the video coming from client A. How do I do this? I'm imagining something like this:
Client A enters, joins room "abc123". Waits for another client to join
Client B enters, also joins room "abc123".
Client A is signaled that Client B has entered the room, so he makes a connection with Client B
Client A and Client B start streaming from their webcam. Client A can see Client B, and Client B can see Client A.
How would I do it using the WebRTC library (you can just assume that the backend server for room matching is created)
The process you are looking for is called JSEP (JavaScript Session Establishment Protocol) and it can be divided in the 3 steps I describe below. These steps start once both clients are in the room and can comunicate through WebSockets, I will use ws as an imaginary WebSocket API for communication between the client and the server and other clients:
1. Invite
During this step, one desinged caller creates and offer and sends it through the server to the other client (callee):
// This is only in Chrome
var pc = new webkitRTCPeerConnection({iceServers:[{url:"stun:stun.l.google.com:19302"}]}, {optional: [{RtpDataChannels: true}]});
// Someone must be chosen to be the caller
// (it can be either latest person who joins the room or the people in it)
ws.on('joined', function() {
var offer = pc.createOffer(function (offer) {
pc.setLocalDescription(offer);
ws.send('offer', offer);
});
});
// The callee receives offer and returns an answer
ws.on('offer', function (offer) {
pc.setRemoteDescription(new RTCSessionDescription(offer));
pc.createAnswer(function(answer) {
pc.setLocalDescription(answer);
ws.send('answer', answer);
}, err => console.log('error'), {});
});
// The caller receives the answer
ws.on('answer', function (answer) {
pc.setRemoteDescription(new RTCSessionDescription(answer));
});
Now both sides are have exchanged SDP packets and are ready to connect to each other.
2. Negotiation (ICE)
ICE candidates are created by each side to find a way to connect to each other, they are pretty much IP addresses where they can be found: localhost, local area network address (192.168.x.x) and external public IP Address (ISP). They are generated automatically by the PC object.
// Both processing them on each end:
ws.on('ICE', candidate => pc.addIceCandidate(new RTCIceCandidate(data)));
// Both sending them:
pc.onicecandidate = candidate => ws.send('ICE', candidate);
After the ICE negotiation, the conexion gets estabished unless you try to connect clients through firewalls on both sides of the connection, p2p communications are NAT traversal but won't work on some scenarios.
3. Data streaming
// Once the connection is established we can start to transfer video,
// audio or data
navigator.getUserMedia(function (stream) {
pc.addStream(stream);
}, err => console.log('Error getting User Media'));
It is a good option to have the stream before making the call and adding it at earlier steps, before creating the offer for the caller and right after receiving the call for the callee, so you don't have to deal with renegotiations. This was a pain a few years ago but it may be better implemented now in WebRTC.
Feel free to check my WebRTC project in GitHub where I create p2p connections in rooms for many participants, it is in GitHub and has a live demo.
MediaStreamTrack.getSources is used to get video devices connected. It seems to be deprecated now. See this stack-overflow question and documentation. Also refer MediaStreamTrack.getSources demo and code.
Yes. getUserMedia is just turning on camera. You can see the demo and code here.
Please refer to this peer connection sample & code here to stream audio and video between users.
Also look at this on Real time communication with WebRTC.

How to use kurento-media-server for audio only stream?

I want to have only audio stream communication between peers , I changed the parts of kurento.utils.js to get only audio stream via getusermedia
but it's not working
I used this example node-hello-world example
WebRtcPeer.prototype.userMediaConstraints = {
audio : true,
video : {
mandatory : {
maxWidth : 640,
maxFrameRate : 15,
minFrameRate : 15
}
}
};
to
WebRtcPeer.prototype.userMediaConstraints = {
audio : true,
video : false
};
is it possible use kurento service for only audio stream?
This is indeed possible with Kurento. There are two ways of doing this, depending on the desired scope of the modification:
Per webrtc endpoint: when you process the SDP offer sent by the client, you get an SDP answer from KMS that you have to send back. After invoking the processOffer method call, you can tamper the SDP to remove all video parts. That way, your client will send back only audio.
Globally: You can edit /etc/kurento/sdp_pattern.txt file removing all video related parts, this will force SdpEndpoints (parent class of WebrtcEndpoint) to only use audio.
EDIT 1
The file sdp_pattern.txt is deprecated in KMS 6.1.0, so method 2 shouldn't be used.
EDIT 2
There was an issue with the kurento-utils library, and the client was not correctly setting the OfferToReceiveAudio. It was fixed some time ago, and you shouldn't need tampering the SDPs now.
git origin: https://github.com/Kurento/kurento-tutorial-js.git
git branch: 6.6.0
My solution is only changing var offerVideo = true; to var offerVideo = false; in generateOffer function of kurento-utils.js file.
My approach is to modify the options that you pass to the WebRtcPeer.
var options = {
onicecandidate: onIceCandidate,
iceServers: iceServers,
mediaConstraints: {
audio:true,
video:false
}
}
Besides, in the kurento-utils.js, the mediaContraints is overidden by this line:
constraints.unshift(MEDIA_CONSTRAINTS);
So comment it.