How looks WebRTC peer negotiation workflow? - webrtc

I need to develop a custom WebRTC peer (I need to establish audio or/and data connection between web-browser and non-browser). I however, struggle to find a proper, clear description of the handshake phase.
Answers to questions such as How to create data channel in WebRTC peer connection? are not entirely helpful, as they are not too detailed. Specifically, they say nothing about SDP contents.
Can anyone explain this or recommend any good documentation?

Here is a page with some graphs showing how the signaling process works. Basically, you set some client side stuff first:
PeerConnectionFactory; to generate PeerConnections,
PeerConnection; one for every connection to another peer you want (usually 1),
MediaStream; to hook up the audio and video from your client device.
Then you generate an SDP offer
peerConnection.createOffer();
on the caller side and send it to the callee. The callee sets this offer
peerConnection.setRemoteDescription(insert-the-offer-here);
and generates an SDP answer
peerConnection.createAnswer();
and sends it back to the caller. The caller receives this answer and sets it.
peerConnection.setRemoteDescription(insert-the-answer-here);
Both the caller and callee get a call to
onAddStream() {...} //needs to be implemented in your code
The callee when the caller's offer is set and the caller when the callee's answer is set. This callback signals the beginning of the connection.
You can also use ICE (STUN/TURN) to avoid firewall and NAT issues, but this is optional. Although in production code, you probably want to implement it anyway.
Note: Webrtc documentation is scarce and subject to change, take everything you read about webrtc (at least anything written as of now) with a grain of salt...

Related

Should I send a WebRTC answer before my side's localMedia tracks were added?

I'm building a video calling app using WebRTC which allows one peer to call another by selecting someone in the lobby. When peer A sends a call request, the other peer B can accept. At this point, WebRTC signaling starts:
Both peers get their local media using MediaDevices.getUserMedia()
Both peers create an RTCPeerConnection and attach event listeners
Both peers calls RTCPeerConnection.addTrack() to add their local media
One peer A (the impolite user) creates an offer, calls RTCPeerConnection.setLocalDescription() to set that offer as the local description, and sends it to the WebSocket server, which forwards it to the other peer B.
The other peer B receives this offer and adds calls RTCPeerConnection.setRemoteDescription() to record it as the remote description
The other peer B then creates an answer and transmits it again to the first peer A.
(Steps based on https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Connectivity)
This flow is almost working well. In 1 out of 10 calls, I receive no video/audio from one of the peers (while both peers have working local video). In such a case, I have noticed that the answer SDP contains a=recvonly while this should be a=sendrecv under normal circumstances.
I have further determined that by the time the other peer receives the offer and needs to reply with an answer, the localMedia of this side has sometimes not yet been added, because MediaDevices.getUserMedia can take a while to complete. I have also confirmed this order of operations by logging and observing that the offer sometimes arrives before local tracks were added.
I'm assuming that I shouldn't send an answer before the local media has been added?
I'm thinking of two ways to fix this, but I am not sure which option is best, if any:
Create the RTCPeerConnection only after MediaDevices.getUserMedia() completes. In the meantime, when receiving an offer, there is no peer connection yet, so we save offers in a buffer to process them later once the RTCPeerConnection is created.
When receiving an offer, and there are no localMedia tracks yet, hold off on creating the answer until the localMedia tracks have been added.
I am having difficulties to decide which solution (or another) matches best with the "Perfect Negotiation" pattern.
Thanks in advance!
Yes, it is good to add the stream before creating an offer if you do it 'statically', but the best way to do it is to do it in the onnegotiationneeded event because the addtrack event triggers an onnegotiationneeded event. So you should add the stream and then use the createoffer inside onnegotiationneeded. As far as the answer you can do it before with no issues, but remember that a well-established connection will let you add/remove tracks with no problems (even after the SDP has been set). You didn't post any code but remember that you also MUST exchange ice candidates.
The last piece of advice, remember that all of the above IS asynchronous! so you should use promises, and await until the description is set, only THEN create an offer/answer.
Hopefully, this will help

In peer connection negotiation phase, is there ever a counter offer?

So far, everything I've read on webrtc peer connections says that an "offer" is sent, and it is responded to with an "answer". Then the connection starts and all is well.
In my understanding, the offer is like "Hey, let's use this codec and encryption". Given that the answer always leads to a connection, it seems the answer is always "okay, let's use that!". Can there be a counter offer like "No, let's use this codec instead!". Who ultimately decides which settings are used?
The offer contains a list of one side's acceptable codecs (prioritzed).
The answer contains the subset of those codecs, listing only the ones that both sides can do - possibly in a different order.
So: No the answer shouldn't contain a codec that wasn't in the offer.
But... Once Offer/Answer has happened, either side can send a second offer (this is typically used to add video to an existing audio-only session) and receive a new answer.
This means you could send an answer with no codecs and then send an second offer with a different set of codecs, but there is no reason to expect that the other side will change it's mind (unless there was some resource exhaustion)

WebRTC: removeStream and then re- addStream after negotiation: Safe?

After a WebRTC session has been established, I would like to stop sending the stream to the peer at some point and then resume sending later.
If I call removeStream, it does indeed stop sending data. If I then call addStream (with the previously removed stream), it resumes. Great!
However, is this safe? Or do I need to re-negotiate/exchange the SDP after the removal and/or after the re-add? I've seen it mentioned in several places that the SDP needs to be re-negotiated after such changes, but I'm wondering if it is ok in this simple case where the same stream is being removed and re-added?
PS: Just in case anyone wants to suggest it: I don't want to change the enabled state of the track since I still need the local stream playing, even while it is not being sent to the peer.
It will only work in Chrome, and is non-spec, so it's neither web compatible nor future-proof.
The spec has pivoted from streams to tracks, sporting addTrack and removeTrack instead of addStream and removeStream. As a result the latter isn't even implemented in Firefox.
Unfortunately, because Chrome hasn't caught up, this means renegotiation currently works differently in different browsers. It is possible to do however with some effort.
The new model has a cleaner separation between streams and what's sent in RTCPeerConnection.
Instead of renegotiating, setting track.enabled = false is a good idea. You can still play a local view of your video by cloning the track.

WebRTC: Why "CreateAnswer can't be called before SetRemoteDescription"?

Browser: Chrome
I am trying to debug a webRTC application which works fine on three out of four fronts! I cannot get video from the receiver to the caller. I can get video and audio from the caller to the receiver and audio from the receiver to the caller. The problem is that the receiver does not fire a video (sdpMid="video") ICE candidate. While desperately trying to solve this problem, I tried to use pc.CreateAnswer before setting pc.remoteDescription and it gave the error quoted in the title.
My question is to understand the reason behind this. An answer SDP would just be the SDP based upon the getUserMedia settings/constraints. So, why do we have to wait for setting remoteDescription. I thought that a createAnswer would start firing the gathering of ICE candidates and this can be done earlier without waiting to set remoteDescription. That is not the case. Why?
Offers and answers aren't independent, they're part of an inherently asymmetric exchange.
An answer is a direct response to a specific offer (hence the name "answer"). Therefore the peer cannot answer before it has an offer, which you set with setRemoteDescription.
An offer contains specific limitations, or envelope (like m-lines), that an answer has to abide by/answer to/stay within. Another way to say it is that the answer is an iteration of the offer.
For instance, an offer created with offer options offerToReceiveVideo: false can only be answered with recvonly for video (meaning receive video from offerer to answerer only), never sendrecv.

Twisted - succes (or failure) callback for LineReceiver sendLine

I'm still trying to master Twisted while in the midst of finishing an application that uses it.
My question is:
My application uses LineReceiver.sendLine to send messages from a Twisted TCP server.
I would like to know if the sendLine succeeded.
I gather that I need to somehow add a success (and error?) callback to sendLine but I don't know how to do this.
Thanks for any pointers / examples
You need to define "succeeded" in order to come up with an answer to this.
All sendLine does immediately (probably) is add some bytes to a send buffer. In some sense, as long as it doesn't raise an exception (eg, MemoryError because your line is too long or TypeError because your line was the number 3 instead of an actual line) it has succeeded.
That's not a very useful kind of success, though. Unfortunately, the useful kind of success is more like "the bytes were added to the send buffer, the send buffer was flushed to the socket, the peer received the bytes, and the receiving application acted on the data in a persistent way".
Nothing in LineReceiver can tell you that all those things happened. The standard solution is to add some kind of acknowledgement to your protocol: when the receiving application has acted on the data, it sends back some bytes that tell the original sender the message has been handled.
You won't get LineReceiver.sendLine to help you much here because all it really knows how to do is send some bytes in a particular format. You need a more complex protocol to handle acknowledgements.
Fortunately, Twisted comes with a few. twisted.protocols.amp is one: it offers remote method calls (complete with responses) as a basic feature. I find that AMP is suitable for a wide range of applications so it's often safe to recommend for new development. It largely supersedes the older twisted.spread (aka "PB") which also provides both remote method calls and remote object references (and is therefore more complex - in my experience, more complex than most applications need). There are also some options that are a bit more standard: for example, Twisted Web includes an HTTP implementation (HTTP, as you may know, is good at request/response style interaction).