I am capturing my screen in real-time and encoding them using the vp9 codec (using JNI). encoded frames are I-frame or P-frame. then I divide them into chunks (sub-frames) and send them to the network. But at receiving end there has been some natural packet loss and even a single miss of sub-frames causing the inability to reconstruct corresponding I/P - frames. I tried to simulate (randomly throwing out some sub-frames) the same thing locally and the same things happened. Doesn't VP9 codec has some built-in packet loss handling? If so, how to enable them and how can it perform well to a certain percentage?
And if there is no built-in packet loss handling Do I have to implement FIR or FEC manually? and where to follow?
Thanks in advance.
Common way to send video stream is RTP protocol based on UDP, among other libs WebRTC also uses this transport under hood. Each encoded frame before sending is packetized, i.e. splitted to one or several RTP packets. In this context term "packet loss" means RTP packet loss. These losses are handled by sender peer using RTCP Receiver Reports from the other peer: the sender can retransmit lost packets. So, such reconstruction is not related to VP9 or any other specific codec.
As vp9 is a entropy coding, even a single packet missing causes inability to reconstruct the I/P frame. Even inability to reconstruct I frame cause inability to construct all subsequent & dependent P frame's. As I am using raw vp9 I have to implement any kind of retransmission or redundancy.
There is a concept of error_resilient packet or golden frame which can be called as budget version of I-frame which I need to send at certain interval from sender so that the I frame & subsequent P
frame's will have some resiliency (I tried it and failed generating golden frame from encoder by enabling the parameter, maybe I will have to generate it myself).
Related
I have been trying to use WebRTC Data Channel for a game, however, I am unable to consistently send live player data without hitting the queue size limit (8KB) after 50-70 secs of playing.
Sine the data is required to be real-time, I have no use for data that comes out of order. I have initialized the data channel with the following attributes:
negotiated: true,
id: id,
ordered: true,
maxRetransmits: 0,
maxPacketLifetime: 66
The MDN Docs said that the buffer cannot be altered in any way.
Is there anyway I can consistently send data without exceeding the buffer space? I don't mind purging the buffer space as it only contains data that has been clogged up over time.
NOTE: The data is transmitting until the buffer size exceeds the 8KB space.
EDIT: I forgot to add that this issue is only occurring when the two sides are on different networks. When both are within the same LAN, there is no buffering (since higher bandwidth, I presume). I tried to add multiple Data Channels (8 in parallel). However, this only increased the time before the failure occurred again. All 8 buffers were full. I also tried creating a new channel each time the buffer was close to being full and switched to the new DC while closing the previous one that was full, but I found out the hard way (reading Note in MDN Docs) that the buffer space is not released immediately, rather tries to transmit all data in the buffer taking away precious bandwidth.
Thanks in advance.
The maxRetransmits value is ignored if the maxPacketLifetime value is set; thus, you've configured your channel to resend packets for up to 66ms. For your application, it is probably better to use a pure unreliable channel by setting maxPacketLifetime to 0.
As Sean said, there is no way to flush the queue. What you can do is to drop packets before sending them if the channel is congested:
if(dc.bufferedAmount > 0)
return;
dc.send(data);
Finally, you should realise that buffering may happen in the network as well as at the sender: any router can buffer packets when it is congested, and many routers have very large buffers (this is called BufferBloat). The WebRTC stack should prevent you from buffering too much data in the network, but if WebRTC's behaviour is not aggressive enough for your needs, you will need to add explicit feedback from the sender to the receiver in order to avoid having too many packets in flight.
I don't believe you can flush the outbound buffer, you will probably need to watch the bufferedAmount and adjust what you are sending if it grows.
Maybe handle the retransmissions yourselves and discard old data if needed? WebRTC doesn't surface the SACKs from SCTP. So I think you will need to implement something yourself.
It's an interesting problem. Would love to hear the WebRTC W3C WorkGroup takes on it if exposing more info would make things easier for you.
Does Google WebRTC Native implementation has support for SFU?
Does Google WebRTC Native implementation support for integrating custom/hardware encoder/decoder?
Not without alteration.
Internally WebRTC's internal audio/video pipelines are directly tied to encoder/decoders.
PeerConnectionFactory allows you to provide a video decoder/encoder factory, so you can short circuit the logic here, and grab the encoded frames, mock up a stream, and feed them directly into it as a relay, creating a new PeerConnection and setting those streams onto it.
The audio end is more difficult. There isn't a codec factory, so you will have to short circuit the logic there probably by alteration of libwebrtc.
The final question is RTCP termination, and how to override the mechanisms for quality/bandwidth control to not create a "One goes out, they all go out." situation.
Since libwebrtc will be the SFU, it will receive RTCP feedback from its remote peer for the content it is proxying, and vice versa.
For a 1-1 situation, it needs to be able to forward the RTCP feedback to the remote peer.
For multipoint, it needs to perform some logic to determine if one of the peers is problematic, and stop sending it video, switch off its video feed, or attempt to switch to a lower bitrate video stream. Basically it needs to act as a conduit that attempts to predict why/how packet loss is occurring, and keep as many audio/video feeds operating normally at at the highest possible quality for each peer.
How exactly to hijack the RTCP feedback mechanisms in libwebrtc, I think that again will likely require some customization/hooks into libwebrtc
I think it will be easier to try with GStreamer implementation of WebRTC. Although it is still in "Bad Plugins" it is way easier to get or provide encoded audio and video. Actually it is implemented in that in mind - to make implementation of MFU and SFU easier.
I have a simple UDP streaming protocol that takes RAW H264 video frames and sends them instantly from server side to the client side.
Using this protocol I can get near network RTT latency (no packet resending and I don't care about packet loss), so if I have 20 ms latency from server to the client I can make a video frame to be ready from encoder output to the client side (ready to be decoded) in... let's say 30 ms.
My question is:
Is WebRTC (over UDP) capable of going down to this kind of latencies?
Not taking into account encoding and decoding times, what is the
lowest latency possible I can get with WebRTC for the protocol layer?
I don't know if this kind of latencies will require my own protocol to be more deeply developed or I may go to something more generic like WebRTC for my video server development in order to instantly be supported by every web browser.
WebRTC can have the same low latency as regular SIP/RTP stacks.
WebRTC stack vendors does their best to reduce delay.
For recording and sending out there is no any delay. The stack will send the packets immediately once received from the recorder device and compressed with the selected codec. Some codec's (and some codec settings) might introduce some delay here to enable some features such as FEC.
Regarding the receiver side:
In optimal circumstances the stack should not delay the playback of the packets, so they can be display as soon as they arrive.
However in sub-optimal circumstances (with network delays or packet loss) the stack will introduce a jitter buffer. The lower is the network quality, the higher will be the jitter buffer length.
So, to achieve the lowest delay, you might have to do the followings:
choose a codec with the smallest processing time
remove FEC and disable any other settings which might cause additional delays
remove the jitter buffer (most WebRTC stacks doesn't have a setting for this so you might have to modify the code yourself, but it is an easy modification, because you just need to deactivate a part of the code)
WebRTC uses RTP as the underlying media transport which has only a small additional header at the beginning of the payload compared to plain UDP. This means it should be on par with what you achieve with plain UDP. RTP is heavily used in latency critical environments like real time audio and video (its the media transport in SIP, H.323, XMPP) and thus you can expect the latency to be sufficient for this purpose.
I'm a newbie with RRTCC and I really dont understand how RRTCC adjusts incoming media bitrate.
I tried to read Internet Draft of RRTCC but cannot figure it out: https://datatracker.ietf.org/doc/html/draft-alvestrand-rmcat-congestion-03
It calculates the estimated bandwidth based on delay and loss, so what's next?
How it can directly control the sending rate to control congestion?
The RTP receiver doesn't directly control the sending rate. It just sends status information to the RTP sender, which is the only one capable of controlling its own sending rate.
draft-alvestrand-rmcat-congestion has been replaced by draft-ietf-rmcat-gcc so I recommend that you use the later one.
In section 2. System model it explains that the RTP receiver should also act as a RTCP sender, in order to send REMB messages to the RTP sender. These messages should be sent by the RTP receiver as soon as it detects any congestion, and then keep sending it at a rate of once per second even if no congestion is happening. Then the RTP sender uses the information in the received message to decide if the transmission bitrate can be raised or should be immediately lowered.
I am one of the authors of the draft. If you are interested in understanding the design criteria you can also take a look at this paper:
G. Carlucci, L. De Cicco, S. Holmer, and S. Mascolo
Analysis and Design of the Google Congestion Control for Web Real-time Communication (WebRTC)
Proc. ACM Mmsys 2016, Klagenfurt, Austria, May 2016
which is available for download at http://c3lab.poliba.it/images/6/65/Gcc-analysis.pdf.
SRTCP tracks the number of sent and lost bytes and packets, last received sequence number, inter-arrival jitter for each SRTP packet, and other SRTP statistics.
Does mentioned browsers do something with SRTCP reports when dealing with audio stream, for example adjust bitrate on the fly if network conditions are changed ?
Given that Chrome does adjust bitrate and resolution of VP8 on the fly in a connection, I would assume that OPUS configurations are changed in the connection as well.
You can see the feed back on the sending audio in this image. The bitrate obviously drops slightly when using opus. However, I would imagine that video bitrate would be the first changed in a video call as changing it would have the greater effect.
Obviously, one cannot change the bitrate on a codec that only supports constant bitrates.
All the other stats are a combination of what the RTCP reports give(packetsLost, Rtt, bits sent, etc.) and googles monitoring of the inputs/outputs(audio level, echo cancellation, etc.).
NOTE: this is taken from a session created by AppRtc in chrome.