I have been using SimpleWebRTC lib for my project.
How to change dynamic remote video resolution during a call (like google hangout when resizing browser)
hangout browser resizing will change remote video resolution size (.videoWidth .videoHeight)
Is this associated with webrtc plan b?
I would like to know how it is implemented for many peer connection.
Tell the sending end (say via DataChannels) to change resolution to NxM. At the sending end, until the new APIs are available to change a getUserMedia/MediaStream capture size on the fly, you can request a second camera/mic stream and replace the existing streams with them. (Note: this will cause onnegotiationneeded i.e. renegotiation, and the far side would see a new output stream.)
Smoother (but only in Firefox thus far -- in the standardization process) would be to use RTPSender.replaceTrack() to change the video track without touching audio or renegotiating.
Another option that will exist (though doesn't yet in either browser) is to use RTPSender.width/height (or whatever syntax gets agreed) to scale outgoing video before encoding.
Plan B for multistream/BUNDLE (which Chrome implements) was not adopted; Firefox has now (in Fx38 which goes out in a few days) implemented the Unified Plan; expect to see a blog post soon from someone on how to force the two to work together (until Chrome gets to implementing Unified Plan)
Related
I am writing a PWA at the moment that uses the camera to do some light computer vision on successive frames from a mobile device camera. When calling takePhoto() a second (or sometimes 3rd time) I get the following error: Uncaught (in promise) DOMException: The associated Track is in an invalid state.
It is noted in the spec that when calling takePhoto()
devices MAY temporarily stop streaming data, reconfigure themselves with the appropriate photo settings, take the photo, and then resume streaming." It is also noted that "If the operation cannot be completed for any reason (for example, upon invocation of multiple takePhoto() method calls in rapid succession), then reject p with a new DOMException whose name is UnknownError, and abort these steps.
Why? Is there a way around this with grabFrame()? Will it change in the future? Is there a discussion within the Chrome team?
I note that I am able to successfully take the photos in quick succession using a Webcam attached to my PC, but NOT using the camera in my phone.
I know this question was already asked number of times a long time ago,but always remained unanswered.
I have webrtc client which transmits stream trough server(flashphoner) to browser. I need the way to mark specific frames by 4byte label on client side and parse this label in browser using js code.
Other theoretical ability is to add textual/qrcode watermarks and parse it on browser side using some ocr or qrparser library. The problem that i dont know how it possible to access decoded frame data on browser side. Any suggestions?
While something like this hasn't been possible in the past, the WebRTC Insertable Streams / Encoded Transform (specification) API allows this but browser support varies.
https://webrtc.github.io/samples/src/content/insertable-streams/endtoend-encryption/ shows a sample that a trivial XOR encryption and, more important for your use-case, adds a four-byte checksum.
We’ve been using the Tokbox platform for several months now with a Javascript web-client as well as an Android phone client, where sessions and connections are managed by a Python server. While integration and bring-up went well on both ends (client and server), we continue to encounter problems with the in-session audio and video experience.
Sessions are always routed and always between two participants only, with much use of a collaborative editor.
The in-session experience is like a coin toss: we never know how it’s going to go, and that’s becoming a business threat.
Web-Client: A/V Resources
The most common problem is the acquisition of audio and/or video: at the beginning of a session, one or the other participants may have problems hearing or seeing the other. Allocating a new connection to establish new streams does not fix that, nor does restarting the browser.
Question: What’s the recommended way to detect possible resource locks (e.g. does another application hog the camera/microphone)?
Web-Client: Network
Bandwidth and packet loss are a challenge, for example this inspector graph:
Audio and video of both participants is all over the place, and while we can not control the network connections the web-client should be able to reliably give useful information.
Question: Other than continuous connection monitoring with getStats() and maybe the experimental navigator.connection property, how can the web-client monitor network connectivity?
Pre-Call Test
We recommend to customers to run a pre-call test and have implemented it on our site as well. However, results of that test often times do not reflect the in-session connectivity. Worse, a pre-call test may detect a low (no video) bandwidth while Skype works just fine.
Question: How can that be?
I'm a member of the TokBox development team. I remember you reported an issue with the Python SDK, thanks for that!
Web-Client: A/V Resources
Most acquisition issues are detected by the JS SDK and if they aren't then we'd really like to hear about it! Please report reproduction steps or affected session IDs to TokBox support (referencing this StackOverflow question): https://support.tokbox.com/hc/en-us/requests/new
Most acquisition errors appear as OT_HARDWARE_UNAVAILABLE or OT_MEDIA_ERR_ABORTED errors. Are you detecting and surfacing these errors to your users? There is also the special OT_CHROME_MICROPHONE_ACQUISITION_ERROR error which is due to a known issue with Chrome that has been mostly fixed since Chrome 63 (see https://bugs.chromium.org/p/webrtc/issues/detail?id=4799).
Web-Client: Network
Unfortunately this is one of the more difficult issues to troubleshoot. Yes, Subscriber#getStats() is the best tool we have at our disposal and is a wrapper around the native RTCPeerConnection#getStats() function. Unfortunately we don't have much control over the values returned by the native function and if you think our SDK is returning incorrect values when compared with values from RTCPeerConnection#getStats() then please let us know!
It would be worthwhile confirming whether the issue is reproducible in all browsers or only a particular one. If you have detailed data regarding the inaccuracy of the native RTCPeerConnection#getStats() function then we could work together to report it to the browser vendor(s).
Fortunately we have just released the new Publisher#getStats() function which lets you get the publisher side of the stats. This should help you narrow down the cause of a connectivity issue to either a publisher or subscriber side. Please let us know if this helps with tracking down these issues.
Pre-Call Test
Again, these tests are based on Subscriber#getStats() which in turn are based on RTCPeerConnection#getStats(), the accuracy of which is out of our hands, but we'd love any reproduction steps to either fix a bug in our client SDK or report a bug to the browser vendors.
Just to confirm though, when you say you've implemented a pre-call test in your site, did you use the official JavaScript network test module? https://github.com/opentok/opentok-network-test-js This is actually what's used by the TokBox pre-call test.
#Aiham, thanks for responding, I've been looking at the the new Publisher#getStats() you linked to (thank you!), so we too can give our users some way of visibly seeing the network conditions that might be affected the quality of their call (and who's causing it). However, it seems as though bytes / packets sent goes up sharply as the number of subscribers increases, even though we're in a routed session.
Am I wrong to expect the Publisher#getStats() statistics to stay fairly stable regardless of the number of subscribers then receiving that stream in a routed session? I expected the nature of a routed call to mean it's sent once to the OpenTok Media Servers, and the statistics would end there.
After a WebRTC session has been established, I would like to stop sending the stream to the peer at some point and then resume sending later.
If I call removeStream, it does indeed stop sending data. If I then call addStream (with the previously removed stream), it resumes. Great!
However, is this safe? Or do I need to re-negotiate/exchange the SDP after the removal and/or after the re-add? I've seen it mentioned in several places that the SDP needs to be re-negotiated after such changes, but I'm wondering if it is ok in this simple case where the same stream is being removed and re-added?
PS: Just in case anyone wants to suggest it: I don't want to change the enabled state of the track since I still need the local stream playing, even while it is not being sent to the peer.
It will only work in Chrome, and is non-spec, so it's neither web compatible nor future-proof.
The spec has pivoted from streams to tracks, sporting addTrack and removeTrack instead of addStream and removeStream. As a result the latter isn't even implemented in Firefox.
Unfortunately, because Chrome hasn't caught up, this means renegotiation currently works differently in different browsers. It is possible to do however with some effort.
The new model has a cleaner separation between streams and what's sent in RTCPeerConnection.
Instead of renegotiating, setting track.enabled = false is a good idea. You can still play a local view of your video by cloning the track.
I have a client/server audio synthesizer where the server (java) dynamically generates an audio stream (Ogg/Vorbis) to be rendered by the client using an HTML5 audio element. Users can tweak various parameters and the server immediately alters the output accordingly. Unfortunately the audio element buffers (prefetches) very aggressively so changes made by the user won't be heard until minutes later, literally.
Trying to disable preload has no effect, and apparently this setting is only 'advisory' so there's no guarantee that it's behavior would be consistent across browsers.
I've been reading everything that I can find on WebRTC and the evolving WebAudio API and it seems like all of the pieces I need are there but I don't know if it's possible to connect them up the way I'd like to.
I looked at RTCPeerConnection, it does provide low latency but it brings in a lot of baggage that I don't want or need (STUN, ICE, offer/answer, etc) and currently it seems to only support a limited set of codecs, mostly geared towards voice. Also since the server side is in java I think I'd have to do a lot of work to teach it to 'speak' the various protocols and formats involved.
AudioContext.decodeAudioData works great for a static sample, but not for a stream since it doesn't process the incoming data until it's consumed the entire stream.
What I want is the exact functionality of the audio tag (i.e. HTMLAudioElement) without any buffering. If I could somehow create a MediaStream object that uses the server URL for its input then I could create a MediaStreamAudioSourceNode and send that output to context.destination. This is not very different than what AudioContext.decodeAudioData already does, except that method creates a static buffer, not a stream.
I would like to keep the Ogg/Vorbis compression and eventually use other codecs, but one thing that I may try next is to send raw PCM and build audio buffers on the fly, just as if they were being generated programatically by javascript code. But again, I think all of the parts already exist, and if there's any way to leverage that I would be most thrilled to know about it!
Thanks in advance,
Joe
How are you getting on ? Did you resolve this question ? I am solving a similar challenge. On the browser side I'm using web audio API which has nice ways to render streaming input audio data, and nodejs on the server side using web sockets as the middleware to send the browser streaming PCM buffers.