I'm trying to stream audio from a browser to a gstreamer pipeline on a server.
I'm currently using Kurento, and modifying the Hello World example to try to connect an RTP Endpoint to the pipeline -- but am having trouble.
I know the media is getting there because when I swap in a Recording Endpoint, I get a valid recording.
The Kurento Node JS is:
pipeline.create("RtpEndpoint", {}, function(error, rtpEndpoint) {
if (error) {
console.log("Recorder problem");
return sendError(res, 500, error);
}
console.log("Creating WebRtcEndpoint");
pipeline.create('WebRtcEndpoint', function(error, webRtcEndpoint) {
if (error) {
return sendError(res, 500, error);
}
console.log("Processing sdpOffer at server and generating sdpAnswer");
webRtcEndpoint.processOffer(sdpOffer, function(error, sdpAnswer) {
if (error) {
webRtcEndpoint.release();
return sendError(res, 500, error);
}
console.log("Connecting loopback");
webRtcEndpoint.connect(webRtcEndpoint, function(error) {
if(error){
webRtcEndpoint.release();
return sendError(res, 500, error);
}
console.log("Sending sdpAnswer to client");
console.log(sdpAnswer);
webRtcEndpoint.connect(rtpEndpoint, function(error) {
if(error) {
webRtcEndpoint.release();
return sendError(res, 500, error);
}
rtpEndpoint.generateOffer(function(error, offer) {
fs.writeFile('/tmp/test.sdp',offer);
console.log("RTP OFFER GENERATED.");
});
});
res.type('application/sdp');
res.send(sdpAnswer);
});
});
});
});
and my GStreamer pipeline is:
gst-launch-1.0 -vvvv filesrc location=/tmp/test.sdp ! sdpdemux ! decodebin ! autovideosink
which returns
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0-actual-sink-glimage': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstUDPSrc:udpsrc0: timeout = 10000000000
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstUDPSrc:udpsrc2: timeout = 10000000000
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstRtpBin:rtpbin0/GstRtpSession:rtpsession0.GstPad:send_rtcp_src: caps = application/x-rtcp
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstRtpBin:rtpbin0.GstGhostPad:send_rtcp_src_0: caps = application/x-rtcp
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstUDPSink:udpsink0.GstPad:sink: caps = application/x-rtcp
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstRtpBin:rtpbin0.GstGhostPad:send_rtcp_src_0.GstProxyPad:proxypad4: caps = application/x-rtcp
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstRtpBin:rtpbin0/GstRtpSession:rtpsession1.GstPad:send_rtcp_src: caps = application/x-rtcp
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstRtpBin:rtpbin0.GstGhostPad:send_rtcp_src_1: caps = application/x-rtcp
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstUDPSink:udpsink1.GstPad:sink: caps = application/x-rtcp
/GstPipeline:pipeline0/GstSDPDemux:sdpdemux0/GstRtpBin:rtpbin0.GstGhostPad:send_rtcp_src_1.GstProxyPad:proxypad7: caps = application/x-rtcp
ERROR: from element /GstPipeline:pipeline0/GstSDPDemux:sdpdemux0: Could not read from resource.
Additional debug info:
gstsdpdemux.c(1213): gst_sdp_demux_handle_message (): /GstPipeline:pipeline0/GstSDPDemux:sdpdemux0:
Could not receive any UDP packets for 10.0000 seconds, maybe your firewall is blocking it.
Execution ended after 0:00:10.062018001
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
It doesnt work on FFPMEG, VLC, etc -- results are similar to "Attempt 5.3" here: https://altanaitelecom.wordpress.com/2015/02/26/continue-streaming-broadcasting-live-video-call-to-non-webrtc-supported-browsers-and-media-players/
I don't think theres a firewall issue as the pipeline and kurento instance are on the same virtual machine (which has no firewall) -- and the recording endpoint works. Is it being linked badly? Is there an easier way?
Using RtpEndpoint is tricky because you need to complete de SDP negotiation. This means that somewhere after the
rtpEndpoint.generateOffer(...
you should be invoking
rtpEndpoint.processAnswer(sdpAnswer, ...)
The tricky part is that you need to obtain the sdpAnswer from your gstreamer pipeline and this is not trivial if you want to it just using gst-launch. Probably your best option is to write a small programm creating the pipeline and generating the sdpAnswer so that you can give it back to the rtpEndpoint through your signaling mechanism.
Related
I am using scp2 to copy a file to targetPath. config contains host, username, privateKey, path and port.
const client = require('scp2');
export function scpAsync(config, targetPath) {
return new Promise((resolve, reject) => {
client.scp(config, targetPath, err => {
if (!err){
resolve();
} else {
const errorMessage = err;
reject(errorMessage);
}
});
});
}
When doing so I am getting the error:
Error: Timed out while waiting for handshake
I tried to pass also
promptForPass: false
but it did not change anything. Besides that I used debug mode which told me that I am connected to the server and I put a higher setTimeout but then the error is just coming later. I was checking the documentation of scp2 and their GitHub. I use the function like explained there (https://www.npmjs.com/package/scp2) and regarding the error they could fix it with an higher setTimeout (https://github.com/spmjs/node-scp2/issues/107). I tried with a local ftp server, ngrok and ftp on ec2 instance. All with the same problem.
I would be happy to get help. I asked this question also on superuser but did not get an answer:
https://superuser.com/questions/1576964/error-timed-out-while-waiting-for-handshake
I just implemented my first backend file where I fetch some user data, messages and so on.
Now I wanted to include error handling if there is no network available.
I don´t know if I did it right but this was my approach so far:
import axios from 'axios'
const host = process.env.VUE_APP_URL
export default {
person: async function (currentPerson) {
let params = {
currentPerson: localStorage.getItem("person"),
};
if (user) {
params['currentPerson'] = currentPerson;
}
return axios.get(`${host}/api/currentPerson`, {
params: params
})
//catching network errors
.catch (error => {
if (error.response) {
/*
* The request was made and the server responded with a
4xx/5xx error
*/
console.log(error.response.data);
console.log(error.response.status);
console.log(error.response.headers);
} else if (error.request) {
/*
* The request was made but no response was received
*/
console.log(error.request);
} else {
// Something happened in setting up the request and triggered an Error
console.log('Error', error.message);
}
console.log(error)
});
},
In my mounted() function of my main view I fetch the data from my backend file from above:
backend.matches().then(function (response) {
self.contacts = response.data.persons;
});
I tried to check in console if it is working but all I get is the following:
In the catch block I check for
response errors: like 4xx/5xx
request errors: if my network not responding in time
and any other errors
Would this be the right approach to check if a network is available or not? Or does it degrade the user experience when the user checks the error?
My backend file includes more methods.. do I have to write for each method these kind of requests?
In your backend file you don't react whether there is a network connection or not I think.
And only for reference: that is not the backend, but communicates with the backend - the backend is the part of your code what you communicate with, e.g. Laravel code, an API, ...
Try adding the following at the beginning of your catch part:
if (!error.response) {
//network error
console.log('No network connection');
} else if (error.response) {
//the rest of your code
This should print out No network connection in your console.
Run your application, turn off the internet connection and check the console.
These kind of code should always be located in your backend part.
My answer maybe different from your question.
When i create a .net core API with Angular i used three things to check is there network or not?
subscribe to windows's offline/online event
create signalR hub from layout component to API server
API request failed (it means lot of incident, but if 1. or 2. case is true i know what cause 3. case
I'm a server side developer with rudimentary JS knowhow. I'm tinkering with Agora's audio broadcasting functionality (specifically for the web). For reference, I've been following this: https://docs.agora.io/en/Audio%20Broadcast/start_live_audio_web?platform=Web
I'm attempting to broadcast audio as a host. I have an HTML button which fires a JS function where I:
Initialize a client
Set the role
Join a predefined channel
Publish a local stream
My understanding is that accomplshing the aforementioned will enable me to broadcast audio. When I try this, I end up getting a client join failed DYNAMIC_KEY_EXPIRED error. I'm unable to find documentation regarding how to resolve this. Can you help me resolve this? An illustrative example would be nice.
My JS code is below. Note that I'm using a temp token to test this functionality on localhost.
// rtc object
var rtc = {
client: null,
joined: false,
published: false,
localStream: null,
remoteStreams: [],
params: {}
};
// Options for joining a channel
var option = {
appID: "anAppID",// from 'Project Management' dashboard
channel: "AudioLive",
uid: null,//The user ID should be unique in a channel. If you set the user ID as null or 0, the Agora server assigns a user ID and returns it in the onSuccess callback.
token: "aTempToken"// TEMP Token
}
function createBroadcast(role) {
console.log("entered createBroadcast");
// Create a client
rtc.client = AgoraRTC.createClient({mode: "live", codec: "h264"});
// Initialize the client
rtc.client.init(option.appID, function () {
console.log("init success");
// Note: in a live broadcast, only the host can be heard and seen. You can also call setClientRole() to change the user role after joining a channel.
rtc.client.setClientRole(role);
console.log("role is set");
// Call Client.join in the onSuccess callback of Client.init
rtc.client.join(option.token ? option.token : null, option.channel, option.uid ? +option.uid : null, function (uid) {
console.log("join channel: " + option.channel + " success, uid: " + uid);
rtc.params.uid = uid;
// Call AgoraRTC.createStream to create a stream in the onSuccess callback of Client.join
rtc.localStream = AgoraRTC.createStream({
streamID: rtc.params.uid,
audio: true,
video: false,
screen: false,
})
// Call Stream.init to initialize the stream after 'creating' the stream above
// Initialize the local stream
rtc.localStream.init(function () {
console.log("init local stream success");
// play stream with html element id "local_stream"
rtc.localStream.play("local_stream");
// Call Client.publish in the onSuccess callback of Stream.init to publish the local stream
// Publish the local stream
rtc.client.publish(rtc.localStream, function (err) {
console.log("publish failed");
console.error(err);
})
}, function (err) {
console.error("init local stream failed ", err);
});
}, function(err) {
console.error("client join failed", err)
})
}, (err) => {
console.error(err);
});
}
<div style="background:#f0f3f4;padding:20px">
<button id="broadast" style="height:40px;width:200px" onclick="createBroadcast('host')">Start Live Broadcast</button>
</div>
I've not added the actual values for appID and token in the code above.
Note: Please ask for more information in case you require it.
The error that you are facing is due to the expiry of the token generated for authentication purposes while generating an APP ID. To resolve this you will have to generate a new token as elaborated in the below given links:
Token-expired
renewToken
A token (or a temporary token) expires after a certain period of time. When the SDK notifies the client that the token is about to expire or has expired by the onTokenPrivilegeWillExpire or onTokenPrivilegeDidExpire callbacks, you need to generate a new token and call the renewToken method.
client.on("onTokenPrivilegeWillExpire", function(){
//After requesting a new token
client.renewToken(token);
});
client.on("onTokenPrivilegeDidExpire", function(){
//After requesting a new token
client.renewToken(token);
});
Include the above functions in your javascript code along with the rest of the eventListeners.
Incase your application doesn't require security you can opt to not use a token and generate an App ID without a certificate.
App ID without certificate
Do get back for further support incase the issue remains unresolved.
In this document, it uses URL.createObjectURL to set the video source. (This is the code to answer a call).
var offer = getOfferFromFriend();
navigator.getUserMedia({video: true}, function(stream) {
pc.onaddstream = e => video.src = URL.createObjectURL(e.stream);
pc.addStream(stream);
pc.setRemoteDescription(new RTCSessionDescription(offer), function() {
pc.createAnswer(function(answer) {
pc.setLocalDescription(answer, function() {
// send the answer to a server to be forwarded back to the caller (you)
}, error);
}, error);
}, error);
});
I expected video.src to be the address to retrieve the remote video. So it should be fixed and given by the other side of the connection (whoever initiated the call). But the value of URL.createObjectURL is generated on the answerer's side, and it event depends on when the function is called. How it can be used to get the remote video stream?
Edit:
The result of URL.createObjectURL looks like blob:http://some.site.com/xxxx-the-token-xxxx. With this string, how does the video component know where to load the remote stream? Is there a hashmap of {url:stream} stored somewhere? If so, how does the video component access the hashmap?
A stream object does store a token string, which you can get with stream.toURL. But it is different from the result of URL.createObjectURL. The value of URL.createObjectURL depends on time. If you call it twice in a row, you get different values.
URL.createObjectURL(stream) is a hack. Stop using it. Efforts are underway to remove it.
Use video.srcObject = stream directly instead. It is standard and well-implemented.
This assignment of a local resource should never have been a URL in the first place, and is a red herring to understanding how WebRTC works.
WebRTC is a transmission API, sending data directly from one peer to another. No content URLs are involved. The remote stream you get from onaddstream is a local object receiver side, and is the live streaming result of the transmission, ready to be played.
The documentation you read is old and outdated. Thanks for pointing it out, I'll fix it. It has other problems: you should call setRemoteDescription immediately, not wait for the receiver to share their camera, otherwise incoming candidates are missed. Instead of the code you show, do this:
pc.onaddstream = e => video.srcObject = e.stream;
function getOfferFromFriend(offer) {
return pc.setRemoteDescription(new RTCSessionDescription(offer))
.then(() => navigator.getUserMedia({video: true}))
.then(stream => {
pc.addStream(stream);
return pc.createAnswer();
})
.then(answer => pc.setLocalDescription(answer))
.then(() => {
// send the answer to a server to be forwarded back to the caller (you)
})
.catch(error);
}
It uses srcObject, avoids the deprecated callback API, and won't cause intermittent ICE failures.
Because a WebRTC connection involves several steps and what you get from such a connection is a stream. But the src property of the video tag does not accept a stream, but a URL. And this is the way to "convert" a stream to a URL.
I've written a very small nodejs script. Nothing fancy, just listen to a request, generate a big random string of 108KB, perform an i/o by writing it to a file, then reading from it.
And then finally, put it back on the response stream.
The idea is to benchmark node.js capabilities against other systems like lamp/asp.net for a web app I'm going to develop:
//server.js
var http = require('http');
var server = http.createServer(handler);
function handler(request, response) {
//console.log('request received!');
response.writeHead(200, {'Content-Type': 'text/plain'});
s=""; //generate a random string of 108KB
for(i=0;i<108000;i++)
{
n=Math.floor(65 + (Math.random()*(122-65)) );
s+=String.fromCharCode(n);
}
//write s to a file
var fs = require('fs');
fs.writeFile("godspeed.txt", s, function(err) {
if (err) throw err;
//console.log("The file was saved!");
//read back from the file
fs.readFile('godspeed.txt', function (err, data) {
if (err) throw err;
result = data;
response.end(result);
});
}
);
}
server.listen(8124);
console.log('Server running at http://127.0.0.1:8124/');
The above works fine and returns the string when I open it from a browser. But when I run the apache benchmarking tool over this (with 2000 total requests and 200 concurrent requests), about 911 requests are shown as failed meaning something is wrong with my code:
Concurrency Level: 200
Time taken for tests: 41.887 seconds
Complete requests: 2000
**Failed requests: 911**
Can you help me figure out what is wrong with my code?
(Apart from these failed requests, node.js is excellent at performance considering the time total taken compared to other systems.)