Hi I have been successfully recording an Agora audio call, where one person speaks in a broadcast role, and during the call mixes in a number of audio files.
All the audio was being recorded until we upgraded to flutter 2 and associated upgraded packages.
Now all that is recorded is the broadcaster voice, and no mixed in audio.
The broadcaster and audience members can all hear the mixed in audio within the call without issue.
The code (Flutter) is similar to this:
Mix in Audio into a valid RTC session, with default settings
final playing = await session.playAudioFile(path, (){
state = MessagePlayerState.STOPPED;
if (!disposing) {
whenFinished();
}
});
The recording options are as follows (My UID is a hardcoded string, that is not the same as any participant UIDs)
http.Response response = await http.post(
Uri.https(AGORA_REST_URL, '$AGORA_REST_API_VERSION/$appId/cloud_recording/resourceid/$resourceId/mode/mix/start'),
headers: <String, String>{
HttpHeaders.authorizationHeader: 'Basic $basicAuth',
HttpHeaders.contentTypeHeader: 'application/json; charset=UTF-8',
},
body: jsonEncode(<String, dynamic>{
'cname': channelName,
'uid': uid,
'clientRequest': {
'recordingConfig':{
'channelType':0,
'streamTypes':2, // TODO: Should be a streamTypes of 0 (audio only), but get failures.
'audioProfile':1,
'videoStreamType':0,
'maxIdleTime':120,
'transcodingConfig':{
'width':360,
'height':640,
'fps':30,
'bitrate':600,
'maxResolutionUid':'1',
'mixedVideoLayout':1
},
'recordingFileConfig':{
'avFileType': ['hls','mp4']
}
},
'storageConfig':{
'vendor':1,
'region':3,
'bucket':AWS_RECORDING_BUCKET, // TODO: Env Var
'accessKey':AWS_BUCKET_ACCESS_KEY,
'secretKey':AWS_BUCKET_SECRET_KEY,
}
},
}),
);
The m3u8 and ts files are present in the S3 bucket.
Adjusting the metadata tags in S3 results in a file that plays fine in Safari, but no mixed in audio is heard.
Converting the file to aac with ffmpeg shows this error
[hls # 0x7fd6cc808200] Opening '2838cfc6254e9fec2e3088976f39d7ce_bip_20210618014151427.ts' for reading
[mpegts # 0x7fd6cc00a600] Packet corrupt (stream = 0, dts = 1437390).
size= 480kB time=00:00:30.69 bitrate= 128.1kbits/s speed=1.49e+03x
video:0kB audio:470kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.093976%
And the result is the same as from the S3 bucket.
Any help or hints appreciated.
This can be closed/ignored. Turns out we had an edge condition that did not show when the app was used normally, but if, for instance, you wanted a very stage managed recording to show off to others it broke.
Related
User started with DataChannel only.
AudioChannel is added through renegotiation later.
var mLocalAudio;
navigator.getUserMedia({ video: false, audio: true },
function (myStream) {
mLocalAudio = myStream;
mConn.addStream(myStream);
}, function (e) { });
On the remote peer, ontrack will be triggered and we add the stream on to an <audio> element.
But since this is a many to many connection, there will have multiple peers trying to swith on / off their audio channel from time to time.
My problem is, how can I identify which audio track is belongs to which user?
I am following this documentation to convert text to speech using the Text To Speech REST API.
I'm successfully able to get a valid response using Postman and I'm able to pay the audio in PostMan. But I am not able to play the audio using JavaScript. Below is my Javascript code. I'm not sure what to do with the response.
function bingSpeech(message) {
var authToken = "TokenToCommunicateWithRestAPI";
var http = new XMLHttpRequest();
var params = `<speak version='1.0' xml:lang='en-US'><voice xml:lang='en-US' xml:gender='Female' name='Microsoft Server Speech Text to Speech Voice (en-US, JessaRUS)'>${message}</voice></speak>`;
http.open('POST', 'https://speech.platform.bing.com/synthesize', true);
//Send the proper header information along with the request
http.setRequestHeader("Content-Type", "application/ssml+xml");
http.setRequestHeader("Authorization", "bearer " + authToken);
http.setRequestHeader("X-Microsoft-OutputFormat", "audio-16khz-32kbitrate-mono-mp3");
http.onreadystatechange = function () {
if (http.readyState == 4 && http.status == 200) {
// I am getting the respone, but I'm not sure how to play the audio file. Need help here
}
}
http.send(params);
}
Thanks.
I referred to the following repository for my code in Java. It plays the audio in IDE and saves the audio file to your system.
https://github.com/Azure-Samples/Cognitive-Speech-TTS/tree/master/Samples-Http/Java/TTSSample/src/com/microsoft/cognitiveservices/ttssample
I created an API function to work with S3. I imported the template swagger. After deployment, I tested with a Node.js project by the npm module aws-api-gateway-client.
It works well with: get bucket lists, get bucket info, get one item, put a bucket, put a plain text object, however I am blocked with put a binary file.
firstly, I ensure ACL is allowed with all permissions on S3. secondly, binary support also added
image/gif
application/octet-stream
The code snippet is as below. The behaviors are:
1) after invokeAPI, the callback function is never hit, after sometime, the Node.js project did not respond. no any error message. The file size (such as an image) is very small.
2) with only two times, the uploading seemed to work, but the result file size is bigger (around 2M bigger) than the original file, so the file is corrupt.
Could you help me out? Thank you!
var filepathname = './items/';
var filename = 'image1.png';
fs.stat(filepathname+filename, function (err, stats) {
var fileSize = stats.size ;
fs.readFile(filepathname+filename,'binary',function(err,data){
var len = data.length;
console.log('file len' + len);
var pathTemplate = '/my-test-bucket/' +filename ;
var method = 'PUT';
var params = {
folder: '',
item:''
};
var additionalParams = {
headers: {
'Content-Type': 'application/octet-stream',
//'Content-Type': 'image/gif',
'Content-Length': len
}
};
var result1 = apigClient.invokeApi(params,pathTemplate,method,additionalParams,data)
.then(function(result){
//never hit :(
console.log(result);
}).catch( function(result){
//never hit :(
console.log(result);
});;
});
});
We encountered the same problem. API Gateway is meant for limited data (10MB as of now), limits shown here,
http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html
Self Signed URL to S3:
Create an S3 self signed URL for POST from the lambda or the endpoint where you are trying to post.
How do I put object to amazon s3 using presigned url?
Now POST the image directly to S3.
Presigned POST:
Apart from posting the image if you want to post additional properties, you can post it in multi-form format as well.
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#createPresignedPost-property
If you want to process the file after delivering to S3, you can create a trigger from S3 upon creation and process with your Lambda or anypoint that need to process.
Hope it helps.
In this document, it uses URL.createObjectURL to set the video source. (This is the code to answer a call).
var offer = getOfferFromFriend();
navigator.getUserMedia({video: true}, function(stream) {
pc.onaddstream = e => video.src = URL.createObjectURL(e.stream);
pc.addStream(stream);
pc.setRemoteDescription(new RTCSessionDescription(offer), function() {
pc.createAnswer(function(answer) {
pc.setLocalDescription(answer, function() {
// send the answer to a server to be forwarded back to the caller (you)
}, error);
}, error);
}, error);
});
I expected video.src to be the address to retrieve the remote video. So it should be fixed and given by the other side of the connection (whoever initiated the call). But the value of URL.createObjectURL is generated on the answerer's side, and it event depends on when the function is called. How it can be used to get the remote video stream?
Edit:
The result of URL.createObjectURL looks like blob:http://some.site.com/xxxx-the-token-xxxx. With this string, how does the video component know where to load the remote stream? Is there a hashmap of {url:stream} stored somewhere? If so, how does the video component access the hashmap?
A stream object does store a token string, which you can get with stream.toURL. But it is different from the result of URL.createObjectURL. The value of URL.createObjectURL depends on time. If you call it twice in a row, you get different values.
URL.createObjectURL(stream) is a hack. Stop using it. Efforts are underway to remove it.
Use video.srcObject = stream directly instead. It is standard and well-implemented.
This assignment of a local resource should never have been a URL in the first place, and is a red herring to understanding how WebRTC works.
WebRTC is a transmission API, sending data directly from one peer to another. No content URLs are involved. The remote stream you get from onaddstream is a local object receiver side, and is the live streaming result of the transmission, ready to be played.
The documentation you read is old and outdated. Thanks for pointing it out, I'll fix it. It has other problems: you should call setRemoteDescription immediately, not wait for the receiver to share their camera, otherwise incoming candidates are missed. Instead of the code you show, do this:
pc.onaddstream = e => video.srcObject = e.stream;
function getOfferFromFriend(offer) {
return pc.setRemoteDescription(new RTCSessionDescription(offer))
.then(() => navigator.getUserMedia({video: true}))
.then(stream => {
pc.addStream(stream);
return pc.createAnswer();
})
.then(answer => pc.setLocalDescription(answer))
.then(() => {
// send the answer to a server to be forwarded back to the caller (you)
})
.catch(error);
}
It uses srcObject, avoids the deprecated callback API, and won't cause intermittent ICE failures.
Because a WebRTC connection involves several steps and what you get from such a connection is a stream. But the src property of the video tag does not accept a stream, but a URL. And this is the way to "convert" a stream to a URL.
I want to have only audio stream communication between peers , I changed the parts of kurento.utils.js to get only audio stream via getusermedia
but it's not working
I used this example node-hello-world example
WebRtcPeer.prototype.userMediaConstraints = {
audio : true,
video : {
mandatory : {
maxWidth : 640,
maxFrameRate : 15,
minFrameRate : 15
}
}
};
to
WebRtcPeer.prototype.userMediaConstraints = {
audio : true,
video : false
};
is it possible use kurento service for only audio stream?
This is indeed possible with Kurento. There are two ways of doing this, depending on the desired scope of the modification:
Per webrtc endpoint: when you process the SDP offer sent by the client, you get an SDP answer from KMS that you have to send back. After invoking the processOffer method call, you can tamper the SDP to remove all video parts. That way, your client will send back only audio.
Globally: You can edit /etc/kurento/sdp_pattern.txt file removing all video related parts, this will force SdpEndpoints (parent class of WebrtcEndpoint) to only use audio.
EDIT 1
The file sdp_pattern.txt is deprecated in KMS 6.1.0, so method 2 shouldn't be used.
EDIT 2
There was an issue with the kurento-utils library, and the client was not correctly setting the OfferToReceiveAudio. It was fixed some time ago, and you shouldn't need tampering the SDPs now.
git origin: https://github.com/Kurento/kurento-tutorial-js.git
git branch: 6.6.0
My solution is only changing var offerVideo = true; to var offerVideo = false; in generateOffer function of kurento-utils.js file.
My approach is to modify the options that you pass to the WebRtcPeer.
var options = {
onicecandidate: onIceCandidate,
iceServers: iceServers,
mediaConstraints: {
audio:true,
video:false
}
}
Besides, in the kurento-utils.js, the mediaContraints is overidden by this line:
constraints.unshift(MEDIA_CONSTRAINTS);
So comment it.