Google Cloud Speech-to-Text: "INVALID_ARGUMENT: Invalid recognition 'config': bad encoding.." codec audio encoding error - getusermedia

I'm recording short audio files (a few seconds) in Chrome using mediaDevices.getUserMedia(), saving the file to Firebase Storage, and then trying to send the files to Google Cloud Speech-to-Text from a Firebase Cloud Function. I'm getting back this error message:
INVALID_ARGUMENT: Invalid recognition 'config': bad encoding.
Google's documentation says that this error message means
Your audio data might not be encoded correctly or is encoded with a
codec different than what you've declared in the RecognitionConfig.
Check the audio input and make sure that you've set the encoding field
correctly.
In the browser I set up the microphone:
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
var options = {
audioBitsPerSecond : 128000,
mimeType : 'audio/webm;codecs=opus'
};
const mediaRecorder = new MediaRecorder(stream, options);
mediaRecorder.start();
...
According to this answer Chrome only supports two codecs:
audio/webm
audio/webm;codecs=opus
Actually, that's one media format and one codec. This blog post also says that Chrome only supports the Opus codec.
I set up my Firebase Cloud Function:
// Imports the Google Cloud client library
const speech = require('#google-cloud/speech');
// Creates a client
const client = new speech.SpeechClient();
const gcsUri = 'gs://my-app.appspot.com/my-file';
const encoding = 'Opus';
const sampleRateHertz = 128000;
const languageCode = 'en-US';
const config = {
encoding: encoding,
sampleRateHertz: sampleRateHertz,
languageCode: languageCode,
};
const audio = {
uri: gcsUri,
};
const request = {
config: config,
audio: audio,
};
// Detects speech in the audio file
return response = client.recognize(request) // square brackets in ES6 construct an array
.then(function(response) {
console.log(response);
...
The audio encoding matches between the browser and the Google Speech-to-Text request. Why does Google Speech tell me that the audio encoding is bad?
I also tried using the default options in the browser, with the same error message:
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
const mediaRecorder = new MediaRecorder(stream);
mediaRecorder.start();
In the Firebase Cloud Function I tried leaving out the line const encoding = 'Opus';, which resulted in an error encoding is not defined. I tried this line const encoding = ''; which resulted in the INVALID_ARGUMENT: Invalid recognition 'config': bad encoding.. error.
I'm getting a similar error message from IBM Watson Speech-to-Text. The file plays back without a problem.

Related

In an Expo React Native app, how to pick a photo, store it locally on the phone, and later upload it to a Node.js service?

I'm building a React Native app with Expo, and I want to include the following workflow: The user takes a picture (either with the camera or picking one from the phone's gallery), which is stored locally on the phone, until the user uploads it some later time to a backend service.
I'm pretty stuck and would appreciate any pointers.
Here is what I have:
I use expo-image-picker to pick a photo from the phone's gallery:
const photo = await launchImageLibraryAsync({
mediaTypes: ImagePicker.MediaTypeOptions.All,
allowsEditing: true,
base64: true,
quality: 1,
});
Then I store the photo locally as a Base64 string using expo-file-system:
const location = `${FileSystem.documentDirectory}${filename}`;
await FileSystem.writeAsStringAsync(location, photo.base64, {
encoding: FileSystem.EncodingType.Base64
});
I keep information about the storage location, file name, and mime type in an image object. Later, I try to upload that image to my own Node.js backend service with axios, sending the following multi-part form data:
const formdata = new FormData();
formdata.append('file', {
path: image.location,
name: image.filename,
type: image.mimetype
} as any);
The backend service that receives the photo uses multer:
const multer = require('multer');
const upload = multer({ storage: multer.memoryStorage() });
router.post('/photo', upload.single('file'), async (request, response) => {
console.log(request.file);
...
});
What arrives at my service is the following:
{
fieldname: 'file',
originalname: '1653135701413.jpg',
encoding: '7bit',
mimetype: 'image/jpg',
buffer: <Buffer >,
size: 0
}
So no data is transferred. (It seems to be properly stored on the phone, because if I use the Expo filesystem's readStringAsAsync, I do get a pretty long Base64 string.)
What am I missing? Do I need to send the image as a blob? (If I try to do so, then request.file is undefined, so I guess I'm doing something wrong there as well.)
And in general, is there a better way to achieve this workflow in a managed React Native app? (For example, is it ok to store the image as a Base64 string, or would it be better to do this differently?)
Edit:
In the form data, I changed path to uri, and I switched from axios to fetch. Now the backend finally receives the image data. 🥳

Upload Video URI from React-Native Picker to AWS S3 Server

I am trying to upload a video from my IOS device library to S3 using axios and a pre-signed url. I've determined the axios/s3 part is working great, but the issue is coming from the uri I receive from 'react-native-image-picker'.
When I record a video in react-native the video uri uploads fine in S3, but when I grab a video from my photo library it uploads to S3 but it's not a video file.
I grab the video uri from my ios device library using react-native-image-picker.
import {launchImageLibrary} from 'react-native-image-picker';
launchImageLibrary({mediaType: "video"}, ({assets}) => {
let {uri} = assets[0] //uri = /var/mobile/Containers/Data/Application/123/tmp/IMG_1779.mov
uploadFile(uri)
});
and then I attempt to upload the uri to S3
//save to s3 using presignedPost
uploadFile(uri){
var formData = new FormData();
...
formData.append("file", { uri });
await axios.post(presignedPost.url, formData,{ 'Content-Type': 'multipart/form-data' } )
}
The function is successful, but when I look in AWS the file is just a text file with a bunch of random characters.
The good news is this same exact uploadFile functions works if I record a video with react-native-camera
import { RNCamera } from 'react-native-camera';
stopRecord(){
let camera = cameraRef.current //grab camera <RNCamera ref={cameraRef} />
let {uri = null} = await camera.recordAsync(); //uri = /var/mobile/Containers/Data/Application/123/Library/Caches/Camera/123.mov
uploadFile(uri)
}
The only difference I can see is the uri after recording a video is stored in cache. Therefore, I attempted to use 'react-native-fs' to grab the picker uri, save it to cache, and then upload the cached file but I got the same error (file uploads to s3 but not a video).
import * as RNFS from 'react-native-fs';
uploadFileTwo(uri){
let base64Data = await RNFS.readFile(uri, 'base64')
let cachePath = RNFS.CachesDirectoryPath + "/" + fileName + ".mov"
await RNFS.writeFile(cachePath, base64Data, 'base64')
var formData = new FormData();
...
formData.append("file", { uri: cachePath });
await axios.post(presignedPost.url, formData,{ 'Content-Type': 'multipart/form-data' } )
}
So now I am out of options. Why would the 'react-native-camera' uri work great, but the 'react-native-image-picker' uri doesn't?
Have a try by following the Tutorial for uploading video file.
I think this tutorial meets all your requirements as it uses react-native-image-picker to upload the video file(s).

React Native Uploading Video to YouTube (Stuck at Processing)

I am attempting to upload video files to YouTube using their v3 API and their MediaUploader. It works in my ReactJS application, but not in my React Native application. When uploading via React Native, the upload completes, then stalls at 100%. In my YouTube account, I can see the new video file, but it is stuck at "Video is still being processed."
I believe the issue may be that I need to send a video file and not an object with a video uri but I don't know how to get around that.
I am using the YouTube MediaUploader from the CORS example at https://github.com/youtube/api-samples/blob/master/javascript/cors_upload.js I am using an OAuth 2.0 client Id, and this setup works correctly when using the ReactJS app via my website. I am using React Native Expo with Camera, which returns me an Object with a URI, for example:
Video File: Object {
"uri": "file:///var/mobile/Containers/Data/Application/353A7969-E2A8-4C80-B641-C80B2B029555/Library/Caches/ExponentExperienceData/%2540dj_walksalot%252Fwandereo/Camera/E971DFEC-AB3E-4B6D-892F-9027AFE47A1A.mov",
}
This file can be viewed in the application, and I can even successfully send this to my server for playback on the web app and in the React Native app. However, sending this object in the MediaUploader does not work. It will take an appropriate amount of time to upload, but then sits at 100%, while my YouTube account will show it has received the video with the correct metadata, but the video itself remains stuck at "Video is still being processed."
video_file: Object {
"uri": "file:///var/mobile/Containers/Data/Application/353A7969-E2A8-4C80-B641-C80B2B029555/Library/Caches/ExponentExperienceData/%2540dj_walksalot%252Fwandereo/Camera/E971DFEC-AB3E-4B6D-892F-9027AFE47A1A.mov",
}
export const uploadToYouTube = (access_token, video_file, metadata) => async (dispatch) => {
...cors_upload...
var uploader = new MediaUploader({
baseUrl: `https://www.googleapis.com/upload/youtube/v3/videos?part=snippet%2Cstatus&key=API_KEY`,
file: video_file,
token: access_token,
metadata: metadata,
contentType: 'video/quicktime',
// contentType: 'application/octet-stream',//"video/*",
// contentType = options.contentType || this.file.type || 'application/octet-stream';
params: {
part: Object.keys(metadata).join(',')
},
onError: function(data) {
// onError code
let err = JSON.parse(data);
dispatch(returnErrors(err.message, err.code))
console.log('Error: ', err);
},
onProgress: function(progressEvent){
// onProgress code
let percentCompleted = Math.round((progressEvent.loaded * 100) / progressEvent.total);
dispatch({
type: UPLOAD_PROGRESS,
payload: percentCompleted
});
},
onComplete: function(data) {
console.log('Complete');
// onComplete code
let responseData = JSON.parse(data);
dispatch({
type: UPLOAD_YOUTUBE_VIDEO,
payload: JSON.parse(data)
})
dispatch({
type: UPLOAD_PROGRESS,
payload: 0
});
}
});
uploader.upload();
}
Similar to my currently-working web app, after completing the upload, the "onComplete" function should fire, and YouTube should process the video. This does not happen. I believe it's because I'm attaching an object with a URI and not the actual file.
I was able to solve this from a post at Expert Mill by Joe Edgar at https://www.expertmill.com/2018/10/19/using-and-uploading-dynamically-created-local-files-in-react-native-and-expo/
By using fetch and .blob() I was able to convert the URI object to a data object and upload. Additional code:
const file = await fetch(video_file.uri);
const file_blob = await file.blob();
No need to install RNFetchBlob since this is in the Expo SDK.

Implementing Google Cloud Speech-to-Text on React-Native

I'm trying to implement google's speech-to-text on a react native app but I can't find an example or documentation about it, I'm fairly new to react-native so I'm kinda lost, there's a sample using node.js on google official docs and I'm trying to 'copy' it to react-native but I didn't had any success.
Here's the Node.js example:
async function main() {
// Imports the Google Cloud client library
const speech = require('#google-cloud/speech');
const fs = require('fs');
// Creates a client
const client = new speech.SpeechClient();
// The name of the audio file to transcribe
const fileName = './resources/audio.raw';
// Reads a local audio file and converts it to base64
const file = fs.readFileSync(fileName);
const audioBytes = file.toString('base64');
// The audio file's encoding, sample rate in hertz, and BCP-47 language code
const audio = {
content: audioBytes,
};
const config = {
encoding: 'LINEAR16',
sampleRateHertz: 16000,
languageCode: 'en-US',
};
const request = {
audio: audio,
config: config,
};
// Detects speech in the audio file
const [response] = await client.recognize(request);
const transcription = response.results
.map(result => result.alternatives[0].transcript)
.join('\n');
console.log(`Transcription: ${transcription}`);
}
main().catch(console.error);
First of all, the 'fs' package doesn't work on react-native, so I had to use 'react-native-fs' which has different functions
Second, should I really use the 'require' to call the speech package? I guess react-native would use 'import' instead, right?
Any tips on how can I implement it on react-native? Thanks!
You can't run #google-cloud/speech on react native as RN runs on JV environment and the library (#google-cloud/speech) runs on nodejs environment. You could create an api and deploy it or else use this library react-native-google-speech-api

Using Fetch instead of XMLHttpRequest in React Native

I was trying to upload images to S3 from my react native app by following and adapting this guide by heroku: https://devcenter.heroku.com/articles/s3-upload-node
Essentially I am using the aws-sdk on my express.js backend to generate pre-signed request for uploading images to S3 from react native.
Everything works well, so then I tried to convert the XMLHttpRequests into fetch requests, which seem to be favoured by react native. After the conversion, the files are still being uploaded to S3, but when I click on the image links, then the images wouldn't not show properly, instead an empty square is shown:
Empty square shown instead of image
More specifically it seems to be this piece of code conversion that causes it to happen:
From:
_uploadFile(file, signedRequest, url){
const xhr = new XMLHttpRequest();
xhr.open('PUT', signedRequest);
xhr.onreadystatechange = () => {
if(xhr.readyState === 4){
if(xhr.status === 200){
console.log("UPLOAD DONE");
} else {
alert('ERROR UPLOADING');
}
}
};
xhr.send(file);
}
To:
_uploadFile(file, signedRequest, url) {
let option = {
method: "PUT",
headers: {
"Content-Type": "image/jpeg",
},
body: JSON.stringify(file)
}
fetch(signedRequest, option)
.then(res => console.log("UPLOAD DONE"))
.catch(err => console.log("ERROR UPLOADING: ", err))
}
The file object being uploaded:
{
name: "profileImage",
type: "image/jpeg",
uri: 'data:image/jpeg;base64,' + response.data, //just a base64 image string
isStatic: true
}
Could anyone shed some light on why this could be happening, or have had similar experiences? Many thanks!
In your fetch example you put a JSON string in your body. It will be sent to S3 but it will not be interpreted as an image upload. You should be able to construct a FormData object yourself and pass it to fetch as the request body, but I think using XHR is the simpler option. According to this comment it's what Facebook does as well (the comment is over a year old).
If at all possible you should also try to use local URIs instead of passing Base64 encoded data. It takes quite a while to transfer a few MB of image data between JS and native.