Google Speech To Text API in React Native - react-native

I'm doing a project by React Native with following features
1. Users start recording audio
2. Stop recording audio
3. Save audio
4. Translate it to text (In my case translate to Vietnamese)
I'm done with 1,2,3. But I'm stuck at 4, I don't know how to use Google STT API with an audio file input, because it seems like just use STT as an intent.
Hope any ideas or solution related!
Here's my code:
requestAPI() {
// Imports the Google Cloud client library
const Speech = require('#google-cloud/speech')({
projectId: 'speech-to-text-175801',
keyFilename: '/keyfile.json'
});
const RNFS = require('react-native-fs');
// Your Google Cloud Platform project ID
const projectId = 'speech-to-text-175801';
// Instantiates a client
const speechClient = Speech({
projectId: projectId,
keyFilename: '/keyfile.json'
});
// The name of the audio file to transcribe
const fileName = this.state.audioPath;
// Reads a local audio file and converts it to base64
const file = RNFS.readFile(fileName);
const audioBytes = file.toString('base64');
// The audio file's encoding, sample rate in hertz, and BCP-47 language code
const audio = {
content: audioBytes
};
const config = {
encoding: 'aac',
sampleRateHertz: 32000,
languageCode: 'vi-VN'
};
const request = {
audio: audio,
config: config
};
// Detects speech in the audio file
speechClient.recognize(request)
.then((results) => {
const transcription = results[0].results[0].alternatives[0].transcript;
this.setState({
textReceived: transcription
})
})
.catch((err) => {
this.setState({
textReceived: 'Có lỗi Google Cloud STT, mời bạn request lại'
})
});
}
Error now: 'Unable to resolve module "child_process" from ...\google-auth-library\lib\auth\googleauth.js"

I think the problem is that you can't use Google speech SDK on react native, it's a node library. So you need to first send your file to nodejs server then call the sdk methods.

You can use react-native-speech-to-text-android to perform speech to text conversion.
import SpeechToText from 'react-native-google-speech-to-text';
const speechToTextHandler = async () => {
let result = null;
try {
result = await SpeechToText.startSpeech('Try saying something', 'en_IN');
console.log('Result: ', result);
} catch (error) {
console.log('error: ', error);
}
}

Related

How do I save a document in internal/private app storage from external file system of react native app managed by expo?

I'm able to pickup the document from file system using expo-document-picker but can't figure out a way to save it. Please guide on this.
import * as DocumentPicker from 'expo-document-picker';
import * as FileSystem from 'expo-file-system';
const pickDocument = async () => {
let result = await DocumentPicker.getDocumentAsync({});
console.log(result); // was able to get result here
saveDocument(result);
};
const saveDocument = async (image) => {
const fileUri = `${FileSystem.documentDirectory}${image.name}`;
// const savedImage = await FileSystem.downloadAsync(image.uri, fileUri);
const savedImage = await FileSystem.moveAsync(image.uri, fileUri);
console.log('saved image', savedImage);
};
Read about expo file system, https://docs.expo.dev/versions/v46.0.0/sdk/filesystem/#filesystemdocumentdirectory
but couldn't find a method which helps save the document. Tried FileSystem.moveAsync but it gave below error:
Error: An exception was thrown while calling `ExponentFileSystem.moveAsync` with arguments `(
"file:///var/mobile/Containers/...

react-native (Expo) upload file on background

In my Expo (react-native) application, I want to do the upload task even if the application is in the background or killed.
the upload should be done to firebase storage, so we don't have a REST API.
checked out the Expo task manager library, but I could not figure out how it should be done. is it even possible to achieve this goal with Expo? is the TaskManager the correct package for this task?
there are only some Expo packages that could be registered as a task (e.g. backgroundFetch), and it is not possible to register a custom function (in this case uploadFile method).
I even got more confused as we should enable add UIBackgroundModes key for iOS but it only has audio,location,voip,external-accessory,bluetooth-central,bluetooth-peripheral,fetch,remote-notification,processing as possible values.
I would appreciate it if you can at least guide me on where to start or what to search for, to be able to upload the file even if the app is in the background is killed/terminated.
import { getStorage, ref, uploadBytes } from "firebase/storage";
const storage = getStorage();
const storageRef = ref(storage, 'videos');
const uploadFile = async (file)=>{
// the file is Blob object
await uploadBytes(storageRef, file);
}
I have already reviewed react-native-background-fetch, react-native-background-upload, react-native-background-job . upload should eject Expo, job does not support iOS, and fetch is a fetching task designed for doing task in intervals.
if there is a way to use mentioned libraries for my purpose, please guide me :)
to my understanding, the Firebase Cloud JSON API does not accept files, does it ? if so please give me an example. If I can make storage json API work with file upload, then I can use Expo asyncUpload probably without ejecting.
I have done something similar like you want, you can use expo-task-manager and expo-background-fetch. Here is the code as I used it. I Hope this would be useful for you.
import * as BackgroundFetch from 'expo-background-fetch';
import * as TaskManager from 'expo-task-manager';
const BACKGROUND_FETCH_TASK = 'background-fetch';
const [isRegistered, setIsRegistered] = useState(false);
const [status, setStatus] = useState(null);
//Valor para que se ejecute en IOS
BackgroundFetch.setMinimumIntervalAsync(60 * 15);
// Define the task to execute
TaskManager.defineTask(BACKGROUND_FETCH_TASK, async () => {
const now = Date.now();
console.log(`Got background fetch call at date: ${new Date(now).toISOString()}`);
// Your function or instructions you want
return BackgroundFetch.Result.NewData;
});
// Register the task in BACKGROUND_FETCH_TASK
async function registerBackgroundFetchAsync() {
return BackgroundFetch.registerTaskAsync(BACKGROUND_FETCH_TASK, {
minimumInterval: 60 * 15, // 1 minutes
stopOnTerminate: false, // android only,
startOnBoot: true, // android only
});
}
// Task Status
const checkStatusAsync = async () => {
const status = await BackgroundFetch.getStatusAsync();
const isRegistered = await TaskManager.isTaskRegisteredAsync(
BACKGROUND_FETCH_TASK
);
setStatus(status);
setIsRegistered(isRegistered);
};
// Check if the task is already register
const toggleFetchTask = async () => {
if (isRegistered) {
console.log('Task ready');
} else {
await registerBackgroundFetchAsync();
console.log('Task registered');
}
checkStatusAsync();
};
useEffect(() => {
toggleFetchTask();
}, []);
Hope this isn't too late to be helpful.
I've been dealing with a variety of expo <-> firebase storage integrations recently, and here's some info that might be helpful.
First, I'd recommend not using the uploadBytes / uploadBytesResumable methods from Firebase. This Thread has a long ongoing discussion about it, but basically it's broken in v9. Maybe in the future the Firebase team will solve the issues, but it's pretty broken with Expo right now.
Instead, I'd recommend either going down the route of writing a small Firebase function that either gives a signed-upload-url or handles the upload itself.
Basically, if you can get storage uploads to work via an http endpoint, you can get any kind of upload mechanism working. (e.g. the FileSystem.uploadAsync() method you're probably looking for here, like #brentvatne pointed out, or fetch, or axios. I'll show a basic wiring at the end).
Server Side
Option 1: Signed URL Upload.
Basically, have a small firebase function that returns a signed url. Your app calls a cloud function like /get-signed-upload-url , which returns the url, which you then use. Check out: https://cloud.google.com/storage/docs/access-control/signed-urls for how you'd go about this.
This might work well for your use case. It can be configured just like any httpsCallable function, so it's not much work to set up, compared to option 2.
However, this doesn't work for the firebase storage / functions emulator! For this reason, I don't use this method, because I like to intensively use the emulators, and they only offer a subset of all the functionalities.
Option 2: Upload the file entirely through a function
This is a little hairier, but gives you a lot more fidelity over your uploads, and will work on an emulator! I like this too because it allows doing upload process within the endpoint execution, instead of as a side effect.
For example, you can have a photo-upload endpoint generate thumbnails, and if the endpoint 201's, then you're good! Rather than the traditional Firebase approach of having a listener to cloud storage which would generate thumbnails as a side effect, which then has all kinds of bad race conditions (checking for processing completion via exponentiational backoff? Gross!)
Here are three resources I'd recommend to go about this approach:
https://cloud.google.com/functions/docs/writing/http#multipart_data
https://github.com/firebase/firebase-js-sdk/issues/5848
https://github.com/mscdex/busboy
Basically, if you can make a Firebase cloud endpoint that accepts a File within formdata, you can have busboy parse it, and then you can do anything you want with it... like upload it to Cloud Storage!
an outline of this:
import * as functions from "firebase-functions";
import * as busboy from "busboy";
import * as os from "os";
import * as path from "path";
import * as fs from "fs";
type FieldMap = {
[fieldKey: string]: string;
};
type Upload = {
filepath: string;
mimeType: string;
};
type UploadMap = {
[fileName: string]: Upload;
};
const MAX_FILE_SIZE = 2 * 1024 * 1024; // 2MB
export const uploadPhoto = functions.https.onRequest(async (req, res) => {
verifyRequest(req); // Verify parameters, auth, etc. Better yet, use a middleware system for this like express.
// This object will accumulate all the fields, keyed by their name
const fields: FieldMap = {};
// This object will accumulate all the uploaded files, keyed by their name.
const uploads: UploadMap = {};
// This will accumulator errors during the busboy process, allowing us to end early.
const errors: string[] = [];
const tmpdir = os.tmpdir();
const fileWrites: Promise<unknown>[] = [];
function cleanup() {
Object.entries(uploads).forEach(([filename, { filepath }]) => {
console.log(`unlinking: ${filename} from ${path}`);
fs.unlinkSync(filepath);
});
}
const bb = busboy({
headers: req.headers,
limits: {
files: 1,
fields: 1,
fileSize: MAX_FILE_SIZE,
},
});
bb.on("file", (name, file, info) => {
verifyFile(name, file, info); // Verify your mimeType / filename, etc.
file.on("limit", () => {
console.log("too big of file!");
});
const { filename, mimeType } = info;
// Note: os.tmpdir() points to an in-memory file system on GCF
// Thus, any files in it must fit in the instance's memory.
console.log(`Processed file ${filename}`);
const filepath = path.join(tmpdir, filename);
uploads[filename] = {
filepath,
mimeType,
};
const writeStream = fs.createWriteStream(filepath);
file.pipe(writeStream);
// File was processed by Busboy; wait for it to be written.
// Note: GCF may not persist saved files across invocations.
// Persistent files must be kept in other locations
// (such as Cloud Storage buckets).
const promise = new Promise((resolve, reject) => {
file.on("end", () => {
writeStream.end();
});
writeStream.on("finish", resolve);
writeStream.on("error", reject);
});
fileWrites.push(promise);
});
bb.on("close", async () => {
await Promise.all(fileWrites);
// Fail if errors:
if (errors.length > 0) {
functions.logger.error("Upload failed", errors);
res.status(400).send(errors.join());
} else {
try {
const upload = Object.values(uploads)[0];
if (!upload) {
functions.logger.debug("No upload found");
res.status(400).send("No file uploaded");
return;
}
const { uploadId } = await processUpload(upload, userId);
cleanup();
res.status(201).send({
uploadId,
});
} catch (error) {
cleanup();
functions.logger.error("Error processing file", error);
res.status(500).send("Error processing file");
}
}
});
bb.end(req.rawBody);
});
Then, that processUpload function can do anything you want with the file, like upload it to cloud storage:
async function processUpload({ filepath, mimeType }: Upload, userId: string) {
const fileId = uuidv4();
const bucket = admin.storage().bucket();
await bucket.upload(filepath, {
destination: `users/${userId}/${fileId}`,
{
contentType: mimeType,
},
});
return { fileId };
}
Mobile Side
Then, on the mobile side, you can interact with it like this:
async function uploadFile(uri: string) {
function getFunctionsUrl(): string {
if (USE_EMULATOR) {
const origin =
Constants?.manifest?.debuggerHost?.split(":").shift() || "localhost";
const functionsPort = 5001;
const functionsHost = `http://${origin}:${functionsPort}/{PROJECT_NAME}/${PROJECT_LOCATION}`;
return functionsHost;
} else {
return `https://{PROJECT_LOCATION}-{PROJECT_NAME}.cloudfunctions.net`;
}
}
// The url of your endpoint. Make this as smart as you want.
const url = `${getFunctionsUrl()}/uploadPhoto`;
await FileSystem.uploadAsync(uploadUrl, uri, {
httpMethod: "POST",
uploadType: FileSystem.FileSystemUploadType.MULTIPART,
fieldName: "file", // Important! make sure this matches however you want bussboy to validate the "name" field on file.
mimeType,
headers: {
"content-type": "multipart/form-data",
Authorization: `${idToken}`,
},
});
});
TLDR
Wrap Cloud Storage in your own endpoint, treat it like a normal http upload, everything plays nice.

I am converting Images to pdf using a npm library in react native why it is giving error of null object?

I am using react-native-image-to-pdf library to convert images to pdf in my react native app. from https://www.npmjs.com/package/react-native-image-to-pdf
var photoPath = ['https://images.pexels.com/photos/20787/pexels-photo.jpg?auto=compress&cs=tinysrgb&h=350','https://images.pexels.com/photos/20787/pexels-photo.jpg?auto=compress&cs=tinysrgb&h=350'];
const myAsyncPDFFunction = async () => {
try {
const options = {
imagePaths: photoPath,
name: 'PDFName',
};
const pdf = await RNImageToPdf.createPDFbyImages(options);
console.log(pdf.filePath);
} catch(e) {
console.log(e);
}
}
but this is giving error Error: Attempt to invoke virtual method 'int android.graphics.Bitmap.getWidth()' on a null object reference
I have also tried giving path as ['./assets/a.png', './assets/b.png']
but still getting same error
Based on the usage example, your photoPath needs to be a local file path and not a remote path.
My recommendation is to first use rn-fetch-blob to download the remote image to the device, and then pass your new local image path to react-native-image-to-pdf. Something like:
RNFetchBlob
.config({
// add this option that makes response data to be stored as a file,
// this is much more performant.
fileCache : true,
})
.fetch('GET', 'http://www.example.com/file/example.png', {
//some headers ..
})
.then(async (res) => {
// the temp file path
console.log('The file saved to ', res.path())
const options = {
imagePaths: [res.path()],
name: 'PDFName',
};
const pdf = await RNImageToPdf.createPDFbyImages(options);
})
from file path remove the text 'file://; with empty string('').
const options = {
imagePaths: [uri.replace('file://', '')],
name: 'FileName',
quality: .9, // optional compression paramter
};
replace('file://', '') it's work for me

How to upload recording file from Expo-av via Axios?

I am trying to upload a file recorded using expo-av via an Axios post request to my server. Forgive me, I am a Python backend dev so I am not that familiar with JS or Axios. I have tried various solutions but can't get anything working.
The code I am using for the recording is this, mostly straight from the Expo docs (the recording element) and it is working fine (the audio file gets created by the simulator, e.g. a .caf file for iOS simulator)
import * as React from 'react';
import { Text, View, StyleSheet, Button } from 'react-native';
import { Audio } from 'expo-av';
export default function App() {
const [recording, setRecording] = React.useState();
async function startRecording() {
try {
console.log('Requesting permissions..');
await Audio.requestPermissionsAsync();
await Audio.setAudioModeAsync({
allowsRecordingIOS: true,
playsInSilentModeIOS: true,
});
console.log('Starting recording..');
const { recording } = await Audio.Recording.createAsync(
Audio.RECORDING_OPTIONS_PRESET_HIGH_QUALITY
);
setRecording(recording);
console.log('Recording started');
} catch (err) {
console.error('Failed to start recording', err);
}
}
async function stopRecording() {
console.log('Stopping recording..');
setRecording(undefined);
await recording.stopAndUnloadAsync();
const uri = recording.getURI();
console.log('Recording stopped and stored at', uri);
//My code for the upload
const formData = new FormData();
formData.append("attribute_x", "123");
formData.append("attribute_y", "456");
let soundFile = //NOT SURE WHAT THE ACTUAL SOUND FILE IS. The uri is just a string
formData.append("recording", {uri: uri, name: 'soundfile.caf', type: 'audio/x-caf'});
const response = await axios.post(my_url, formData, {
headers: {
'Content-Type': 'multipart/form-data',
'X-CSRFToken': 'csrftoken',
}
})
... deal with the response...
}
return (
<View style={styles.container}>
<Button
title={recording ? 'Stop Recording' : 'Start Recording'}
onPress={recording ? stopRecording : startRecording}
/>
</View>
);
}
The request body I need to send should be:
{
"attribute_x": 333,
"attribute_y": 291,
"recording": THE SOUND FILE
}
It's .caf for iOS and will be .mp4 for Android (defaults) but I can figure out how to deal with that myself. Just really pulling my hair out now with the upload/post request.
Thanks in advance.
I was unable to get it work with react-native, main cause is that multipart/form-data header wasn't set correctly and formdata.getHeaders() isn't defined in react-native.
Tried all that was tested here, but without success: https://github.com/axios/axios/issues/318
Solution: use https://github.com/sindresorhus/ky
const uri = recording.getURI();
const filetype = uri.split(".").pop();
const filename = uri.split("/").pop();
const fd = new FormData();
fd.append("audio-record", {
uri,
type: `audio/${filetype}`,
name: filename,
});
await ky.post("audio/upload", {
body: fd,
});
the headers will set automatically like with fetch api, and the boudary part will be provided !

How to upload file to server using react-native

I am developing a app where i need to upload an image to the server. Based on the image i get a response which i need to render?.
Can you please help me how to upload an image using react-native?.
There is file uploading built into React Native.
Example from React Native code:
var photo = {
uri: uriFromCameraRoll,
type: 'image/jpeg',
name: 'photo.jpg',
};
var body = new FormData();
body.append('authToken', 'secret');
body.append('photo', photo);
body.append('title', 'A beautiful photo!');
var xhr = new XMLHttpRequest();
xhr.open('POST', serverURL);
xhr.send(body);
My solution is using fetch API and FormData.
Tested on Android.
const file = {
uri, // e.g. 'file:///path/to/file/image123.jpg'
name, // e.g. 'image123.jpg',
type // e.g. 'image/jpg'
}
const body = new FormData()
body.append('file', file)
fetch(url, {
method: 'POST',
body
})
I wrote something like that. Check out https://github.com/kamilkp/react-native-file-transfer
I have been struggling to upload images recently on react-native. I didn't seem to get the images uploaded. This is actually because i was using the react-native-debugger and network inspect on while sending the requests. Immediately i switch off network inspect, the request were successful and the files uploaded.
I am using the example from this answer above it works for me.
This article on github about the limitations of network inspect feature may clear things for you.
Just to build on the answer by Dev1, this is a good way to upload files from react native if you also want to show upload progress. It's pure JS, so this would actually work on any Javascript file.
(Note that in step #4 you have to replace the variables inside the strings with the type and file endings. That said, you could just take those fields out.)
Here's a gist I made on Github: https://gist.github.com/nandorojo/c641c176a053a9ab43462c6da1553a1b
1. for uploading one file:
// 1. initialize request
const xhr = new XMLHttpRequest();
// 2. open request
xhr.open('POST', uploadUrl);
// 3. set up callback for request
xhr.onload = () => {
const response = JSON.parse(xhr.response);
console.log(response);
// ... do something with the successful response
};
// 4. catch for request error
xhr.onerror = e => {
console.log(e, 'upload failed');
};
// 4. catch for request timeout
xhr.ontimeout = e => {
console.log(e, 'cloudinary timeout');
};
// 4. create formData to upload
const formData = new FormData();
formData.append('file', {
uri: 'some-file-path', // this is the path to your file. see Expo ImagePicker or React Native ImagePicker
type: `${type}/${fileEnding}`, // example: image/jpg
name: `upload.${fileEnding}` // example: upload.jpg
});
// 6. upload the request
xhr.send(formData);
// 7. track upload progress
if (xhr.upload) {
// track the upload progress
xhr.upload.onprogress = ({ total, loaded }) => {
const uploadProgress = (loaded / total);
console.log(uploadProgress);
};
}
2. uploading multiple files
Assuming you have an array of files you want to upload, you'd just change #4 from the code above to look like this:
// 4. create formData to upload
const arrayOfFilesToUpload = [
// ...
];
const formData = new FormData();
arrayOfFilesToUpload.forEach(file => {
formData.append('file', {
uri: file.uri, // this is the path to your file. see Expo ImagePicker or React Native ImagePicker
type: `${type}/${fileEnding}`, // example: image/jpg
name: `upload.${fileEnding}` // example: upload.jpg
});
})
In my opinion, the best way to send the file to the server is to use react-native-fs package, so install the package
with the following command
npm install react-native-fs
then create a file called file.service.js and modify it as follow:
import { uploadFiles } from "react-native-fs";
export async function sendFileToServer(files) {
return uploadFiles({
toUrl: `http://xxx/YOUR_URL`,
files: files,
method: "POST",
headers: { Accept: "application/json" },
begin: () => {
// console.log('File Uploading Started...')
},
progress: ({ totalBytesSent, totalBytesExpectedToSend }) => {
// console.log({ totalBytesSent, totalBytesExpectedToSend })
},
})
.promise.then(({ body }) => {
// Response Here...
// const data = JSON.parse(body); => You can access to body here....
})
.catch(_ => {
// console.log('Error')
})
}
NOTE: do not forget to change the URL.
NOTE: You can use this service to send any file to the server.
then call that service like the following:
var files = [{ name: "xx", filename:"xx", filepath: "xx", filetype: "xx" }];
await sendFileToServer(files)
NOTE: each object must have name,filename,filepath,filetype
A couple of potential alternatives are available. Firstly, you could use the XHR polyfill:
http://facebook.github.io/react-native/docs/network.html
Secondly, just ask the question: how would I upload a file in Obj-C? Answer that and then you could just implement a native module to call it from JavaScript.
There's some further discussion on all of this on this Github issue.
Tom's answer didn't work for me. So I implemented a native FilePickerModule which helps me choose the file and then use the remobile's react-native-file-transfer package to upload it. FilePickerModule returns the path of the selected file (FileURL) which is used by react-native-file-transfer to upload it.
Here's the code:
var FileTransfer = require('#remobile/react-native-file-transfer');
var FilePickerModule = NativeModules.FilePickerModule;
var that = this;
var fileTransfer = new FileTransfer();
FilePickerModule.chooseFile()
.then(function(fileURL){
var options = {};
options.fileKey = 'file';
options.fileName = fileURL.substr(fileURL.lastIndexOf('/')+1);
options.mimeType = 'text/plain';
var headers = {
'X-XSRF-TOKEN':that.state.token
};
options.headers = headers;
var url = "Set the URL here" ;
fileTransfer.upload(fileURL, encodeURI(url),(result)=>
{
console.log(result);
}, (error)=>{
console.log(error);
}, options);
})
Upload Files : using expo-image-picker npm module. Here we can upload any files or images etc. The files in a device can be accessed using the launchImageLibrary method. Then access the media on that device.
import * as ImagePicker from "expo-image-picker";
const loadFile = async () => {
let result = await ImagePicker.launchImageLibraryAsync({
mediaTypes: ImagePicker.MediaTypeOptions.All,
aspect: [4, 3],
});
return <Button title="Pick an image from camera roll" onPress={loadFile} />
}
The above code used to access the files on a device.
Also, use the camera to capture the image/video to upload by using
launchCameraAsync with mediaTypeOptions to videos or photos.