Expo: Anyway of adding Speech to Text? - react-native

I want to include speech-to-text in my Expo app.
There are api's available such as google's speech to text and watson etc...
Has anyone come up with a solution or has any advice on how to include Speech-to-Text in their Expo or React-Native application?
I have looked at various github repos that provide Speech-to-Text for React-Native applications but they do not look production ready, and are strictly React-Native solutions as you need access to Java/Swift code.
I am not disinclined towards that if that is the only option but would prefer a Expo solution if possible.
Regards,
Emir

If you want to implement Speech-To-Text on expo then you need to create an api and deploy it else you need to detach the project and use the library react-native-google-speech-api
This is what i have implemented using google app engine, google cloud storage and google speech to text.
const format = require('util').format;
const fs = require('fs');
const express = require('express');
const multer = require('multer');
const requestHttp = require('request');
const {Storage} = require('#google-cloud/storage');
// Instantiate a storage client
const storage = new Storage();
// const upload = multer();
const app = express();
// Imports the Google Cloud client library
const speech = require('#google-cloud/speech');
// Creates a client
const client = new speech.SpeechClient();
/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
const encoding = 'LINEAR16';
const sampleRateHertz = 16000;
const languageCode = 'en-US';
const upload = multer({
storage: multer.memoryStorage(),
limits: {
fileSize: 5 * 1024 * 1024, // no larger than 5mb, you can change as needed.
},
});
const bucket = storage.bucket(process.env.GCLOUD_STORAGE_BUCKET);
app.post('/upload', upload.single('file') , async (req, res) => {
const file = await req.file
if (!file) {
const error = new Error('Please upload a file')
error.httpStatusCode = 400
return next(error)
}
// Create a new blob in the bucket and upload the file data.
const blob = bucket.file(req.file.originalname);
const blobStream = blob.createWriteStream({
resumable: false,
});
blobStream.on('error', err => {
next(err);
});
blobStream.on('finish', async () => {
// The public URL can be used to directly access the file via HTTP.
const publicUrl = await format(
`https://storage.googleapis.com/${bucket.name}/${blob.name}`
);
const request = {
config: {
encoding: encoding,
sampleRateHertz: sampleRateHertz,
languageCode: languageCode,
},
audio: {
uri: 'gs://YOUR-Bucket-Name/File-name.ext'
}
};
// Stream the audio to the Google Cloud Speech API
const [response] = await client.recognize(request);
const transcription = response.results
.map(result => result.alternatives[0].transcript)
.join('\n');
console.log(`Transcription: `, transcription);
res.status(200)
.send({
success: 'true',
message: 'Text retrieved successfully',
text: transcription
})
.end();
});
blobStream.end(req.file.buffer);
});
const PORT = process.env.PORT || 8080;
app.listen(PORT, () => {
console.log(`App listening on port ${PORT}`);
console.log('Press Ctrl+C to quit.');
});

Related

Shopify App hosted on fly.io oAuth/Auth callbacks failing

Currently I am trying to deploy my Shopify Custom App to Fly.io. Installing this app is succeeding on my development store but I get an error right after with the oAuth callback with status code 400. This is the URL it shows upon installing:
https://appname.fly.dev/api/auth/callback?code=71bfdaadd63b87eb72d9d3dc516ea1ea&hmac=1efd4ff63ebca8f28c733f464ded354ba2f0995aeb1910114e0139eaefd4cce3&host=YWRtaW4uc2hvcGlmeS5jb20vc3RvcmUvc2hvb3B5bG9vcHkx&shop=shoopyloopy1.myshopify.com&state=920113322594675&timestamp=1676563785
With text in body: Invalid OAuth callback.
The app works with all the callbacks working with a ngrok tunnel during development. Just not when deployed to fly.io. The apps frontend also works after deployment to fly.io, but all the api and auth callbacks fail to work. I get the following response on those API calls:
On performing API calls on the /api/ route I get the following error in the return of the api call:
Failed to parse session token 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczpcL1wvc2hvb3B5bG9vcHkxLm15c2hvcGlmeS5jb21cL2FkbWluIiwiZGVzdCI6Imh0dHBzOlwvXC9zaG9vcHlsb29weTEubXlzaG9waWZ5LmNvbSIsImF1ZCI6IjU4YTAzZjkwZTk4Yjc5NGRlZmE5NDZlMWZiNmVlMzRiIiwic3ViIjoiNzQ3NzAxNTM2NjEiLCJleHAiOjE2NzY1NjQyMjYsIm5iZiI6MTY3NjU2NDE2NiwiaWF0IjoxNjc2NTY0MTY2LCJqdGkiOiI0OTcyNDEwOC0zNWQ2LTRjODEtOWJkNS0wZWRkMWM4MWIxMDYiLCJzaWQiOiIxOGZmZjg5NTMyZGRiODdiOWQ3OTBhYmY1M2EwOTZiMDNkNmE4ZWU1ZTA0ZmRjZmFmOWUxOWM2OGQxZGFjN2Q2In0.XeuA5W95YjjVLZYOvmRJ9a90xpPNEukhNQ1_z4Kw_xA': signature verification failed
My fly.toml file:
app = "appname"
kill_signal = "SIGINT"
kill_timeout = 5
processes = []
[env]
PORT = "8081"
HOST = "https://appname.fly.dev"
SHOPIFY_API_KEY = "58a03f90e98b794defa946e1fb6ee34b"
SCOPES = "write_products,read_script_tags,write_script_tags"
[experimental]
auto_rollback = true
[[services]]
http_checks = []
internal_port = 8081
processes = ["app"]
protocol = "tcp"
script_checks = []
[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"
[[services.ports]]
force_https = true
handlers = ["http"]
port = 80
[[services.ports]]
handlers = ["tls", "http"]
port = 443
[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 0
timeout = "2s"
My index.js file starting up express:
// #ts-check
import { join } from "path";
import { readFileSync } from "fs";
import express from "express";
import serveStatic from "serve-static";
import shopify from "./shopify.js";
import productCreator from "./product-creator.js";
import GDPRWebhookHandlers from "./gdpr.js";
import {addScriptTag, deleteScriptTag, getProductInfo, getProductsFromIds, getScriptTags} from "./graph-functions.js";
const PORT = parseInt(process.env.BACKEND_PORT || process.env.PORT, 10);
const STATIC_PATH =
process.env.NODE_ENV === "production"
? `${process.cwd()}/frontend/dist`
: `${process.cwd()}/frontend/`;
const app = express();
// Set up Shopify authentication and webhook handling
app.get(shopify.config.auth.path, shopify.auth.begin());
app.get(
shopify.config.auth.callbackPath,
shopify.auth.callback(),
shopify.redirectToShopifyOrAppRoot()
);
app.post(
shopify.config.webhooks.path,
shopify.processWebhooks({ webhookHandlers: GDPRWebhookHandlers })
);
// All endpoints after this point will require an active session
app.use("/api/*", shopify.validateAuthenticatedSession());
app.use(express.json());
app.get("/api/products/count", async (_req, res) => {
const countData = await shopify.api.rest.Product.count({
session: res.locals.shopify.session,
});
res.status(200).send(countData);
});
app.get("/api/get-product", async (_req, res) => {
const products = await getProductInfo(res.locals.shopify.session, _req.query.id);
res.status(200).send(products);
});
app.get("/api/get-products", async (_req, res) => {
const products = await getProductsFromIds(res.locals.shopify.session,_req.query.ids);
res.status(200).send(products);
});
app.get("/api/add-script", async (_req, res) => {
const data = await addScriptTag(res.locals.shopify.session,_req.query.src);
console.log(_req.query.ids);
res.status(200).send(data);
});
app.get("/api/get-scripts", async (_req, res) => {
const data = await getScriptTags(res.locals.shopify.session);
res.status(200).send(data?.body?.data ? data?.body?.data : data);
});
app.get("/api/delete-script", async (_req, res) => {
const data = await deleteScriptTag(res.locals.shopify.session,_req.query.id);
res.status(200).send(data);
});
app.get("/api/products/create", async (_req, res) => {
let status = 200;
let error = null;
try {
await productCreator(res.locals.shopify.session);
} catch (e) {
console.log(`Failed to process products/create: ${e.message}`);
status = 500;
error = e.message;
}
res.status(status).send({ success: status === 200, error });
});
app.use(serveStatic(STATIC_PATH, { index: false }));
app.use("/*", shopify.ensureInstalledOnShop(), async (_req, res, _next) => {
return res
.status(200)
.set("Content-Type", "text/html")
.send(readFileSync(join(STATIC_PATH, "index.html")));
});
app.listen(PORT);
Any help would be highly appreciated.
I followed the official documentation: Shopify Official Docs
The Dockerfile has the same port 8081 as assigned to in the fly.toml file.
Edit (Added shopify app implementation with Database):
import { LATEST_API_VERSION } from "#shopify/shopify-api";
import { shopifyApp } from "#shopify/shopify-app-express";
import { SQLiteSessionStorage } from "#shopify/shopify-app-session-storage-sqlite";
import { restResources } from "#shopify/shopify-api/rest/admin/2023-01";
const DB_PATH = `${process.cwd()}/database.sqlite`;
const shopify = shopifyApp({
api: {
apiVersion: LATEST_API_VERSION,
restResources,
billing: undefined, // or replace with billingConfig above to enable example billing
},
auth: {
path: "/api/auth",
callbackPath: "/api/auth/callback",
},
webhooks: {
path: "/api/webhooks",
},
// This should be replaced with your preferred storage strategy
sessionStorage: new SQLiteSessionStorage(DB_PATH),
});
export default shopify;

Can't seem to get req.body in Express when embedded in Zoho Catalyst framework

My Zoho Catalyst framework isn't passing the request.body. Here is the code.
module.exports = (req, res1) => {
const debug = require('debug');
const https = require('https');
const tools = require('./tools.js');
const crypto = require('crypto');
const express = require('express');
const app = express();
app.use(express.json())
app.use(express.urlencoded({ extended: true }));
app.use(express.text());
function getHash(){
var hmac = crypto.createHmac('sha256', apisecretkey);
hmac.update(dataToSign);
return hmac.digest('base64');
};
var url = req.url;
switch (url) {
case '/scanName':
//var s = JSON.stringify(req.body)
console.log(req.body)
console.log(req.get('Accept'))
console.log(req.accepts('application/json'));
res1.write('xx')
res1.end()
break;
case '/':
Here is the output:
undefined
*/*
application/json
I've tried every form of POST from Postman that I can think of, and still nothing.
You have bound the express app variable to recognize the incoming data of the following types JSON, urlencoded and text. But you haven’t used that app variable to get the incoming request so technically it is like you have declared it but never used it. So, your function code couldn’t be able to identify the type of incoming data in the request body. You can modify your code like below:
'use strict';
const debug = require('debug');
const https = require('https');
const tools = require('./tools.js');
const crypto = require('crypto');
const express = require('express');
const app = express();
app.use(express.json());
app.use(express.text());
app.use(express.urlencoded({ extended: true }));
app.post('/scanName',async(req, res) => {
try {
let body = req.body;
console.log(body);
res.status(200).send(body);
} catch (error) {
console.log(error);
res.status(500).send(error);
}
});
module.exports = app;
You have to export the express app variable to make the endpoints accessible. You can check out this tutorial where we have shown an example of how to get and send data in the Catalyst Advanced IO function using Express.js.

React native zoomclone app, how to connect different devices via reactapp and able to render other devices streams

how to connect other devices via socketio and webrtc in a react native app. I have tried using API_URL = my pc's ip address || 5000, can anyone tell me how to connect other android devices after installing the react native app. I am done with the coding part, but in node server I am not able to see any user connected, or mediadetails, or roomID or deviceID? I am using nodejs as backend.
I have tried creating peerServer =new Peer(undefined,{host:"/",port:443}) object inside async function.
I am not able to get massage connected client on the server side. i.e
socket.on('connection', () => console.log('connected client')); this line of code is not able to execute correctly.
server.js code
const express = require('express');
const http = require('http');
const socketio = require('socket.io');
const morgan = require('morgan');
const {ExpressPeerServer} = require('peer');
const { join } = require('path');
const app = express();
const server = http.createServer(app)
const io = socketio(server).sockets;
// Borderparser
app.use(express.json());
const customGenerationFunction = () =>
(Math.random().toString(36) + "00000000000000000").substr(2,16);
const peerServer = ExpressPeerServer(server, {
debug: true,
path:'/',
generateClientId: customGenerationFunction,
});
app.use("/mypeer", peerServer);
io.on('connection', function (socket){
socket.on('join-room', ({roomID, userId}) => {
socket.join(roomID);
socket.to(roomID).broadcast.emit("user-connected", userId);
});
});
const port = process.env.PORT || 5000;
server.listen(port, () => console.log(`Server is running on port
${port}`))
--------------------video-action.js codes-------------------------------
import IO from 'socket.io-client';
import Peer from 'react-native-peerjs';
import AsyncStorage from '#react-native-async-storage/async-storage';
import {mediaDevices} from 'react-native-webrtc';
import {ADD_STREAM, MY_STREAM, ADD_REMOTE_STREAM} from './types';
//* api uri*/
export const API_URI = `http://192.168.01.36:5000`;
//**socket config */
export const socket = IO(`${API_URI}`,{
forceNew: true
})
peerServer =new Peer(undefined,
{
host:'192.168.216.202',
secure: false,
port:5000,
path: '/mypeer',
})
peerServer.on('error', console.log);
socket.on('connection', () => console.log('connected client'));
export const joinRoom = (stream) => async (dispatch) => {
const roomID = 'ahdbflarlkhnlfkjvaerbjhfbjds';
//set my own stream
dispatch({type: MY_STREAM, payload: stream});
//open a connection to our server
peerServer.on('open', (userId) => {
socket.emit('join-room', {userId, roomID});
});
socket.on('user-connected', (userId) => {
connectToNewUser(userId, stream, dispatch)
});
// recieve a call
peerServer.on('call', (call) => {
call.answer(stream);
// stream back the call
call.on('stream', (stream) => {
dispatch({type: ADD_STREAM, payload: stream});
});
});
};
function connectToNewUser(userId, stream, dispatch) {
const call = peerServer.call(userId, stream);
}

React Native: Failed to execute 'append' on 'FormData': parameter 2 is not of type 'Blob'. at new ApolloError

I am trying to upload image from my react native app to graphql by using Apollo client with createUploadLink(). When I am trying to mutate data by passing a ReactNativeFile as a variable, then it says
"network request failed: Failed to execute 'append' on 'FormData': parameter 2 is not of type 'Blob'. at new ApolloError ".
This this the mutation which i am trying to use
mutation publishPost(
$content: String!
$LocationInput: LocationInput!
$InputPostAttachment: [InputPostAttachment!]
) {
publishPost(
content: $content
location: $LocationInput
attachments: $InputPostAttachment
) {
content
}
}
InputPostAttachment has type
type InputPostAttachment {
type: PostAttachmentType!
file: Upload!
}
Apollo client settings and i am using apollo-upload-client
const httpLink = createUploadLink({
uri: 'http://localhost:8000/graphql',
});
const authLink = setContext(async (headers: any) => {
const token = await getToken();
return {
...headers,
headers: {
authorization: token ? `Bearer ${token}` : null,
},
};
});
const link = authLink.concat(httpLink);
// create an inmemory cache instance for caching graphql data
const cache = new InMemoryCache();
// instantiate apollo client with apollo link instance and cache instance
export const client = new ApolloClient({
link,
cache,
});
File upload Function and i am using react-native-image-crop-picker for multi image selection
const [image, setimage] = useState([]);
const _pickImage = () => {
ImagePicker.openPicker({
includeBase64: true,
multiple: true,
}).then((images: any) => {
let imageData: any = [];
images.map((data: any) => {
const file = new ReactNativeFile({
uri: data.path,
name: data.filename,
type: data.mime,
});
imageData.push({
type: 'IMAGE',
file: file,
});
});
setimage(imageData);
console.log(images);
});
};
const handlePost = async () => {
const InputPostAttachment: any = [...image];
const LocationInput = {
place: place,
vicinity: vicinity,
province: province,
};
publishPost({variables: {content, LocationInput, InputPostAttachment}})
.then(({data}) => {
console.log(data);
props.navigation.navigate('Home');
})
.catch((err) => {
console.log('err happened');
console.log(err);
});
};
could someone please help me out from this?
In addition to the chrome debugger issue, this error also happens on the expo web.
To anyone uploading images on expo web (or react-native web), here's a working solution:
/** Load image from camera/roll. */
const result = await ImagePicker.launchImageLibraryAsync({
mediaTypes: ImagePicker.MediaTypeOptions.All,
allowsEditing: true,
quality: 1,
});
if (result.cancelled) {
return;
}
/** web platform: blob. */
const convertBase64ToBlob = async (base64) => {
const response = await fetch(base64);
const blob = await response.blob();
return blob;
};
/** android/ios platform: ReactNativeFile.*/
const createReactNativeFile = (uri) => {
const file = new ReactNativeFile({
uri,
type: mime.lookup(uri) || 'image',
name: `file-${Date.now()}`,
});
return file;
};
/** Use blob for web, ReactNativeFile otherwise. */
const file = Platform.OS === 'web'
? await convertBase64ToBlob(result.uri)
: createReactNativeFile(result.uri);
/** Upload image with apollo. */
mutate({ variables: { file } });
On the web platform, ImagePicker returns a base64 value instead of a file path. This problem doesn't happen if the platform is Android or iOS, as ImagePicker returns a file path, which is expected by apollo-upload-client.
The solution is to detect if the URI is base64 (which happens when the platform is "web") and convert it to a blob.
My apollo-client was configured using apollo-boost and i was using chrome debugger to intercept the network was causing me this issue.
To be more specific I was using the below code to get the network requests sent by my app in the chrome debugger
global.XMLHttpRequest =
global.originalXMLHttpRequest || global.XMLHttpRequest;
global.FormData = global.originalFormData || global.FormData;
if (window.FETCH_SUPPORT) {
window.FETCH_SUPPORT.blob = false;
} else {
global.Blob = global.originalBlob || global.Blob;
global.FileReader = global.originalFileReader || global.FileReader;
}
apollo-upload-client wont send the data in multipart data if we are using chrome debugger. We will face network issue.This issue has the answer. or I had not removed apollo-boost and some part of my app was using it that was also a issue.

Sending Images Using React Native

I am trying to add sending images feature in my react native mobile application using base64. Is tha essential thing to use following all prameters when use base64 ?fs.writeFile(./uploads/${req.body.imageName}.${req.body.imageExt}, binaryData, err =
//const Order = require("../models/image.model.js");
const mongoose = require("mongoose");
const fs = require('fs')
exports.submit = (req, res) => {
const binaryData = new Buffer(req.body.image, 'base64');
fs.writeFile(`./uploads/${req.body.imageName}.${req.body.imageExt}`, binaryData, err => {
if(err) {
console.log(err)
res.status(400).json({
message: "Couldn't upload"
})
} else {
// Image saved
res.status(200).json({
message: "Success"
})
/*
Use GCP bucket
Upload -> callback -> fetchUrl
*/
}
})
};