Multer google storage returns 'Internal Server Error' - file-upload

I have just switched image upload with Multer from local to Google Cloud Storage using 'multer-google-storage'. It used to work fine earlier, but now sends a 500 Internal Server Error without message. I am using Nodejs and Express, React for front end. FormData is formatted correctly since it works fine if I go back to local storage. Any ideas on how to fix this? Or display an error message? I am not able to find much documentation on 'multer-google-storage'. Thanks for the help!
Here the back-end post route (I hid the configuration options)
const multer = require('multer');
const multerGoogleStorage = require('multer-google-storage');
const upload = multer({
storage: multerGoogleStorage.storageEngine({
autoRetry: true,
bucket: '******',
projectId: '******',
keyFilename: '../server/config/key.json',
filename: (req, file, callback) => {
callback(null, file.originalname);
},
}),
});
//#route POST api/listings
//#description Create listing
//#access Private
router.post(
'/',
upload.any(),
[
isLoggedIn,
[
check('title', 'Title is required').not().isEmpty(),
check('coordinates').not().isEmpty(),
check('address').not().isEmpty(),
check('price', 'Set a price').not().isEmpty(),
check('description', 'Type a description').not().isEmpty(),
check('condition', 'Declare the condition').not().isEmpty(),
check('category', 'Please select a category').not().isEmpty(),
],
],
async (req, res) => {
const errors = validationResult(req);
if (!errors.isEmpty()) {
console.log('validation error');
return res.status(400).json({ errors: errors.array() });
}
try {
const files = req.files;
let images = [];
for (let image of files) {
images.push(image.originalname);
}
const newListing = new Listing({
title: req.body.title,
images: images,
coordinates: JSON.parse(req.body.coordinates),
price: req.body.price,
description: req.body.description,
condition: req.body.condition,
dimensions: req.body.dimensions,
quantity: req.body.quantity,
address: req.body.address,
author: req.user.id,
category: JSON.parse(req.body.category),
});
const author = await User.findById(req.user.id);
await author.listings.push(newListing);
await author.save();
const listing = await newListing.save();
res.json(listing);
} catch (error) {
console.log('error');
console.error(error);
res.json(error);
res.status(500).send('Server Error');
}
}
);

I have solved the issue, it was a permission problem. My previous Google Cloud Storage bucket had access control 'Uniform' while it should have been 'Fine-grained'.

Related

Cloudinary\Error: Missing required parameter - file - Express and Postman

first time trying to upload images to Cloudinary and I have come across an issue when using Express via Postman.
Using form-data on setting 'file' to upload an image to Cloudinary
As of now, when I try to access the req.body I get an empty object, so I guess that has to do with why cloudinary.uploader.upload cannot read the file passed as its first param, since its req.body.file, as shown in the code below.
cloudinary.config({
cloud_name: process.env.CLOUDINARY_CLOUD_NAME,
api_key: process.env.CLOUDINARY_KEY,
api_secret: process.env.CLOUDINARY_SECRET
})
exports.upload = async (req, res) => {
try{
console.log(req.body);
const result = await cloudinary.uploader.upload(req.body.file, {
public_id: `${Date.now()}`,
resource_type: "auto"
})
return res.json({
public_id: result.public_id,
url: result.secure_url
})
}catch(err){
console.log(err)
}
}
The error message I get:
{
message: 'Missing required parameter - file',
name: 'Error',
http_code: 400
}
Any suggestions to solve this issue?
I solved it! I was not able to pass the form-data as req.body to the server, so I had to try and access it through req.files, but was not able to with that either, so I searched a bit and found a middleware 'express-fileupload', and that did the trick. I just added it in my app.js and used
const fileupload = require('express-fileupload');
app.use(fileupload({useTempFiles: true}))
So now I can access my req.files.
exports.upload = async (req, res) => {
const file = req.files.image
try{
console.log(file);
const result = await cloudinary.uploader.upload(file.tempFilePath, {
public_id: `${Date.now()}`,
resource_type: "auto"
})
res.json({
public_id: result.public_id,
url: result.secure_url
})
}catch(err){
console.log("Error", err)
return res.status(400).json({error: err})
}
}
The response I get is:
{
name: 'some-image.png',
data: <Buffer >,
size: 99770,
encoding: '7bit',
tempFilePath: ' **C:\\filepath\some-image.png** ',
truncated: false,
mimetype: 'image/png',
md5: 'b5f612a571442bf604952fd12c47c1bf',
mv: [Function: mv]
}
POST /cloudinary/upload-images 200 1617.944 ms - 119
And it is uploaded successfully to my Cloudinary.
const result = await cloudinary.uploader.upload(req.file, {
public_id: ${Date.now()},
resource_type: "auto"
})
and add file from form data and type should be File
Solved!
This is how i am setting the FormData
let myTestForm = new FormData();
myTestForm.set("name", name);
myTestForm.set("email", email);
myTestForm.set("Avatar", Avatar);
myTestForm.set("password", password);
This is how i am using the FormData
const config = {
headers: {
"Content-type": "multipart/form-data",
},
};
const { data } = await axios.post(`/api/v1/register`, userData, { config });
please don't pass it this way { userData} , had struggled for with this :/
This is how i am uploading image
const myCloud = await cloudinary.v2.uploader.upload(req.body.Avatar, {
folder: "Avatars",
width: 150,
crop: "scale",
public_id: `${Date.now()}`,
resource_type: "auto",
});
PS : in my case i had to upload only 1 image. Have not passed any parameter in app.js file
app.use(bodyParser.urlencoded({ extended: true }));
app.use(fileUpload());

How to upload and get image / formData with react native axios and multer

I try to post images to MongoDB but get status CODE 404,
I have multer and static path '/uploads/' and this directory is on the frontend with some images, I get the array from the server but i dont know how to show them :
i tried in flat list to show them like:
<Image source={require("../../server/uploads/a3.jpg")} />//here its work.
and when i replace to {require("../../server/${item.imageurl}" iI get error
when i console log the require path I get the objects with url->> ../../server/uploads\a1.jpg
see that the / is the opposite side \ , maybe its the problem?
now for upload I tried to :
I getting object(selectedImage) when I choosing file :
"localUri": "file:/data/user/0/host.exp.exponent/cache/ExperienceData/%2540anonymous%252FLior-
3698f362-fc81-4a77-8c97-8cd1349ce090/ImagePicker/d4c4b4ec-72e6-4e20-b6eb-4727bc84d93e.jpg",
and my axios post:
if (selectedImage !== null) {
let formData = new FormData();
formData.append("image", selectedImage);
console.log(formData);
try {
const response = await indexApi.post(`/uploads`, formData);
console.log("res", response);
} catch (err) {
console.log("c", err);
}
}
};
the form data that i send look like:
FormData {
"_parts": Array [
Array [
"image",
Object {
"localUri":
"file:/data/user/0/host.exp.exponent/cache/ExperienceData/%2540anonymous%252FLior-3698f362-fc81-
4a77-8c97-8cd1349ce090/ImagePicker/d4c4b4ec-72e6-4e20-b6eb-4727bc84d93e.jpg",
},
],
],
}
and I get error 404, or 503.....
the backend parts:
image controller:
const Image = require("../models/image");
const _ = require("lodash");
exports.getImages = (req, res) => {
Image.find()
.select("_id image desc")
.then(images => {
res.json({ images });
})
.catch(err => console.log("get images errors", err));
};
//INSERT NEW IMAGE//
exports.uploadImage = (req, res) => {
const image = new Image({
image: req.file.path,
desc: req.body.desc,
});
console.log(image);
console.log(image);
console.log("ss", image);
image.save(err => {
if (err) {
return res.status(400).json({ error: "העלאת תמונה נכשלה" });
}
res.json({ message: "העלאת תמונה עברה בהצלחה" });
});
};
image route:
const express = require("express");
const { getImages, uploadImage } = require("../controllers/image");
const multer = require("multer");
//configure the images
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, "./uploads");
},
filename: function (req, file, cb) {
console.log("f", file);
cb(null, file.originalname);
},
});
const upload = multer({
storage: storage,
limits: {
fileSize: 1024 * 1024 * 55,
},
});
const router = express.Router();
router.get("/images", getImages);
router.post("/images/new", upload.single("image"), uploadImage);
module.exports = router;
so maybe my prolbem go from the route... beacuse i dont understand the that thing that i have one route (uploads) for the images, and another route with get and post method(/images) and for post (/images/new)
Edit. Error Picture for the first part
Try fixing this first:-
a. Your require might not able to read the incoming value from item.imageUrl since you called it inside double quotes (""):-
try using backticks (``):-
<Image source={require(`../../server/${item.imageUrl}`)} />
OR
You could try one of these two ways:-
saving image path in db
in imageUpload method/function:-
//INSERT NEW IMAGE//
exports.uploadImage = (req, res) => {
const image = new Image({
// don't do this
// image: req.file.path,
// instead try do this
image: '/uploads/' + req.file.originalname,
desc: req.body.desc,
});
console.log(image);
console.log(image);
console.log("ss", image);
image.save(err => {
if (err) {
return res.status(400).json({ error: "העלאת תמונה נכשלה" });
}
res.json({ message: "העלאת תמונה עברה בהצלחה" });
});
saving image originalname only in db
in imageUpload method/function:-
//INSERT NEW IMAGE//
exports.uploadImage = (req, res) => {
const image = new Image({
// don't do this
// image: req.file.path,
// instead try do this
image: req.file.originalname,
desc: req.body.desc,
});
console.log(image);
console.log(image);
console.log("ss", image);
image.save(err => {
if (err) {
return res.status(400).json({ error: "העלאת תמונה נכשלה" });
}
res.json({ message: "העלאת תמונה עברה בהצלחה" });
});
in 'view' or front-end:-
<Image source={require(`../../server/uploads/${item.imageUrl}`)} />

Getting Error 500 (general-error) while uploading files in feathersjs

I am getting this error on Postman when I send request for uploading files on Feathersjs:
{
"name": "GeneralError",
"message": "ENOENT: no such file or directory, open 'public/uploads/pic'",
"code": 500,
"className": "general-error",
"data": {},
"errors": {}
}
My uploads.service.js:
const {Uploads} = require('./uploads.class');
const createModel = require('../../models/uploads.model');
const hooks = require('./uploads.hooks');
const multer = require('multer');
const storage = multer.diskStorage({
destination: (_req, _file, cb) => cb(null, 'public/uploads'), // where the files are being stored
filename: (_req, file, cb) => {
console.log(_req.body);
//cb(null, ${_req.body.name});
cb(null, `${_req.body.name}`); //
}, // getting the file name
});
const uploads = multer({
storage,
limits: {
fieldSize: 1e8,
fileSize: 1e7,
},
});
module.exports = function(app) {
const options = {
Model: createModel(app),
paginate: app.get('paginate'),
multi: true,
};
// Initialize our service with any options it requires
app.use(
'/uploads',
uploads.array('files'),
(req, _res, next) => {
const {method} = req;
if (method === 'POST' || method === 'PATCH') {
console.log(req.files);
console.log(req.body);
req.feathers.files = req.body.files;
const body = [];
for (const file of req.files)
body.push({
name: req.body.name,
newNameWithPath: file.path,
});
req.body = method === 'POST' ? body : body[0];
}
next();
},
new Uploads(options, app),
);
// Get our initialized service so that we can register hooks
const service = app.service('uploads');
service.hooks(hooks);
};
This is my uploads.model.js:
module.exports = function(app) {
const modelName = 'uploads';
const mongooseClient = app.get('mongooseClient');
const {Schema} = mongooseClient;
const schema = new Schema(
{
name: {type: String, required: true},
},
{
timestamps: true,
},
);
// This is necessary to avoid model compilation errors in watch mode
// see https://mongoosejs.com/docs/api/connection.html#connection_Connection-deleteModel
if (mongooseClient.modelNames().includes(modelName)) {
mongooseClient.deleteModel(modelName);
}
return mongooseClient.model(modelName, schema);
};
I really cannot figure out where exactly the problem is. According to me it is supposed to make the folder automatically when I upload the file.
I would really appreciate some help. Thank you in advance.
It was my own mistake. I made the uploads folder inside public folder myself and now it's working.

How can I speed up S3 signedUrl upload from 1Mbps

I am currently utilizing s3 signedUrl's in order to hide my credentials from users on the front end. I have it set up and working but it is extremely slow around 1.2mb/s. Using speed test my wifi is showing 11.9mb/s so I don't believe it is my network. The image is only 8MB in size that I have been testing.
Server
const { uploadFile } = require("../services/aws");
app.post("/activity/image-upload", async (req, res) => {
try {
const { _projectId, name, type } = req.body;
const key = `${_projectId}/activities/${name}`;
const signedUrl = await uploadFile({ key, type });
res.status(200).send(signedUrl);
} catch (err) {
console.log("/activity/upload-image err", err);
res.status(422).send();
}
});
AWS Service
const aws = require("aws-sdk");
const keys = require("../config/keys");
aws.config.update({
accessKeyId: keys.aws.accessKeyId,
secretAccessKey: keys.aws.secretAccessKey,
useAccelerateEndpoint: true,
signatureVersion: "v4",
region: "my-region",
});
const s3 = new aws.S3();
exports.uploadFile = async ({ type, key }) => {
try {
const awsUrl = await s3.getSignedUrl("putObject", {
Bucket: keys.aws.bucket,
ContentType: type,
Key: key,
ACL: "public-read",
});
return awsUrl;
} catch (err) {
throw err;
}
};
Front End
const handleUpload = async ({ file, onSuccess, onProgress }) => {
try {
const res = await api.post("/activity/image-upload", {
type: file.type,
name: file.name,
_projectId,
});
const upload = await axios.put(res.data, file, {
headers: {
"Content-Type": file.type,
},
onUploadProgress: handleProgressChange,
});
} catch (err) {
console.log("err", err);
}
};
Image of Request Speeds
You can see above the call to image-upload is returning in 63ms so the hang up isn't my server getting the signedURL. The axios PUT request to s3 signedURL is 6.37s. Unless I am horrible at math that for the 8MB file I am uploading that is roughly 1.2mb/s. What am I missing?
Update 7/23
Here is a picture of my speed test through Google showing upload speed of 10.8mbs.
I tried uploading the image in s3 console to compare speeds. When I uploaded it through the s3 console it was 10.11s!!! Are their different plans that throttle speeds? I am even utilizing Transfer Acceleration and its this slow.

Unexpected end of multipart data nodejs multer s3

iam trying to upload image in s3 this is my code
const upload = require('../services/file_upload');
const singleUpload = upload.single('image');
module.exports.uploadImage = (req,res) => {
singleUpload(req, res, function (err) {
if (err) {
console.log(err);
return res.status(401).send({ errors: [{ title: 'File Upload Error', detail: err}] });
}
console.log(res);
return res.json({ 'imageUrl': req.file.location });
});
}
FileUpload.js
const aws = require('aws-sdk');
const multer = require('multer');
const multerS3 = require('multer-s3');
const s3 = new aws.S3();
const fileFilter = (req, file, cb) => {
if (file.mimetype === 'image/jpeg' || file.mimetype === 'image/png') {
cb(null, true)
} else {
cb(new Error('Invalid Mime Type, only JPEG and PNG'), false);
}
}
const upload = multer({
fileFilter,
storage: multerS3({
s3,
bucket: 'image-bucket',
acl: 'public-read',
contentType: multerS3.AUTO_CONTENT_TYPE,
metadata: function (req, file, cb) {
cb(null, {fieldName: 'TESTING_META_DATA!'});
},
key: function (req, file, cb) {
cb(null,"category_"+Date.now().toString()+".png")
}
})
})
module.exports = upload;
I tried to test api with postmanin serverless local it is giving this error
Error: Unexpected end of multipart data
at D:\Flutter\aws\mishpix_web\node_modules\dicer\lib\Dicer.js:62:28
at process._tickCallback (internal/process/next_tick.js:61:11) storageErrors: []
After deploying online. i tried the api. the file is uploaded in server but its a broken
Are you using aws-serverless-express. aws-serverless-express will forward original request to Buffer as utf8 encoding. So multipart data will be lost or error. I am not sure why.
So, I change aws-serverless-express to aws-serverless-express-binary and everything worked.
yarn add aws-serverless-express-binary
Hope this help!