How to post a file to Amazon S3 using node and react and save it path to mongoDB. with mongoose and formidable.
private async storeFile(file: { buffer: Buffer, fileId: string }): Promise<string> {
try {
const awsConfig = new AWS.Config(storageConfig);
const s3 = new AWS.S3(awsConfig);
let storageLink = undefined;
fs.readFile(file.buffer, (err, data) => {
if (err) {
throw err;
}
const params = {
Bucket:storageConfig.s3Bucket,
Key: `${storageConfig.s3Prefix}${file.fileId}`,
Body: data,
};
s3.upload(params, (s3Err: Error, s3Data: AWS.S3.ManagedUpload.SendData) => {
if (s3Err) {
throw s3Err;
}
storageLink = s3Data.Location;
});
});
return storageLink;
} catch (error) {
throw error;
}
}
In your Service file where you wanna call this function, update with record in collection
const storageLink = this.storeFile({ buffer, fileId });
const file = await file.updateOne({ _id: fileId }, {
status: fileStatus.UPLOADED, // just a flag
fileId: storageLink,
});
Related
I currently use multer middleware like below
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, "public");
},
filename: function (req, file, cb) {
cb(null, req.params.id + "_" + file.originalname);
},
});
export const multerUploadSingle = (req: Request, res: Response, next: NextFunction) => {
const upload = multer({ storage: storage }).single("file");
upload(req, res, (error: unknown) => {
if (error instanceof multer.MulterError) {
const message = `file upload fail: ${error.message}`;
next(new HttpException(message, HttpStatus.BadRequest));
} else if (error instanceof Error) {
const message = `file upload fail: ${error.message}`;
next(new HttpException(message, HttpStatus.InternalServerError));
} else {
// upload success
next();
}
});
}
and use in router like this
FileRouter.post("/upload/:id", multerUploadSingle, (req, res) => {...});
However, I felt I want to refactor this middleware in class, and rewrote the code like this,
export class Multer {
private readonly storage: multer.StorageEngine;
constructor() {
this.storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, "public");
},
filename: function (req, file, cb) {
cb(null, req.params.id + "_" + file.originalname);
},
});
}
uploadSingle(req: Request, res: Response, next: NextFunction) {
const upload = multer({ storage: this.storage }).single("file");
upload(req, res, (error: unknown) => {
if (error instanceof multer.MulterError) {
const message = `file upload fail: ${error.message}`;
next(new HttpException(message, HttpStatus.BadRequest));
} else if (error instanceof Error) {
const message = `file upload fail: ${error.message}`;
next(new HttpException(message, HttpStatus.InternalServerError));
} else {
// upload success
next();
}
});
}
}
const multer = new Multer();
FileRouter.post("/upload/:id", multer.uploadSingle, (req, res) => {...});
With my short knowledge, I think both case should have the same result, but the latter case which uses class made middleware doesn't work at all. It's seems method "uploadSingle" is never called, thus multer not uploading the file.
Did I make any mistake with class usage? or is it just express can only use function defined middleware?
Your code should follow the MVC pattern.
You can do stuff like this:
routerFile.js
const upload = require("../../configs/multer");
const postController = require("../../controllers/postController");
const multiUploadEvent = upload.fields([
{ name: "images", maxCount: 2 },
{ name: "video", maxCount: 2 }
]);
router.post("/add-event-post", multiUploadEvent, postController.addEventPost);
module.exports = router;
multer.js
const multer = require('multer');
const multerFilter = (req, file, cb) => {
console.log("Mime type :", file.mimetype.split('/')[0]);
if (file.mimetype.split('/')[0] === 'image' || file.mimetype.split('/')[0] === 'video' || file.mimetype.split('/')[0] === 'audio') {
cb(null, true);
} else {
cb(new Error('Please upload img, audio, or video file only.'), false);
}
};
const storage = multer.memoryStorage();
const upload = multer({
storage: storage,
fileFilter: multerFilter,
limits: {
fileSize: , 50 * 1024 * 1024// 50 Mb
},
});
module.exports = upload;
postController.js
const addEventPost = async (request, response) => {
try {
let { title, ..... } = request.body;
const images = request.files.images;
const video = request.files.video;
console.log(title);
console.log(images);
console.log(videos);
//upload to services likes aws and save to database
.
.
.
return response
.status(200)
.json({
message: "Event post added successfully"
});
} catch (error) {
console.log(error);
response.status(500).json({
error: "Something went wrong",
});
}
}
I am trying to upload a lot of files from S3 to IPFS via Pinata. I haven't found in Pinata documentation something like that.
This is my solution, using the form-data library. I haven't tested it yet (I will do it soon, I need to code some things).
Is it a correct approach? anyone who has done something similar?
async uploadImagesFolder(
items: ItemDocument[],
bucket?: string,
path?: string,
) {
try {
const form = new FormData();
for (const item of items) {
const file = getObjectStream(item.tokenURI, bucket, path);
form.append('file', file, {
filename: item.tokenURI,
});
}
console.log(`Uploading files to IPFS`);
const pinataOptions: PinataOptions = {
cidVersion: 1,
};
const result = await pinata.pinFileToIPFS(form, {
pinataOptions,
});
console.log(`Piñata Response:`, JSON.stringify(result, null, 2));
return result.IpfsHash;
} catch (e) {
console.error(e);
}
}
I had the same problem
So, I have found this: https://medium.com/pinata/stream-files-from-aws-s3-to-ipfs-a0e23ffb7ae5
But in the article If am not wrong, is used a different version to the JavaScript AWS SDK v3 (nowadays the most recent: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/index.html).
This is for the Client side with TypeScript:
If you have this version, for me works this code snippet:
export const getStreamObjectInAwsS3 = async (data: YourParamsType) => {
try {
const BUCKET = data.bucketTarget
const KEY = data.key
const client = new S3Client({
region: 'your-region',
credentials: {
accessKeyId: 'your-access-key',
secretAccessKey: 'secret-key'
}
})
const resource = await client.send(new GetObjectCommand({
Bucket: BUCKET,
Key: KEY
}))
const response = resource.Body
if (response) {
return new Response(await response.transformToByteArray()).blob()
}
return null
} catch (error) {
return null
}
}
With the previous code, you can get the Blob Object for pass it to the File object with this method and get the URL resource using the API:
export const uploadFileToIPFS = async(file: Response) => {
const url = `https://api.pinata.cloud/pinning/pinFileToIPFS`
const data = new FormData()
data.append('file', file)
try {
const response = await axios.post(url, data, {
maxBodyLength: Infinity,
headers: {
pinata_api_key: 'your-api',
pinata_secret_api_key: 'your-secret'
},
data: data
})
return {
success: true,
pinataURL: `https://gateway.pinata.cloud/ipfs/${ response.data.IpfsHash }`
}
} catch (error) {
console.log(error)
return null
}
}
I have found this solution from this nice article and you can explore other implementations (including the Node.js side)
I'm using aws-sdk to upload images to DigitalOceans bucket. On localhost it works 100% but production seems like the function goes on without an error but the file does not upload to the bucket.
I cannot figure out what is going on and can't think of a way to debug this. tried aswell executing the POST request with Postman multipart/form-data + adding file to the body of the request and it is the same for localhost, working, and production is not.
my api endpoint:
import AWS from 'aws-sdk'
import formidable from "formidable"
import fs from 'fs'
const s3Client = new AWS.S3({
endpoint: process.env.DO_SPACES_URL,
region: 'fra1',
credentials: {
accessKeyId: process.env.DO_SPACES_KEY,
secretAccessKey: process.env.DO_SPACES_SECRET
}
})
export const config = {
api: {
bodyParser: false
}
}
export default async function uploadFile(req, res) {
const { method } = req
const form = formidable()
const now = new Date()
const fileGenericName = `${now.getTime()}`
const allowedFileTypes = ['jpg', 'jpeg', 'png', 'webp']
switch (method) {
case "POST":
try {
form.parse(req, async (err, fields, files) => {
const fileType = files.file?.originalFilename?.split('.').pop().toLowerCase()
if (!files.file) {
return res.status(400).json({
status: 400,
message: 'no files'
})
}
if (allowedFileTypes.indexOf(fileType) === -1) {
return res.status(400).json({
message: 'bad file type'
})
}
const fileName = `${fileGenericName}.${fileType}`
try {
s3Client.putObject({
Bucket: process.env.DO_SPACES_BUCKET,
Key: `${fileName}`,
Body: fs.createReadStream(files.file.filepath),
ACL: "public-read"
}, (err, data) => {
console.log(err)
console.log(data)
})
const url = `${process.env.FILE_URL}/${fileName}`
return res.status(200).json({ url })
} catch (error) {
console.log(error)
throw new Error('Error Occured While Uploading File')
}
});
return res.status(200)
} catch (error) {
console.log(error)
return res.status(500).end()
}
default:
return res.status(405).end('Method is not allowed')
}
}
I am trying to display the image from firebase storage. Below is the file location copied from firebase storage. It is a jpeg file
profile/2186uPKjgo4pMOQNm0Cm/profilepic
My following code returned error.
useEffect(() => {
function geturl(){
const filename = "profile/"+userid+"/profilepic.jpeg";
var ref = firebase.storage().ref(filename);
console.log(filename);
// This returns the exact file name
ref.getDownloadURL().then((url)=> {
console.log(url);
});
}
geturl();
}, []);
I got this error [object Object]. After that, I tried the following code async await
useEffect(() => {
async function geturl(){
const filename = "profile/"+userid+"/profilepic.jpeg";
var ref = firebase.storage().ref(filename);
console.log("inside geturl");
const downloadurl = await ref.getDownloadURL();
console.log(downloadurl);
}
geturl();
}, []);
Now Im getting the following error.
Possible Unhandled Promise Rejection (id: 29):
"code_": "storage/object-not-found",
"message_": "Firebase Storage: Object 'profile/2186uPKjgo4pMOQNm0Cm/profilepic.jpeg' does not exist.",
"name_": "FirebaseError",
"serverResponse_": "{
\"error\": {
\"code\": 404,
\"message\": \"Not Found. Could not get object\",
\"status\": \"GET_OBJECT\"
}
}",
}
Please let me know how I can get the url?
here you go you can use this function it uploads image to firebase storage and get the image uri at the same time
const uploadImage = async () => {
const response = await fetch(image);
const blob = await response.blob();
let filename = image.substring(image.lastIndexOf('/')+1);
const ext = filename.split('.').pop();
const name = filename.split('.').slice(0, -1).join('.');
filename = name + Date.now() + '.' + ext;
try {
var ref = firebase.storage().ref().child('post-images/'+filename);
await ref.put(blob)
.then(snapshot => {
return snapshot.ref.getDownloadURL();
})
.then(downloadURL => {
console.log(`Successfully uploaded file and got download link');
return downloadURL;
});
return null;
} catch (error) {
return null;
}
}
I am new to Node.js and express.
I use following function to upload image to s3.
function defaultContentType(req, file, cb) {
setImmediate(function () {
var ct = file.contentType || file.mimetype || 'application/octet-stream'
cb(null, ct);
});
}
module.exports = async function (fileName, file) {
aws.config.update({
secretAccessKey: process.env.AWSSecretKey,
accessKeyId: process.env.AWSAccessKeyId,
contentType: defaultContentType,
});
var s3bucket = new aws.S3({
params: {
Bucket: process.env.S3_Bucket_Name,
}
});
var params = {
Key: fileName,
Body: file
};
var fileData = await s3bucket.upload(params, function (err, data) {
if (err) {
throw err;
} else {
return data;
}
});
return fileData;
}
before uploading the image, I resize it using
request(req.file.location, async function (err, response, body) {
var fileInstance = await sharp(body);
var resizeFile = await fileInstance.resize({
height: 150,
fit: 'inside'
});
var data = await s3Upload('mobile_' + req.file.key, resizeFile);
req.mobile = data.Location;
next();
});
The problem I have is;
The image does get resized and saved to s3.
But "s3Upload" function does not return the file location.
Seems like it take some time to complete the operation. Before it get completed, undefined value get return.
Can anyone suggest a way to fix this?
Modified method
module.exports = function (fileName, file, finishCallback) {
// more code
s3bucket.upload(params, function (err, data) {
if (err) {
throw err;
} else {
finishCallback(data);
}
});
}
modified the upload method as
s3Upload('mobile_' + req.file.key, resizeFile, (data) => {
req.mobile = data.Location;
next();
});
This seems to be working as expected.
I am not really sure this is the correct way to do things.
Is there a way to do this correctly?