Netlify server functions unable to handle multipart/form-data - pdf

I’m working on a Netlify Function where we take form data for a job application (including a file upload) and pass the data on to a third-party API for use in their system. I was following along with this handy post (thanks!) —
https://www.netlify.com/blog/2021/07/29/how-to-process-multipart-form-data-with-a-netlify-function/
— but seem to have run into a situation where the data in the file is not handled properly (for example, PDFs turn up with blank content, though ASCII metadata appears to be at least partly intact), at least when using the Netlify CLI; I have yet to try on a deploy preview. Writing to a local directory confirms that the issue isn’t with the third party API. Is there something I’m missing when working with these files? Example code below (note that I’ve also attempted to work with the Buffer data, with identical results).
Fetch function to call the Netlify Function:
const data = new FormData(form);
fetch('/.netlify/functions/apply', {
method: 'POST',
body: data,
}).then(res => {
if (!res.ok && res.status !== 406) {
throw new Error('oh no');
}
return res.json();
}).then(data => {
if (Array.isArray(data.missingRequiredFields) && data.missingRequiredFields.length > 0) {
console.log(data);
showMissingFields(data.missingRequiredFields);
} else {
showConfirmationMessage(data.message);
}
}).catch(err => {
showWarningMessage('Something went wrong; please try again.');
}).finally(() => {
submitButton.removeAttribute('disabled');
});
And here’s our Netlify Function:
const Busboy = require("busboy");
const FormData = require("form-data");
const fetch = require("node-fetch");
function parseMultipartForm(event) {
// source: https://www.netlify.com/blog/2021/07/29/how-to-process-multipart-form-data-with-a-netlify-function/
return new Promise(resolve => {
const fields = {};
const busboy = new Busboy({
// uses request headers to extract the form boundary value
headers: event.headers,
});
// before parsing anything, we need to set up some handlers.
// whenever busboy comes across a file ...
const f = [];
busboy.on("file", (fieldname, filestream, filename, transferEncoding, mimeType) => {
// ... we take a look at the file's data ...
filestream.on("data", data => {
fields[fieldname] = {
filename,
type: mimeType,
content: data,
transferEncoding,
};
});
});
// whenever busboy comes across a normal field ...
busboy.on("field", (fieldName, value) => {
// ... we write its value into `fields`.
fields[fieldName] = value;
});
// once busboy is finished, we resolve the promise with the resulted fields.
busboy.on("finish", () => {
resolve(fields);
});
// now that all handlers are set up, we can finally start processing our request!
busboy.write(event.body);
});
}
/** ***************************************************************************
* Serverless function
**************************************************************************** */
exports.handler = async function(event, context) {
// parse the incoming multipart/form-data data into fields object
const fields = await parseMultipartForm(event);
// create new formdata object to be send to Lever
const form = new FormData();
for (const [key, value] of Object.entries(fields)) {
if (key === "resume") {
// append "resume" with the file buffer and add the file name
form.append("resume", value.content, { filename: value.filename });
} else {
form.append(key, value);
}
}
};
Any guidance you might have would be greatly appreciated.

Related

Sequelize, findOrCreate + findAll unexpected behavior

I'm trying to fetch data from a dog API and I want to add to my database only their temperaments. I tried using some loops and a split to isolate the data and then using
findOrCreate() to add only those who are not already in the DB, after that I use findAll()
to get that info from the DB to send it using expressJS.
The unexpected behavior comes when I go to the route that executes all this and the route
only gives about half of the temperaments (they are 124 and it displays arround 54), then when I refresh the page it shows all 124 of them. The DB gets populated with all the 124 in one go so the problem is with findAll()
This is the function that isolates the temperaments and append them to the DB:
module.exports = async () => {
const info = await getAllDogs();
info.forEach(async (element) => {
const { temperament } = element;
if (temperament) {
const eachOne = temperament.split(", ");
for (i in eachOne) {
await Temperament.findOrCreate({
where: { name: eachOne[i] },
});
}
}
});
};
And this is the code that gets executed when I hit my expressJS sv to get the info
exports.temperaments = async (req, res) => {
try {
await getTemperaments(); //this function is the above function
} catch (error) {
res.status(500).send("something gone wrong", error);
}
const temperamentsDB = await Temperament.findAll();
res.json(temperamentsDB);
};
So as you can see the last function executes the function that appends all the data to the DB and then sends it with findAll and res.json()
forEach is a synchronous method so it doesn't await a result of the async callback. You need to do for of in order to get wait for all results:
module.exports = async () => {
const info = await getAllDogs();
for (element of info) {
const { temperament } = element;
if (temperament) {
const eachOne = temperament.split(", ");
for (i in eachOne) {
await Temperament.findOrCreate({
where: { name: eachOne[i] },
});
}
}
}
};

How to return blob item from localForage in service worker?

in my service worker i store mp4 video in indexedDB with localforage library in a blob data. It's work ! but i don't know how can i return this blob data.
This is my fetchHandler code :
const fetchHandler = async (event) => {
const getResponse = async () => {
const request = event.request;
if( request.destination === 'video' ){
// Get from indexedDB
const value = await localforage.getItem('video');
// How return the video from indexedDB cache ?
if( value ) return value; // not working
// Add in indexedDB
var networkResponse = await fetch(event.request);
localforage.setItem('video', networkResponse.blob() ).then(function (value) {
// Do other things once the value has been saved.
console.log(value);
}).catch(function(err) {
// This code runs if there were any errors
console.log(err);
});
}else{
const openedCache = await caches.open(SW_CACHE_NAME);
const cacheResponse = await openedCache.match(request);
if (cacheResponse) return cacheResponse;
var networkResponse = await fetch(event.request);
const cachePutResponse = await openedCache.put(request, networkResponse.clone());
if (cachePutResponse) return cachePutResponse;
}
return networkResponse;
};
event.respondWith(getResponse());
};
thanks for your help
You need to pass a valid Response object to event.respondWith(). That entails a response body (which is what you get back from localforeage.getItem()), but also some response headers.
You can use the Response constructor to create that, and return it from your getResponse() function.
The code could look something like:
const value = await localforage.getItem('video');
if (value) {
// See https://fetch.spec.whatwg.org/#bodyinit for what's accepted
// as a BodyInit.
return new Response(value, {
headers: {
// Replace this with the actual MIME type for the video.
'content-type': 'video/mp4',
// Include any other headers here if desired.
}
});
}

Vue.js Axios responseType blob or json object

I have an Axios request to download the .xls file. Problem is that the object returned as a response from backend doesn't always has to be a real file. If I return json object instead of file data. How I would read this json then?
Here is the function:
downloadReport() {
let postConfig = {
headers: {
'X-Requested-With': 'XMLHttpRequest'
},
responseType: 'blob',
} as AxiosRequestConfig
axios
.post(this.urls.exportDiscountReport, this.discount.id, postConfig)
.then((response) => {
let blob = new Blob([response.data], { type: 'application/vnd.ms-excel' });
let url = window['URL'].createObjectURL(blob);
let a = document.createElement('a');
a.href = url;
a.download = this.discount.id + ' discount draft.xlsx';
a.click();
window['URL'].revokeObjectURL(url);
})
.catch(error => {
})
}
I would like to read the response and if it contains some data in it - don't create the blob and initiate the download, instead, just show some message or whatever. If I remove the responseType: 'blob' then the .xls file downloads as unreadable and not valid file.
So the problem is that every returned response becomes blob type and I don't see my returned data in it. Any ideas?
I solved this by reading the blob response and checking if it has JSON parameter status. But this looks like an overengineering to me. Is there a better solution?
let postConfig = {
headers: {
'X-Requested-With': 'XMLHttpRequest'
},
responseType: 'blob',
} as AxiosRequestConfig
axios
.post(this.urls.exportDiscountReport, this.discount.id, postConfig)
.then((responseBlob) => {
const self = this;
let reader = new FileReader();
reader.onload = function() {
let response = { status: true, message: '' };
try {
response = JSON.parse(<string>reader.result);
} catch (e) {}
if (response.status) {
let blob = new Blob([responseBlob.data], { type: 'application/vnd.ms-excel' });
let url = window['URL'].createObjectURL(blob);
let a = document.createElement('a');
a.href = url;
a.download = self.discount.id + ' discount draft.xlsx';
a.click();
window['URL'].revokeObjectURL(url);
} else {
alert(response.message)
}
}
reader.readAsText(responseBlob.data);
})
.catch(error => {
})
I also found the same solution here: https://github.com/axios/axios/issues/815#issuecomment-340972365
Still looks way too hacky.
Have you tried checking the responseBlob.type property? It gives the MIME type of the returned data.
So for example you could have something like this:
const jsonMimeType = 'application/json';
const dataType = responseBlob.type;
// The instanceof Blob check here is redundant, however it's worth making sure
const isBlob = responseBlob instanceof Blob && dataType !== jsonMimeType;
if (isBlob) {
// do your blob download handling
} else {
responseBlob.text().then(text => {
const res = JSON.parse(text);
// do your JSON handling
});
}
This I find is much simpler and works for me, however it depends on your backend setup. The BLOB response is still a text response, but it's handled slightly differently.

Axios interceptors don't send data to API in production Heroku app

This is part 2 of me debugging my application in production
In part 1, I managed to at least see what was causing my problem and managed to solve that.
When I send a request to my API which is hosted on Heroku using axios interceptor, every single request object looks like this in the API
{ 'object Object': '' }
Before sending out data to the API, I console.log() the transformRequest in axios and I can see that the data I am sending is actually there.
Note: I have tested this process simply using
axios.<HTTP_METHOD>('my/path', myData)
// ACTUAL EXAMPLE
await axios.post(
`${process.env.VUE_APP_BASE_URL}/auth/login`,
userToLogin
);
and everything works and I get data back from the server.
While that is great and all, I would like to abstract my request implementation into a separate class like I did below.
Does anyone know why the interceptor is causing this issue? Am I misusing it?
request.ts
import axios from "axios";
import { Message } from "element-ui";
import logger from "#/plugins/logger";
import { UsersModule } from "#/store/modules/users";
const DEBUG = process.env.NODE_ENV === "development";
const service = axios.create({
baseURL: process.env.VUE_APP_BASE_URL,
timeout: 5000,
transformRequest: [function (data) {
console.log('data', data)
return data;
}],
});
service.interceptors.request.use(
config => {
if (DEBUG) {
logger.request({
method: config.method,
url: config.url
});
}
return config;
},
error => {
return Promise.reject(error);
}
);
service.interceptors.response.use(
response => {
console.log('axios interception response', response)
return response.data;
},
error => {
const { response } = error;
console.error('axios interception error', error)
if (DEBUG) {
logger.error(response.data.message, response);
}
Message({
message: `Error: ${response.data.message}`,
type: "error",
duration: 5 * 1000
});
return Promise.reject({ ...error });
}
);
export default service;
Login.vue
/**
* Sign user in
*/
async onClickLogin() {
const userToLogin = {
username: this.loginForm.username,
password: this.loginForm.password
};
try {
const res = await UsersModule.LOGIN_USER(userToLogin);
console.log("res", res);
this.onClickLoginSuccess();
} catch (error) {
throw new Error(error);
}
}
UsersModule (VUEX Store)
#Action({ rawError: true })
async [LOGIN_USER](params: UserSubmitLogin) {
const response: any = await login(params);
console.log('response in VUEX', response)
if (typeof response !== "undefined") {
const { accessToken, username, name, uid } = response;
setToken(accessToken);
this.SET_UID(uid);
this.SET_TOKEN(accessToken);
this.SET_USERNAME(username);
this.SET_NAME(name);
}
}
users api class
export const login = async (data: UserSubmitLogin) => {
return await request({
url: "/auth/login",
method: "post",
data
});
};
I'm not sure what you're trying to do with transformRequest but that probably isn't what you want.
A quote from the documentation, https://github.com/axios/axios#request-config:
The last function in the array must return a string or an instance of Buffer, ArrayBuffer, FormData or Stream
If you just return a normal JavaScript object instead it will be mangled in the way you've observed.
transformRequest is responsible for taking the data value and converting it into something that can actually be sent over the wire. The default implementation does quite a lot of work manipulating the data and setting relevant headers, in particular Content-Type. See:
https://github.com/axios/axios/blob/885ada6d9b87801a57fe1d19f57304c315703079/lib/defaults.js#L31
If you specify your own transformRequest then you are replacing that default, so none of that stuff will happen automatically.
Without knowing what you're trying to do it's difficult to advise further but you should probably use a request interceptor rather than transformRequest for whatever it is you're trying to do.

How do I serve a file from S3 through Meteor Iron Router

My question is very similar to this one which describes how to serve a local file using Iron Router. I need to do the same, but instead of reading the file synchronously from disk, I need to get the file from S3 which is an asynchronous call.
The problem appears to be the fact that the action method has returned before the asynchronous s3.getObject completes giving me the following error.
Error: Can't render headers after they are sent to the client.
I'm assuming that Iron Router is generating the response for me when it realizes that I haven't handled the response in my action method, but I'm stumped about how to tell it to wait for my asynchronous call to finish.
Here is my code.
Router.map(function () {
this.route('resumeDownload', {
where: 'server',
path: '/resume/:_id',
action: function () {
var response = this.response;
var candidate = Candidates.findOne(this.params._id);
if (!candidate || !candidate.resumeS3Key) {
// this works fine because the method hasn't returned yet.
response.writeHead(404);
return response.end();
}
var s3 = new AWS.S3();
s3.getObject({Bucket: 'myBucket', Key: candidate.resumeS3Key}, function (err, data) {
if (err) {
// this will cause the error to be displayed
response.writeHead(500);
return response.end();
}
// this will also cause the error to be displayed
response.writeHead(200, {'Content-Type': data.ContentType});
response.end(data.Body);
});
}
});
});
I was able to solve this one myself. I needed to use a future in my action method.
Here is the working code.
Router.map(function () {
this.route('resumeDownload', {
where: 'server',
path: '/resume/:_id',
action: function () {
var response = this.response,
candidate = Candidates.findOne(this.params._id);
if (!candidate || !candidate.resumeS3Key) {
response.writeHead(404);
return response.end();
}
var Future = Npm.require('fibers/future'),
s3 = new AWS.S3(),
futureGetObject = Future.wrap(s3.getObject.bind(s3)),
data = futureGetObject({Bucket: 'myBucket', Key: candidate.resumeS3Key}).wait();
response.writeHead(200, {'Content-Type': data.ContentType});
response.end(data.Body);
}
});
});