We've created a Cloud Function that generates a PDF. The library that we're using is
https://www.npmjs.com/package/html-pdf
The problem is when we try to execute the
.create()
method it times out with the following errors
"Error: html-pdf: PDF generation timeout. Phantom.js script did not exit.
at Timeout.execTimeout (/srv/node_modules/html-pdf/lib/pdf.js:91:19)
at ontimeout (timers.js:498:11)
This works fine on localhost but happens when we deploy the function on GCP.
Some solutions we've already tried:
Solution #1
Yes we've updated the timeout settings to
const options = {
format: "A3",
orientation: "portrait",
timeout: "100000"
// zoomFactor: "0.5"
// orientation: "portrait"
};
and it still doesn't work.
here's the final snippet that triggers the PDF function
const options = {
format: "A3",
orientation: "portrait",
timeout: "100000"
// zoomFactor: "0.5"
// orientation: "portrait"
};
try {
// let pdfRes = await new Promise(async (resolve, reject) => {
console.log("Before pdf.create()")
let pdfResponse = await pdf.create(html, options).toFile(localPDFFile, async function (err, res) {
if (err) {
console.log(err)
}
console.log('response of pdf.create(): ', res);
let uploadBucket = await bucket.upload(localPDFFile, {
metadata: { contentType: "application/octet-stream" }
});
let docRef = await db
.collection("Organizations")
.doc(context.params.orgId)
.collection("regulations")
.doc(context.params.regulationId)
.collection("reports")
.doc(context.params.reportId);
await docRef.update({
pdf: {
status: "created",
reportName: pdfName
}
});
});
} catch (error) {
console.log('error: ', error);
}
``
I have seen many cases like this even in my current project we use step functions (when cloud functions needs more computational power we divide them into chunks i.e mini cloud functions).
But i think step functions will not work in your case either because you are using single module.
In your case you should use compute engine to perform this operation.
Using promise, We can fix this timeout error
var Handlebars = require('handlebars');
var pdf = require('html-pdf');
var options = {
height: "10.5in", // allowed units: mm, cm, in, px
width: "8in" // allowed units: mm, cm, in, px
"timeout": 600000
};
var document = {
html: html1,
path: resolvedPath + "/" + filename,
data: {}
};
var create = function(document, options) {
return new Promise((resolve, reject) => {
// Compiles a template
var html = Handlebars.compile(document.html)(document.data);
var pdfPromise = pdf.create(html, options);
// Create PDF from html template generated by handlebars
// Output will be PDF file
pdfPromise.toFile(document.path, (err, res) => {
if (!err)
resolve(res);
else
reject(err);
});
});
}
This seems to be a problem with the html, my problem was that I had an image source linked to a deleted image in a server and that was what caused the time out, I solved it by putting the image in the server's route and that was it, I hope this to be useful to someone
Related
I am using xhr to upload images and videos in a React Native mobile app (currently only tested on Android).
The actual upload works correctly however the xhr.upload.onprogress callback is reporting inaccurate data. For example, when uploading a large file (~70mb) this returns 0%, then 69%, then 98%, then 100% - this is returned over the first few seconds even though the actual file upload takes ~1-2 minutes.
Here is my code:
const formData = new FormData();
formData.append("FileInput", {
uri: uri,
type: "video/" + ext,
name: fileName,
});
const xhr = new XMLHttpRequest();
xhr.open("POST", url);
xhr.onload = () => {
const response = JSON.parse(xhr.response);
resolve(true);
};
xhr.onerror = (e) => {
console.log(e, "upload failed");
};
xhr.ontimeout = (e) => {
console.log(e, "upload timeout");
};
xhr.send(formData);
if (xhr.upload) {
xhr.upload.onprogress = ({ total, loaded }) => {
uploadProgress = Math.round((loaded / total) * 100);
console.log(uploadProgress, total, loaded);
};
}
Any pointers to what might be going on here would be really appreciated.
UPDATE: I have also implemented this upload using axios and get exactly the same issue where the onUploadProgress reports 100% very quickly even though the actual upload takes much longer.
const config = {
onUploadProgress: (progressEvent) => {
uploadProgress = Math.round(progressEvent.loaded / progressEvent.total) * 100;
console.log(uploadProgress);
},
headers: { "Content-Type": "multipart/form-data" },
};
const upload = await axios.post(url, formData, config);
Ok, I've figured this out. Just in case this helps someone else:
The issue was occurring when running a development bundle on a metro server - axios/xhr was reporting on the status of the upload of the file to the metro proxy rather than to it's final destination on the net.
When I created an apk build everything was working correctly.
I am trying to upload a lot of files from S3 to IPFS via Pinata. I haven't found in Pinata documentation something like that.
This is my solution, using the form-data library. I haven't tested it yet (I will do it soon, I need to code some things).
Is it a correct approach? anyone who has done something similar?
async uploadImagesFolder(
items: ItemDocument[],
bucket?: string,
path?: string,
) {
try {
const form = new FormData();
for (const item of items) {
const file = getObjectStream(item.tokenURI, bucket, path);
form.append('file', file, {
filename: item.tokenURI,
});
}
console.log(`Uploading files to IPFS`);
const pinataOptions: PinataOptions = {
cidVersion: 1,
};
const result = await pinata.pinFileToIPFS(form, {
pinataOptions,
});
console.log(`Piñata Response:`, JSON.stringify(result, null, 2));
return result.IpfsHash;
} catch (e) {
console.error(e);
}
}
I had the same problem
So, I have found this: https://medium.com/pinata/stream-files-from-aws-s3-to-ipfs-a0e23ffb7ae5
But in the article If am not wrong, is used a different version to the JavaScript AWS SDK v3 (nowadays the most recent: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/index.html).
This is for the Client side with TypeScript:
If you have this version, for me works this code snippet:
export const getStreamObjectInAwsS3 = async (data: YourParamsType) => {
try {
const BUCKET = data.bucketTarget
const KEY = data.key
const client = new S3Client({
region: 'your-region',
credentials: {
accessKeyId: 'your-access-key',
secretAccessKey: 'secret-key'
}
})
const resource = await client.send(new GetObjectCommand({
Bucket: BUCKET,
Key: KEY
}))
const response = resource.Body
if (response) {
return new Response(await response.transformToByteArray()).blob()
}
return null
} catch (error) {
return null
}
}
With the previous code, you can get the Blob Object for pass it to the File object with this method and get the URL resource using the API:
export const uploadFileToIPFS = async(file: Response) => {
const url = `https://api.pinata.cloud/pinning/pinFileToIPFS`
const data = new FormData()
data.append('file', file)
try {
const response = await axios.post(url, data, {
maxBodyLength: Infinity,
headers: {
pinata_api_key: 'your-api',
pinata_secret_api_key: 'your-secret'
},
data: data
})
return {
success: true,
pinataURL: `https://gateway.pinata.cloud/ipfs/${ response.data.IpfsHash }`
}
} catch (error) {
console.log(error)
return null
}
}
I have found this solution from this nice article and you can explore other implementations (including the Node.js side)
I’m working on a Netlify Function where we take form data for a job application (including a file upload) and pass the data on to a third-party API for use in their system. I was following along with this handy post (thanks!) —
https://www.netlify.com/blog/2021/07/29/how-to-process-multipart-form-data-with-a-netlify-function/
— but seem to have run into a situation where the data in the file is not handled properly (for example, PDFs turn up with blank content, though ASCII metadata appears to be at least partly intact), at least when using the Netlify CLI; I have yet to try on a deploy preview. Writing to a local directory confirms that the issue isn’t with the third party API. Is there something I’m missing when working with these files? Example code below (note that I’ve also attempted to work with the Buffer data, with identical results).
Fetch function to call the Netlify Function:
const data = new FormData(form);
fetch('/.netlify/functions/apply', {
method: 'POST',
body: data,
}).then(res => {
if (!res.ok && res.status !== 406) {
throw new Error('oh no');
}
return res.json();
}).then(data => {
if (Array.isArray(data.missingRequiredFields) && data.missingRequiredFields.length > 0) {
console.log(data);
showMissingFields(data.missingRequiredFields);
} else {
showConfirmationMessage(data.message);
}
}).catch(err => {
showWarningMessage('Something went wrong; please try again.');
}).finally(() => {
submitButton.removeAttribute('disabled');
});
And here’s our Netlify Function:
const Busboy = require("busboy");
const FormData = require("form-data");
const fetch = require("node-fetch");
function parseMultipartForm(event) {
// source: https://www.netlify.com/blog/2021/07/29/how-to-process-multipart-form-data-with-a-netlify-function/
return new Promise(resolve => {
const fields = {};
const busboy = new Busboy({
// uses request headers to extract the form boundary value
headers: event.headers,
});
// before parsing anything, we need to set up some handlers.
// whenever busboy comes across a file ...
const f = [];
busboy.on("file", (fieldname, filestream, filename, transferEncoding, mimeType) => {
// ... we take a look at the file's data ...
filestream.on("data", data => {
fields[fieldname] = {
filename,
type: mimeType,
content: data,
transferEncoding,
};
});
});
// whenever busboy comes across a normal field ...
busboy.on("field", (fieldName, value) => {
// ... we write its value into `fields`.
fields[fieldName] = value;
});
// once busboy is finished, we resolve the promise with the resulted fields.
busboy.on("finish", () => {
resolve(fields);
});
// now that all handlers are set up, we can finally start processing our request!
busboy.write(event.body);
});
}
/** ***************************************************************************
* Serverless function
**************************************************************************** */
exports.handler = async function(event, context) {
// parse the incoming multipart/form-data data into fields object
const fields = await parseMultipartForm(event);
// create new formdata object to be send to Lever
const form = new FormData();
for (const [key, value] of Object.entries(fields)) {
if (key === "resume") {
// append "resume" with the file buffer and add the file name
form.append("resume", value.content, { filename: value.filename });
} else {
form.append(key, value);
}
}
};
Any guidance you might have would be greatly appreciated.
Using Adobe PDF Embed API and want to save annotated PDFs in a browser window to Firestore.
The file is uploaded to Firebase but corrupt and only about 9 bytes in size.
Please see the below code. Is there something I need to do with "content" in the callback?
Attached also a picture of the console.log.
const previewConfig = {
embedMode: "FULL_WINDOW",
showAnnotationTools: true,
showDownloadPDF: true,
showPrintPDF: true,
showPageControls: true
}
document.addEventListener("adobe_dc_view_sdk.ready", function () {
var adobeDCView = new AdobeDC.View({
clientId: "2eab88022c63447f8796b580d5058e71",
divId: "adobe-dc-view"
});
adobeDCView.previewFile({
content: { location: { url: decoded } },
metaData: { fileName: decodedTitle }
}, previewConfig);
/* Register save callback */
adobeDCView.registerCallback(
AdobeDC.View.Enum.CallbackType.SAVE_API,
async function (metaData, content, options) {
console.log(metaData);
console.log(content);
var meta = {
contentType: 'application/pdf'
};
var pdfRef = storageRef.child(decodedTitle);
var upload = await pdfRef.put(content, meta);
console.log('Uploaded a file!');
return new Promise(function (resolve, reject) {
/* Dummy implementation of Save API, replace with your business logic */
setTimeout(function () {
var response = {
code: AdobeDC.View.Enum.ApiResponseCode.SUCCESS,
data: {
metaData: Object.assign(metaData, { updatedAt: new Date().getTime() })
},
};
resolve(response);
}, 2000);
});
}
);
});
I was able to use putString() in Firebase Storage to upload the PDF to storage in the end.
Before I was only using put() which ended up having a corrupt file.
I am new to Node.js and express.
I use following function to upload image to s3.
function defaultContentType(req, file, cb) {
setImmediate(function () {
var ct = file.contentType || file.mimetype || 'application/octet-stream'
cb(null, ct);
});
}
module.exports = async function (fileName, file) {
aws.config.update({
secretAccessKey: process.env.AWSSecretKey,
accessKeyId: process.env.AWSAccessKeyId,
contentType: defaultContentType,
});
var s3bucket = new aws.S3({
params: {
Bucket: process.env.S3_Bucket_Name,
}
});
var params = {
Key: fileName,
Body: file
};
var fileData = await s3bucket.upload(params, function (err, data) {
if (err) {
throw err;
} else {
return data;
}
});
return fileData;
}
before uploading the image, I resize it using
request(req.file.location, async function (err, response, body) {
var fileInstance = await sharp(body);
var resizeFile = await fileInstance.resize({
height: 150,
fit: 'inside'
});
var data = await s3Upload('mobile_' + req.file.key, resizeFile);
req.mobile = data.Location;
next();
});
The problem I have is;
The image does get resized and saved to s3.
But "s3Upload" function does not return the file location.
Seems like it take some time to complete the operation. Before it get completed, undefined value get return.
Can anyone suggest a way to fix this?
Modified method
module.exports = function (fileName, file, finishCallback) {
// more code
s3bucket.upload(params, function (err, data) {
if (err) {
throw err;
} else {
finishCallback(data);
}
});
}
modified the upload method as
s3Upload('mobile_' + req.file.key, resizeFile, (data) => {
req.mobile = data.Location;
next();
});
This seems to be working as expected.
I am not really sure this is the correct way to do things.
Is there a way to do this correctly?