How to download multiple PDF files in CasperJS - pdf

Question

I want to download a list of .pdf files from a site using CasperJS. It (mostly) works, but the last few PDF files are truncated. Here's the relevant code snippet:
casper.then(function a09() {
for (var index = bill_count-1; index>=0; --index) {
casper.then(downloadOnePdf(index));
};
});
function downloadOnePdf(index) {
return function() {
var selector = 'div#myAjaxDiv tbody tr:nth-child(' + (index+1) + ') form a'
casper.log('click ' + selector, 'info');
casper.click(selector);
// casper.waitForResource(/\/Document/); -- see note
// casper.waitForText('%%EOF'); -- see note
};
};
casper.run();
Watching this code execute via web proxy, I can see the .pdf files arrive as the response body. However, the last two or three files aren't fully downloaded -- they get truncated -- and I see a message in the response header 'Client closed connection before receiving entire response'.
This supports my hunch that the casperjs code exits before the pdf is fully downloaded. I tried adding a
casper.waitForResource(/\/Document/)
in my code, but that didn't help. I also tried:
casper.waitForText('%%EOF')
but that timed out even though I could see '%%EOF' in the response body.
So the question is: what's the correct way to ensure that the entire .pdf has arrived in the response body?
ps: The observant reader will notice that I'm not actually saving the .pdf data. That's a problem for another day...

Answer 1

It turns out that the truncated files in the original post is only a small part of a larger problem, but the solution isn't that difficult. Let me explain...
CasperJS doesn't offer direct access to the body of a response, so the OP's approach of casper.click()ing on a form that returns the PDF data in the response won't get anywhere. AFAIK, there's no way to actually save that data on the local filesystem.
Instead, you need to call casper.download() to POST the same form that you would have gotten when you clicked on the form. The key points to this approach are straightforward, albeit thinly documented:
Load the DOM that contains the table in whatever way is appropriate
for the situation.
Use CSS's nth-child() pseudo class to select individual rows from the table.
Use casper.getFormValues() to construct a POSTable form.
Use casper.download() to post the form and save the resulting data.
The relevant code excerpts follow. I hope someone will find this useful (even if that someone is me a few months from now :)
// ========== helpers
// The following helpers assume that the current DOM contains the table with the download forms
// Return the number of PDFs available for download.
function countPDFs() {
return casper.getElementsAttribute('div#myAjaxDiv tbody tr form input[name="id"]', 'value').length
}
// Get the invoice ID of the index'th invoice: 0 <= index < countPDFs().
function getInvoiceID(index) {
return casper.getElementAttribute('div#myAjaxDiv tbody tr:nth-child(' + (index+1) + ') form input[name="id"]', 'value');
}
// Return the index'th form for downloading a .pdf: 0 <= index < countPDFs().
function getDownloadForm(index) {
return casper.getFormValues('div#myAjaxDiv tbody tr:nth-child(' + (index+1) + ') form')
}
// Download the index'th PDF file, saving it to <target_directory>/<invoiceID>.pdf.
// 0 <= index < countPDFs().
function downloadOnePDF(index, target_directory) {
var
url = 'https://example.com/Invoice',
target = target_directory + '/' + getInvoiceID(index) + '.pdf',
data = getDownloadForm(index);
casper.then(function d01() {
casper.log('downloading pdf ' + index + ' to ' + url);
casper.download(url, target, 'POST', data);
});
}
// ========== casper agenda items
// (initial steps omitted)
// Click on "Invoice" button to bring up the Invoice page
var invoice_link_css = 'a#mnuInvoiceSubmit'
casper.then(function a06() {
casper.click(invoice_link_css)
});
// Make sure the Invoice page has loaded, as evidenced by the presence of the
// bill history table.
casper.then(function a07() {
casper.waitForSelector('div#myAjaxDiv tbody');
});
// Download each .pdf file referenced in the bill history table.
casper.then(function a08() {
var pdf_count = countPDFs();
casper.echo('found ' + pdf_count + ' past bill' + ((pdf_count == 1) ? '' : 's'));
for (var index = pdf_count-1; index>=0; --index) {
downloadOnePDF(index, target_directory);
}
});
casper.run();
This approach saves each .pdf file to the local filesystem, and doesn't exhibit any of the truncating problems in the OP.