I am trying to download a csv file(advert report) from a site using the below code. The issue is, it will download the HTML page and not the csv file. I cannot give you the URL as it is behind the login, but it is similar case when you download Firefox from the below URL
http://www.mozilla.org/en-US/firefox/new/
It is a GET request and when I do inspect element Network Tab the get request gets Cancelled. I am new to Casper and don't know how to handle such requests. Any help would be appreciated
casper.then(function() {
var downloadURL = "";
this.evaluate(function() {
var downloadURL = "http://www.lijit.com"+jQuery("#dailyCSV").attr('href');
});
this.download(downloadURL, '/Users/Ujwal/Downloads/caspertests/stats.csv');
});
Response Header
Age:0
Cache-Control:max-age=0
Connection:keep-alive
Content-Disposition:attachment; filename=stats.csv
Content-Encoding:gzip
Content-Length:1634
Content-Type:text/x-csv
Date:Sat, 05 Oct 2013 15:28:21 GMT
Expires:Sat, 05 Oct 2013 15:28:21 GMT
P3P:CP="CUR ADM OUR NOR STA NID"
Server:PWS/8.0.16
Vary:Accept-Encoding
X-Px:ms h0-s28.p9-jfk ( h0-s62.p9-jfk), ms h0-s62.p9-jfk ( origin>CONN)
Answered my own question, here is the solution
reference: https://github.com/knorrium/google-books-downloader/blob/master/gbd.js
//Download the daily csv
casper.then(function() {
this.click('#dailyCSV');
});
casper.on('resource.received', function (resource) {
"use strict";
if ((resource.url.indexOf("publisherCSV/?startDate=") !== -1) ) {
this.echo(resource.url);
var url, file;
url = resource.url;
file = "stats.csv";
try {
this.echo("Attempting to download file " + file);
var fs = require('fs');
casper.download(resource.url, fs.workingDirectory+'/'+file);
} catch (e) {
this.echo(e);
}
}
});
Related
in my blazor server app (.NET6.0) i serve some static files and show them by embedding them a iframe (so the browser deal directly with the type of file, can be image, video, sound, pdf, etc)
I notice encoding problem on accentuated character on the txt and the html files when its shown in the iframe
I try to insert a inside the head on the iframe but same result
i notice on the http call to the file these Response Headers
accept-ranges: bytes
content-encoding: br
content-type: text/plain
date: Thu, 23 Dec 2021 13:15:47 GMT
etag: "1d7f7ff334b008b"
last-modified: Thu, 23 Dec 2021 13:15:48 GMT
server: Kestrel
vary: Accept-Encoding
Im surprise there is no utf-8 specified in the content-type header, im wondering if this is the source of my problem ?
I expected something like that content-type: text/plain; charset=utf-8
i try to play with the StaticFileOptions in Startup to change the headers but even put empty option maker the app broken at startup
app.UseStaticFiles(new StaticFileOptions {
});
//even doing this break the app, when the app start the file blazor.server.js finish in 404 on client side
So i can't really make something here
th serve my static files, i use a virtual directory on this manner
app.UseFileServer(new FileServerOptions
{
FileProvider = new PhysicalFileProvider(Sys.Web.AppliIs.Path_Webdav),
RequestPath = new PathString("/" + Sys.Web.AppliIs.WEBDAV_FOLDER),
EnableDirectoryBrowsing = false
});
I notice i don't have encoding problem when i open the link directly with chrome, its only inside my iframe for now, i can't explain that.
Thanks for your help
I was able to override the OnPrepareResponse to add the charset, i don't know why but i have to put ISO charset to resolve encoding problems
var lOptions = new FileServerOptions
{
FileProvider = new PhysicalFileProvider(Sys.Web.AppliIs.Path_Webdav),
RequestPath = new PathString("/" + Sys.Web.AppliIs.WEBDAV_FOLDER),
EnableDirectoryBrowsing = false,
};
lOptions.StaticFileOptions.OnPrepareResponse = (context) =>
{
var headers = context.Context.Response.Headers;
var contentType = headers["Content-Type"];
contentType += "; charset=ISO-8859-1";
headers["Content-Type"] = contentType;
};
app.UseFileServer(lOptions);
for me the subject is close, but if anybody know why i have to specify this charset im still intersted.
If I have a file on s3 how can I change metadata of that file?
It looks like I can "copy" it to the same location with new headers which would effectively be the same thing.
I'm using knox as the node client to do this. The file in question already has the Content-Type header set to video/mp4 but I want to change it to application/octet-stream. The reason for this is so that this link will trigger the browser to download the resource instead of displaying it in the browser window.
Link to knox source for this function
var filename = "/example/file.mp4",
headers = {'Content-Type': "application/octet-stream"};
client.copyFile(filename, filename, headers, function(error, resp) {
//response is successful
});
The response is successful, but when I reload the resource in s3 I don't see that headers have changed.
I can see that the underlying API call is this:
'PUT /example/file.mp4 HTTP/1.1\r\nContent-Type: application/octet-stream
x-amz-copy-source: /bucket/example/file.mp4
Content-Length: 0\r\nDate: Thu, 28 Jan 2016 21:13:12 GMT
Host: cc-video-archives-dev.s3.amazonaws.com
Authorization: <redacted>=\r\nConnection: close\r\n\r\n',
I was missing this header:
"x-amz-metadata-directive": "REPLACE"
var filename = "/example/file.mp4",
headers = {
"x-amz-metadata-directive": "REPLACE",
'Content-Type': "application/octet-stream"
};
client.copyFile(filename, filename, headers, function(error, resp) {
//response is successful
});
I have been tearing my hair out for the last couple of hours and I hope someone here could help me out. I have an angular application which is using (https://github.com/danialfarid/ng-file-upload) and no matter what I try I can't seem to be able to get the files out at the other end.
I do see the correct request payload in the Chrome developer tools:
------WebKitFormBoundaryEwG0XOfjS0IjwRji
Content-Disposition: form-data; name="file"; filename="images.png"
Content-Type: image/png
------WebKitFormBoundaryEwG0XOfjS0IjwRji-
My code on the server side looks as follows, I am using a router which is using connect-multiparty.
Router.js:
router.post('/api/v1/uploaddocument', multipartyMiddleware, UserFunctions.saveDocument);
The actual save document function:
saveDocument: function(req,res)
{
console.log(req.body, req.files, req.data, req.file)
}
The controller posting this message in angular is:
iSelectClient.controller('PageUploadDocumentCtrl', ['$scope', 'Upload', function ($scope, Upload) {
$scope.$watch('files', function () {
$scope.upload($scope.files);
});
$scope.upload = function (files) {
if (files && files.length) {
for (var i = 0; i < files.length; i++) {
var file = files[i];
Upload.upload({
url: 'http://localhost:3000/api/v1/uploaddocument',
fields: {'username': $scope.username},
file: file
}).progress(function (evt) {
var progressPercentage = parseInt(100.0 * evt.loaded / evt.total);
console.log('progress: ' + progressPercentage + '% ' + evt.config.file.name);
}).success(function (data, status, headers, config) {
console.log('file ' + config.file.name + 'uploaded. Response: ' + data);
});
}
}
};
}]);
I am not using app.use(bodyparser()) on the server side, for each individual route I am defining which parser to use.
What o what could it be?
EDIT
I had an interceptor on each call which set the application type to json, so it never reached the other end correctly. Fixed now, using the exact same setup as below
I had an interceptor on each call which set the application type to json, so it never reached the other end correctly. Fixed now, using the exact same setup as below
Hey I'm using dropboxuploader.php to login into dropbox. All was working fine, but when i came into work yesterday i could no longer connect. This is what dropbox is returning to me.
HTTP/1.1 100 Continue
HTTP/1.1 403 Forbidden
Server: nginx/1.2.3
Date: Thu, 04 Oct 2012 08:44:36 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
It seems you tried to do something we can't verify. Did you log into a different Dropbox account in a different window? Try clicking here to go back to the page you came from, or just go home.
Replace the login function with below code and it should work:
protected function login() {
$data = $this->request('https://www.dropbox.com/login');
$str = '<input type="hidden" name="t" value="';
$start = strpos($data,$str);
$val = "";
if($start !== false)
{
$val = substr($data,$start+strlen($str),24);
}
$data = $this->request('https://www.dropbox.com/login', true, array('login_email'=>$this->email, 'login_password'=>$this->password, 't'=>$val));
if (stripos($data, 'location: /home') === false)
throw new Exception('Login unsuccessful.');
$this->loggedIn = true;
}
Just update your dropbox uploader file instead doing your fixes.
https://github.com/jakajancar/DropboxUploader
I am trying to upload a .mp4 file to some server. I am using the HTTP client provided by titanium. when I upload the file, HTTP client is adding some headers in the file due to which the file gets corrupted and cannot be played. When I download the uploaded file and open it in notepad I can see the header which are added to the file.
What should I do so that these headers are not added to the file?
Thanks a lot!
// CODE
var uploadFile = Titanium.Filesystem.getFile(dir, _previewUrl);
var fileUploadUrl = 'Some Url for the server to upload';
var headers = { 'Content-Type' : 'multipart/form-data' };
var content = { 'file' : uploadFile };
var xhr = Titanium.Network.createHTTPClient();
for(var key in _headers) {
xhr.setRequestHeader(key, _headers[key]);
}
xhr.onerror = function(e)
{
Ti.UI.createAlertDialog({title:'Error', message:e.error}).show();
Ti.API.info('IN ERROR ' + e.error);
};
xhr.setTimeout(20000);
xhr.onload = function(e)
{
Ti.UI.createAlertDialog({title:'Success', message:'status code ' + this.status}).show();
Ti.API.info('IN ONLOAD ' + this.status + ' readyState ' + this.readyState);
};
xhr.onsendstream = function(e)
{
ind.value = e.progress ;
Ti.API.info('ONSENDSTREAM - PROGRESS: ' + e.progress);
};
// open the client
xhr.open('POST',fileUploadUrl);
// send the data
xhr.send(content);
// END
try setting the headers after you call xhr.open
// open the client
xhr.open('POST',fileUploadUrl);
for(var key in _headers) {
xhr.setRequestHeader(key, _headers[key]);
}
Do not add { 'Content-Type' : 'multipart/form-data' }; header. This way you should get the file properly without any headers like boundary and file name etc. I could send image, 3gpp file like that successfully But, when I send a video file, my server PHP code $_FILES will be empty array. Even the $_FILES["files"]["error"] have no value. There should some other trick to send video file. (Titanium SDK 3.1.1 & android 4.1.2)
xhr.open("POST", URL);
xhr.send({
files : Titanium.Filesystem.getFile(Titanium.Filesystem.applicationDataDirectory, sourcefilename)
});
}
Try not sending the raw blob itself. Send base64 encoded string instead.
var uploadFile = Titanium.Filesystem.getFile(dir, _previewUrl);
var base64File = Ti.Utils.base64encode(uploadFile.read()).toString();
And try changing the header to
xhr.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
xhr.send(base64File);
That will solve your problem.