I know with application/octent-stream can skip set up of boundary mess in the request.
What are the other advantages of file upload with content type application/octent-stream over multipartdata.
Related
I started a weed filer.backup process to backup all the data to an S3 bucket. Lot of logs are getting generated with the below error messages. Do i need to update any config to resolve or these messages can be ignored?
s3_write.go:99] [persistent-backup] completeMultipartUpload buckets/persistent/BEV_Processed_Data/2011_09_30/2011_09_30/GT_BEV_Output/0000000168.png: EntityTooSmall: Your proposed upload is smaller than the minimum allowed size
Apr 21 09:20:14 worker-server-004 seaweedfs-filer-backup[3076983]: #011status code: 400, request id: 10N2S6X73QVWK78G, host id: y2dsSnf7YTtMLIQSCW1eqrgvkom3lQ5HZegDjL4MgU8KkjDG/4U83BOr6qdUtHm8S4ScxI5HwZw=
Another message
malformed xml the xml you provided was not well formed or did not validate against
This issue happens with empty files or files with small content. Looks like aws s3 multipart upload does not accept streaming empty files. Is there any setting on SeaweedFs that i am missing?
I'm using JMeter 3.2.
My requirement is to read an XML file from the disk, replace some tags with dynamic values to ensure each thread sends a unique xml file upload (NOT SOAP Request). The following code in JSR223 sampler works perfectly fine when I try to upload the newfile through POST using a http sampler with ${newfilename} file text/xml.
import org.apache.commons.io.FileUtils;
try {
String content = FileUtils.readFileToString(new File("E:/test.xml"));
content = content.replaceAll("SUB_ID", "${__UUID}");
content = content.replaceAll("ABN_ID", "${empabn}");
content = content.replaceAll("EMPNAME", "${empname}");
vars.put("content", content);
FileUtils.writeStringToFile(new File("E:/testnew${empname}.xml"), content);
}
catch (Throwable ex) {
log.info("What happened?", ex);
throw ex;
}
Instead of writing again to the disk and uploading again, how can I send the contents of string 'content' as part of request body? I have looked at many posts that talk about the input output streams but they are confusing. When I try to send just ${content} in body, the application throws following error:
HTTP Status 500 - Could not write JSON: Name is null (through reference chain: com.xxx.xxx.datafile.rest.DataFileResponse["validationStatus"]); nested exception is com.fasterxml.jackson.databind.JsonMappingException: Name is null (through reference chain:
Appreciate your help.
Multipart POST requests which are being used for files upload are different from normal POST requests hence there no possibility to simply substitute the file with the generated in-memory string.
You need to replicate the request exactly as it would be send by JMeter or real browser and manually populate each part starting from defining boundary using the HTTP Header Manager and ending up with the creation of Content-Disposition and specify your file contents there.
A little hint: you don't need to generate/substitute values for each call, it is enough to replace them once and JMeter will substitute them on its own given you use __eval() and __FileToString() functions combination.
You can check out Testing REST API File Uploads in JMeter for an example of creation a relatively complex file upload request, in your case it will be easier but still tricky.
I'm trying to write a controller that uploads a file to S3 location. However, before upload I need to validate if the incoming file type is a csv or not. And then I need to read the file to check for header colummns in the files etc. I got the type of the file as per below snippet:
req.file('foo')._files[0].stream
But, how to read the entire file stream and check for headers and data etc?There were other similar Qs like (Sails.js Skipper: How to read the uploaded file stream during upload?). But the solution mentioned is to use skipper-csv adapter(which i cannot use as I already use skipper-s3 to upload to s3).
Can someone please post an example on how to read the upstreams and perform any validations before the upload?
Here is how my problem got solved: I'm making a copy of the stream to validate before actual upload. And then checking my validations on the original stream and once passed, I upload the copied stream to my desired location.
For reading the Csv stream, I found a npm package: csv-parser(https://github.com/mafintosh/csv-parser) , which I felt easy to handle events like headers, data.
For creating the copy of the stream, I used the following logic:
const upstream = req.file('file');
const fileStreamMap = {};
const fileStreamMapCopy = {};
_.each(upstream._files, (file) => {
const stream = PassThrough();
const streamCopy = PassThrough();
file.stream.pipe(stream);
file.stream.pipe(streamCopy);
fileStreamMap[fileName] = stream;
fileStreamMapCopy[fileName] = streamCopy;
});
// validate and upload files to S3, if Valid.
validateAndUploadFile(fileStreamMap, fileStreamMapCopy);
}
validateAndUploadFile() contains my custom validation logic for my csv upload.
Also, we can use aws-sdk(https://www.npmjs.com/package/aws-sdk) for s3 upload.
Hope, this helps someone.
When I create a share link in the UI with the "Anyone with this link can view this item" option, I get a URL that looks like https://onedrive.live.com/redir?resid=XXX!YYYY&authkey=!ZZZZZ&ithint=<contentType>. What I can't figure out is how to use this URL from code to download the content of the file. Hitting the link gives HTML for a page to show the file.
How can I construct a call to download the file? Also, is there a way to construct a call to get some (XML/JSON) metadata about the file, and maybe even a preview or something? I want to be able to do this all without prompting a user for credentials, and all the API docs are about how to make authenticated calls. I want to make anonymous calls to get publicly-shared files.
Have a read over https://dev.onedrive.com - it documents how you can make a query to our service to get the metadata for an item, along with URLs that can be used to directly download the content.
Update with more details
Sorry, the documentation you need for your specific scenario is still in process (along with the associated SDK changes) so I'll give you an overview of how to do it.
There's a sibling to the /drives path called /shares which accepts a sharing URL (such as the one you have above) in an encoded format and allows you to get metadata for the item it represents. This does not require authentication provided the sharing URL has a valid authkey.
The encoding scheme for the id is u!<UrlSafeBase64EncodedUrl>, where <UrlSafeBase64EncodedUrl> follows the guidelines outlined here (trim the = characters from the end).
Here's a snippet that should give you an idea of the whole process:
string originalUrl = "https://onedrive.live.com/redir?resid=XXX!YYYY&authkey=!foo";
byte[] urlAsUtf8Bytes = Encoding.UTF8.GetBytes(originalUrl);
string utf8BytesAsBase64String = Convert.ToBase64String(urlAsUtf8Bytes);
string encodedUrl = "u!" + utf8BytesAsBase64String.TrimEnd('=').Replace('/', '_').Replace('+', '-');
string metadataUrl = "https://api.onedrive.com/v1.0/shares/" + encodedUrl + "/root";
From there you can append /content if you want to get the contents of the file, or you can start navigating through if the URL represents a folder (e.g. /children/childfile.txt)
I want to set header for each request.
header is -
"token:value"
For every request this header value will be different and I have a csv file having all tokens listed there. Now, like we use csv for POST data values, I want to use this csv file to set header for each request.
CSV file -
"token1,token2,token3"
How to do this in jmeter ?
To achieve your goal, you should do following for each request:
Add HTTP Header Manager:
2 . Add CSV Data Set Config configured like this:
the idea is pretty good, if the referer field is widely used.
I'm not sure that you can use CSV data sets, because it reads ONE line for each user/ thread.
In a situation similar to yours, I had to use a __CSVread function, that allows to manually control if and when to read the next line.
Refer to: "CSVRead">http://jmeter.apache.org/usermanual/functions.html#_CSVRead"
If request the same but header changes, then you obviously need CSV Data Set Config element. Example of usage you can find in %JMETER_HOME%/bin/examples. If requests are different then I'm not sure it makes sens to use CSV file.