I want to read the csv file from the s3 bucket using boto3 and upload it to external API using multipart/form-data request.
so far I am able to read the csv
response = s3.get_object(Bucket=bucket, Key=key)
body = response['Body']
Not sure on how to convert this body into multipart.
External api will be taking request in multipart/form-data.
Any Suggestions would be helpful.
Following method solved my issue.
body = response['Body'].read()
multipart_data = MultipartEncoder(
fields={
'file': (file_name, body, 'application/vnd.ms-excel'),
'field01': 'test'
}
)
.read() method will convert the file into binary string.
Related
I'm trying to upload images on specific slack channel using Karate but no luck, I tried multiple times with different steps but still have 200 response and the image is not displayed in the channel.
Tried to post text content and successfully found the text on the channel.
Bellow are 2 of my tries following the Karate documentation:
#post
Feature: Post images
Background:
* url 'https://slack.com/api/files.upload'
* def req_params= {token: 'xxxxxxx',channels:'team',filename:'from Karate',pretty:'1'}
Scenario: upload image
Given path 'api','files'
And params req_headers
And multipart file myFile = { read: 'thumb.jpg', filename:
'upload-name.jpg', contentType: 'image/jpg' }
And multipart field message = 'image upload test'
And request req_headers
When method post
Then status 200
OR
Given path 'files','binary'
And param req_params
And request read('thumb.jpg')
When method post
Then status 200
Am I missing something? Tried the same examples found in Karate demo GitHub repository of uploading pdf and jpg but no luck.
Note: worked using Slack API UI.
You seem to be mixing up things, there is no need for a request body when you are using multipart. Your headers / params look off. Also based on the doc here, the name of the file-upload field is file. Try this:
Scenario: upload image
Given url 'https://slack.com/api/files.upload'
And multipart file file = { read: 'thumb.jpg', filename:
'upload-name.jpg', contentType: 'image/jpg' }
And multipart field token = 'xxxx-xxxxxxxxx-xxxx'
When method post
Then status 200
If this doesn't work, take the help of someone who can understand how to interpret the Slack API doc. Or get a Postman test working, then you'll easily figure out what you missed.
I'm trying to write a controller that uploads a file to S3 location. However, before upload I need to validate if the incoming file type is a csv or not. And then I need to read the file to check for header colummns in the files etc. I got the type of the file as per below snippet:
req.file('foo')._files[0].stream
But, how to read the entire file stream and check for headers and data etc?There were other similar Qs like (Sails.js Skipper: How to read the uploaded file stream during upload?). But the solution mentioned is to use skipper-csv adapter(which i cannot use as I already use skipper-s3 to upload to s3).
Can someone please post an example on how to read the upstreams and perform any validations before the upload?
Here is how my problem got solved: I'm making a copy of the stream to validate before actual upload. And then checking my validations on the original stream and once passed, I upload the copied stream to my desired location.
For reading the Csv stream, I found a npm package: csv-parser(https://github.com/mafintosh/csv-parser) , which I felt easy to handle events like headers, data.
For creating the copy of the stream, I used the following logic:
const upstream = req.file('file');
const fileStreamMap = {};
const fileStreamMapCopy = {};
_.each(upstream._files, (file) => {
const stream = PassThrough();
const streamCopy = PassThrough();
file.stream.pipe(stream);
file.stream.pipe(streamCopy);
fileStreamMap[fileName] = stream;
fileStreamMapCopy[fileName] = streamCopy;
});
// validate and upload files to S3, if Valid.
validateAndUploadFile(fileStreamMap, fileStreamMapCopy);
}
validateAndUploadFile() contains my custom validation logic for my csv upload.
Also, we can use aws-sdk(https://www.npmjs.com/package/aws-sdk) for s3 upload.
Hope, this helps someone.
The API docs show that you should be able to make insert Jobs resumable, but with no actual ability to receive the resumable upload URI or any examples showing where to set the request as 'resumable'. I am not uploading via a csv/json file, I am streaming bytes in json format straight to the Job.
Some of my code (Java using bigquery v2 api):
AbstractInputStreamContent content = new ByteArrayContent("application/json", stream.toByteArray());
Job insertJob = service.jobs().insert(projectId, loadJob, content).execute();
Please explain where to make them resumable. JobConfiguration? Job? AbstractInputStreamContent? service.jobs.insert?
I'm trying to upload a file to a rest api using HTTPoison.
I can get a file upload to work with the following code, but I need to replace {"name", "value"} with a json payload or find a way to have json/nested data be sent with same request.
url = 'https://sandbox.e-signlive.com/api/packages/dc337476-9841-454a-b5d4-fe5a2b60136d/documents'
headers = [{"Authorization", "secret_key"}]
HTTPoison.post!("httpbin.org/post", {:multipart, [{"name", "value"}, {:file, "/Path/to/document.pdf"}]}, headers)
It seems enctype is missing:
enctype="multipart/form-data"
I'd like to upload a file to S3 in parts, and set some metadata on the file. I'm using boto to interact with S3. I'm able to set metadata with single-operation uploads like so:
Is there a way to set metadata with a multipart upload? I've tried this method of copying the key to change the metadata, but it fails with the error: InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: <size>
I've also tried doing the following:
key = bucket.create_key(key_name)
key.set_metadata('some-key', 'value')
<multipart upload>
...but the multipart upload overwrites the metadata.
I'm using code similar to this to do the multipart upload.
Sorry, I just found the answer:
Per the docs:
If you want to provide any metadata describing the object being uploaded, you must provide it in the request to initiate multipart upload.
So in boto, the metadata can be set in the initiate_multipart_upload call. Docs here.
Faced such issue earlier today and discovered that there is no information on how to do that right.
The code example on how we solved that issue provided below.
$uploader = new MultipartUploader($client, $source, [
'bucket' => $bucketName,
'key' => $filename,
'before_initiate' => function (\Aws\Command $command) {
$command['ContentType'] = 'application/octet-stream';
$command['ContentDisposition'] = 'attachment';
},
]);
Unfortunately, documentation https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html#customizing-a-multipart-upload doesn't make it clear and easy to understand that if you'd like to provide alternative meta data with multipart upload you have to go this way.
I hope that will help.