The API docs show that you should be able to make insert Jobs resumable, but with no actual ability to receive the resumable upload URI or any examples showing where to set the request as 'resumable'. I am not uploading via a csv/json file, I am streaming bytes in json format straight to the Job.
Some of my code (Java using bigquery v2 api):
AbstractInputStreamContent content = new ByteArrayContent("application/json", stream.toByteArray());
Job insertJob = service.jobs().insert(projectId, loadJob, content).execute();
Please explain where to make them resumable. JobConfiguration? Job? AbstractInputStreamContent? service.jobs.insert?
Related
I want to read the csv file from the s3 bucket using boto3 and upload it to external API using multipart/form-data request.
so far I am able to read the csv
response = s3.get_object(Bucket=bucket, Key=key)
body = response['Body']
Not sure on how to convert this body into multipart.
External api will be taking request in multipart/form-data.
Any Suggestions would be helpful.
Following method solved my issue.
body = response['Body'].read()
multipart_data = MultipartEncoder(
fields={
'file': (file_name, body, 'application/vnd.ms-excel'),
'field01': 'test'
}
)
.read() method will convert the file into binary string.
I am trying to send an audio file recorded using expo-av library to my server using web sockets.
Websocket will allow me to only send String, ArrayBuffer or Blob. I spent whole day trying to find out how to convert my .wav recording into a blob but without success. I tried to use expo-file-system method FileSystem.readAsStringAsync to read the filed as a string but I get an error that the file could not be read. How is that possible? I passed it the correct URI (using recording.getURI()).
I tried to re-engineer my approach to use fetch and FormData post request with the same URI and audio gets sent correctly. But I really would like to use WebSockets so that later I could try to make it stream the sound to the server in real time instead of recording it first and then sending it.
You can try ... But I can't find a way to read the blob itself
// this is from my code ...
let recording = Audio.Recording ;
const info = await FileSystem.getInfoAsync(recording.getURI() || "");
console.log(`FILE INFO: ${JSON.stringify(info)}`);
// get the file as a blob
const response = await fetch(info.uri);
const blob = await response.blob(); // slow - Takes a lot of time
I'm using JMeter 3.2.
My requirement is to read an XML file from the disk, replace some tags with dynamic values to ensure each thread sends a unique xml file upload (NOT SOAP Request). The following code in JSR223 sampler works perfectly fine when I try to upload the newfile through POST using a http sampler with ${newfilename} file text/xml.
import org.apache.commons.io.FileUtils;
try {
String content = FileUtils.readFileToString(new File("E:/test.xml"));
content = content.replaceAll("SUB_ID", "${__UUID}");
content = content.replaceAll("ABN_ID", "${empabn}");
content = content.replaceAll("EMPNAME", "${empname}");
vars.put("content", content);
FileUtils.writeStringToFile(new File("E:/testnew${empname}.xml"), content);
}
catch (Throwable ex) {
log.info("What happened?", ex);
throw ex;
}
Instead of writing again to the disk and uploading again, how can I send the contents of string 'content' as part of request body? I have looked at many posts that talk about the input output streams but they are confusing. When I try to send just ${content} in body, the application throws following error:
HTTP Status 500 - Could not write JSON: Name is null (through reference chain: com.xxx.xxx.datafile.rest.DataFileResponse["validationStatus"]); nested exception is com.fasterxml.jackson.databind.JsonMappingException: Name is null (through reference chain:
Appreciate your help.
Multipart POST requests which are being used for files upload are different from normal POST requests hence there no possibility to simply substitute the file with the generated in-memory string.
You need to replicate the request exactly as it would be send by JMeter or real browser and manually populate each part starting from defining boundary using the HTTP Header Manager and ending up with the creation of Content-Disposition and specify your file contents there.
A little hint: you don't need to generate/substitute values for each call, it is enough to replace them once and JMeter will substitute them on its own given you use __eval() and __FileToString() functions combination.
You can check out Testing REST API File Uploads in JMeter for an example of creation a relatively complex file upload request, in your case it will be easier but still tricky.
On Gmail Api documentation i read that i've to make an HTTP request at "/upload/gmail/v1/users/userId/messages/send" when sending message larger then 5mb but i don't find any example how to implement this using the Client Library in .net
All the examples on the site refer to the "messages.Send" function that takes as parameter a raw message and the user id but i saw that there's another overload that also take the stream of the content to upload and the content type of it.
The problem is that i've no clue how to correctly call it.
Does anyone have successfully achived it ?
Thx for your answer
Simone
Simone, it's means that you use simple upload:
uploadType=media. For quick transfer of smaller files, for example, 5 MB or less.
You must use Multipart upload or Resumable upload (https://developers.google.com/gmail/api/guides/uploads)
You can send a post query with payload (see CURLOPT_POSTFIELDS, if use a CURL) on the https://www.googleapis.com/gmail/v1/users/me/messages/send?access_token=your_access_token&uploadType=multipart. Payload must contain json encoded message. Structure of this message for example:
$message = [
'message' => [
'raw' => str_replace(['+', '/'], ['-', '_'], base64_encode($mimeString)),
'threadId' => $yourThreadId
]
];
Variable $mimeString must contain correct mime string
I'd like to upload a file to S3 in parts, and set some metadata on the file. I'm using boto to interact with S3. I'm able to set metadata with single-operation uploads like so:
Is there a way to set metadata with a multipart upload? I've tried this method of copying the key to change the metadata, but it fails with the error: InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: <size>
I've also tried doing the following:
key = bucket.create_key(key_name)
key.set_metadata('some-key', 'value')
<multipart upload>
...but the multipart upload overwrites the metadata.
I'm using code similar to this to do the multipart upload.
Sorry, I just found the answer:
Per the docs:
If you want to provide any metadata describing the object being uploaded, you must provide it in the request to initiate multipart upload.
So in boto, the metadata can be set in the initiate_multipart_upload call. Docs here.
Faced such issue earlier today and discovered that there is no information on how to do that right.
The code example on how we solved that issue provided below.
$uploader = new MultipartUploader($client, $source, [
'bucket' => $bucketName,
'key' => $filename,
'before_initiate' => function (\Aws\Command $command) {
$command['ContentType'] = 'application/octet-stream';
$command['ContentDisposition'] = 'attachment';
},
]);
Unfortunately, documentation https://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html#customizing-a-multipart-upload doesn't make it clear and easy to understand that if you'd like to provide alternative meta data with multipart upload you have to go this way.
I hope that will help.