I am trying to stream a big zip file with Multi through GRPC service as follows:
#GrpcService
class HelloGrpcService : HelloGrpc {
override fun source(request: Empty?): Multi<SourceResponse> {
val file = File("/Users/developer/Downloads/Archive.zip")
val res = SourceResponse.newBuilder().setData(ByteString.readFrom(file.inputStream())).build()
return Multi.createFrom().item(res)
}
}
Unfortunately, I have received the following exception:
System.ArgumentException: The JSON value of length 212507417 is too large and not supported.
My goal is to stream the file and not to send it once.
The question is how to stream a big file in GRPC?
Here is the proto file:
syntax = "proto3";
import "google/protobuf/empty.proto";
option java_multiple_files = true;
option java_package = "io.acme";
option java_outer_classname = "HelloGrpcProto";
package hello;
service HelloGrpc {
rpc Source(google.protobuf.Empty) returns (stream SourceResponse) {}
}
message SourceResponse {
bytes data = 1;
}
Related
How can I upload large files to MinIO (AWS S3 compatible API) via gRPC service without buffering data?
I have gRPC service with following definition:
service MediaService {
rpc UploadMedia(stream UploadMediaRequest) returns (UploadMediaResponse);
}
message UploadMediaRequest {
oneof Data {
UploadMediaMetadata metadata = 1;
UploadMediaStream fileStream = 2;
}
}
message UploadMediaMetadata {
string bucket = 1;
string virtialDirectory = 2;
string fileName = 3;
string contentType = 4;
map<string, string> attributes = 6;
}
message UploadMediaStream {
bytes bytes = 1;
}
And implementation of UploadMedia:
public override async Task<UploadMediaResponse> UploadMedia(
IAsyncStreamReader<UploadMediaRequest> requestStream,
ServerCallContext context)
{
UploadMediaMetadata? metadata = null;
var token = context.CancellationToken;
var traceId = context.GetHttpContext().TraceIdentifier;
await using var memoryStream = new MemoryStream();
await foreach (var req in requestStream.ReadAllAsync(token))
{
if (req.DataCase == UploadMediaRequest.DataOneofCase.Metadata)
{
metadata = req.Metadata;
_logger.LogTrace("[Req: {TraceId}] Received metadata", traceId);
}
else
{
await memoryStream.WriteAsync(req.FileStream.Bytes.Memory, token);
_logger.LogTrace("[Req: {TraceId}] Received chunk of bytes", traceId);
}
}
if (metadata == null)
{
throw new RpcException(new Status(StatusCode.InvalidArgument, "Not found metadata."));
}
memoryStream.Seek(0L, SeekOrigin.Begin);
var uploadModel = _mapper.Map<UploadModel>(metadata);
uploadModel.FileStream = memoryStream;
var file = await _fileService.UploadFile(uploadModel, token);
await _eventsService.Notify(new MediaUploadedEvent(file.PublicId), token);
_logger.LogTrace("[Req: {TraceId}] File uploaded", traceId);
return new UploadMediaResponse { File = _mapper.Map<RpcFileModel>(file) };
}
At the method I read request stream and write data to MemoryStream. After that I upload file to storage:
var putObjectArgs = new PutObjectArgs()
.WithStreamData(fileStream)
.WithObjectSize(fileStream.Length)
.WithObject(virtualPath)
.WithBucket(bucket)
.WithContentType(contentType)
.WithHeaders(attributes);
return _storage.PutObjectAsync(putObjectArgs, token);
I want to upload files without buffering data in Memory.
I think I can write bytes from stream to disk and after that create FileStream, but I don't want one more dependency.
I'm trying to save pdf in wildfly, I'm using RestEasy MultipartFormDataInput provided with wildfly 20.0.1,
but it doesn't work.
This is what I have:
public static Response uploadPdfFile(MultipartFormDataInput multipartFormDataInput) {
// local variables
MultivaluedMap<String, String> multivaluedMap = null;
String fileName = null;
InputStream inputStream = null;
String uploadFilePath = null;
try {
Map<String, List<InputPart>> map = multipartFormDataInput.getFormDataMap();
List<InputPart> lstInputPart = map.get("poc");
for(InputPart inputPart : lstInputPart){
// get filename to be uploaded
multivaluedMap = inputPart.getHeaders();
fileName = getFileName(multivaluedMap);
if(null != fileName && !"".equalsIgnoreCase(fileName)){
try {
// write & upload file to UPLOAD_FILE_SERVER
//here I have the error: Unable to find a MessageBodyReader for media type:
//application/pdf
inputStream = inputPart.getBody(InputStream.class,InputStream.class);
uploadFilePath = writeToFileServer(inputStream, fileName);
}catch (Exception e) {
e.printStackTrace();
}
// close the stream
inputStream.close();
}
}
}
catch(IOException ioe){
ioe.printStackTrace();
}
finally{
// release resources, if any
}
return Response.ok("File uploaded successfully at " + uploadFilePath).build();
}
I'm using postman for test, http POST method, in the body I send: form-data - file and selected the file.pdf.
When I sent the request, I have the next RunTimeException when I try:
inputStream = inputPart.getBody(InputStream.class,null);
I get:
java.lang.RuntimeException: RESTEASY007545: Unable to find a MessageBodyReader for media type: application/pdf and class type org.jboss.resteasy.util.Base64$InputStream
At the moment I am saving the file receiving it in Base64, but I think that with MultipartFormDataInput it is the correct way.
This is what I have when debug:
Thanks for your support.
I solved this changing the InputStream from "org.jboss.resteasy.util.Base64.InputStream"
to "java.io.InputStream"
I am currently using the following to upload the files selected. However, I would like to use the file name of the local file for the server file name. I have not been able to discover how to recover that name.
public async Task ReadFile()
{
foreach (var file in await fileReaderService.CreateReference(inputTypeFileElement).EnumerateFilesAsync())
{
// Read into buffer and act (uses less memory)
await using (Stream stream = await file.OpenReadAsync())
{
buffer = new Byte[stream.Length];
// Do (async) stuff with stream...
await stream.ReadAsync(buffer);
}
// Read file fully into memory and act
using (MemoryStream memoryStream = await file.CreateMemoryStreamAsync(4096))
{
// Sync calls are ok once file is in memory
memoryStream.Read(buffer);
docManager.WriteImageToFile(memoryStream, upLoadFileName, Season);
}
}
}
How can I get the local file name in code?
You can get it from the FileInfo using the following code inside your foreach-loop:
IFileInfo fileInfo = await file.ReadFileInfoAsync();
string fileName = fileInfo.Name;
string contentType = fileInfo.Type;
MFP 8, Javascript adapters should read the JSON file (TestData.json) placed in
"\src\main\adapter-resources"
I need to read the static array contents from json/text file within the same JAVASCRIPT HTTP adapter. How to do the same ?
Read the static arrays from file (Text/JSON) within same adapter folder.
As explained in this solution, you can read a JSON file with Javascript as shown below.
function readFile(filename) {
var content = "";
var fileReader = new java.io.FileReader(filename);
var bufferedReader = new java.io.BufferedReader(fileReader);
var line;
while((line = bufferedReader.readLine()) != null) {
content += line;
}
bufferedReader.close();
return content;
}
function test() {
var file = 'yourfilename.json';
var fileContents;
try {
fileContents = JSON.parse(readFile(file));
} catch(ex) {
// handle error
}
return {
fileContents: fileContents
};
Ok, so I am writing a service to recieve file uploads from an iPhone application through phonegap. They send me a file and I am trying to grab the actual file without saving it to any type of file system. Currently this is what I have
[HttpPost]
public string processRequest()
{
string ext = "Entered";
Request.Content.ReadAsMultipartAsync<MultipartMemoryStreamProvider>(new MultipartMemoryStreamProvider()).ContinueWith((tsk) =>
{
ext = "Request";
MultipartMemoryStreamProvider prvdr = tsk.Result;
foreach (HttpContent ctnt in prvdr.Contents)
{
ext = "Foreach";
// You would get hold of the inner memory stream here
Stream stream = ctnt.ReadAsStreamAsync().Result;
if (stream == null)
{
ext = "Null Stream";
}
Image img = Image.FromStream(stream);
if (ImageFormat.Jpeg.Equals(img.RawFormat))
{
ext = "jpeg";
}
else if (ImageFormat.Png.Equals(img.RawFormat))
{
ext = "Png";
}
else if (ImageFormat.Gif.Equals(img.RawFormat))
{
ext = "Gif";
}
// do something witht his stream now
}
});
return ext;
}
I have put various responses in there so I can see where the function is getting to. Right now it always returns "Entered" which means its not even reading the content of the request, the end game is for me to grab the file object, convert it into an image and then to base 64. Any direction would be appreciated. Remember I want to do this without any file system so no solutions that involve mapping a path to a server folder.
Ok so a little update, I have edited my code according to my first response and at least it attempts to execute now but it just gets infinitely stuck inside the code. This happens during the ReadAsMultipartAsync function
[HttpPost]
public string processRequest()
{
string ext = "Entered";
Request.Content.ReadAsMultipartAsync(new MultipartMemoryStreamProvider()).ContinueWith((tsk) =>
{
ext = "Request";
MultipartMemoryStreamProvider prvdr = tsk.Result;
foreach (HttpContent ctnt in prvdr.Contents)
{
ext = "Foreach";
// You would get hold of the inner memory stream here
Stream stream = ctnt.ReadAsStreamAsync().Result;
if (stream == null)
{
ext = "Null Stream";
}
Image img = Image.FromStream(stream);
if (ImageFormat.Jpeg.Equals(img.RawFormat))
{
ext = "jpeg";
}
else if (ImageFormat.Png.Equals(img.RawFormat))
{
ext = "Png";
}
else if (ImageFormat.Gif.Equals(img.RawFormat))
{
ext = "Gif";
}
// do something witht his stream now
}
}).Wait();
return ext;
}
The block inside ContinueWith also runs asynchronously (if you look at the signature for ContinueWith, you'll see that it returns a Task as well). So, with the above code, essentially you're returning before any of that has a chance to execute.
Try doing:
Request.Content.ReadAsMultipartAsync().ContinueWith(...).Wait();
Also, not sure you need to go to the trouble of doing Request.Content.ReadAsMultipartAsync<MultipartMemoryStreamProvider>(new MultipartMemoryStreamProvider()); I believe Request.Content.ReadAsMultipartAsync() should suffice.
Hope that helps!