Convert writes to OutputStream into a Flux<DataBuffer> usable by ServerResponse - spring-webflux

I have a legacy library that I have to use to retrieve a file. This legacy library doesn't return in InputStream, as you usually expect for reading stuff, but it expects that it is passed an open OutputStream, that it can write to.
I have to write a Webflux REST service, that writes this OutputStream to the org.springframework.web.reactive.function.server.ServerResponse body.
legacyLib.BlobRead(outputStream); // writes the stream to an outputstream, that has to be provided by me, and somehow has to end up in the ServerResponse
Since I want to pass along the Stream directly to the ServerResponse, I probably have to do something like this, right?
ServerResponse.ok().body(magicOutpuStreamToFluxConverter(), DataBuffer.class);

Here is the part of the RequestHandler that's important. I left out some errorhandling/catching of exceptions, that might generally not be needed. Note that I publishedOn a different Scheduler for the read (or at least, that's what I wanted to do), so that this blocking read doesn't interfere with my main event thread:
private Mono<ServerResponse> writeToServerResponse(#NotNull FPTag tag) {
final long blobSize = tag.getBlobSize();
return ServerResponse.ok()
.contentType(MediaType.APPLICATION_OCTET_STREAM)
.body(Flux.<DataBuffer>create((FluxSink<DataBuffer> emitter) -> {
// for a really big blob I want to read it in chunks, so that my server doesn't use too much memory
for(int i = 0; i < blobSize; i+= tagChunkSize) {
// new DataBuffer that is written to, then emitted later
DefaultDataBuffer dataBuffer = new DefaultDataBufferFactory().allocateBuffer();
try (OutputStream outputStream = dataBuffer.asOutputStream()) {
// write to the outputstream of DataBuffer
tag.BlobReadPartial(outputStream, i, tagChunkSize, FPLibraryConstants.FP_OPTION_DEFAULT_OPTIONS);
// don't know if flushing is strictly neccessary
outputStream.flush();
} catch (IOException | FPLibraryException e) {
log.error("Error reading + writing from tag to http outputstream", e);
emitter.error(e);
}
emitter.next(dataBuffer);
}
// if blob is finished, send "complete" to my flux of DataBuffers
emitter.complete();
}, FluxSink.OverflowStrategy.BUFFER).publishOn(Schedulers.newElastic("centera")).doOnComplete(() -> closeQuietly(tag)), DataBuffer.class);
}

Related

Merging Mono and Flux in Spring WebFlux

Let's say I have a method store(Flux<DataBuffer> bufferFlux) which receives some data as a flux of DataBuffers, calculates an identifier, creates an AsynchronousFileChannel and then uses DataBufferUtils the data to the channel.
I started like this. Please note, that the following code will not work. It should just illustrate how I create a FileChannel and how I would like to write the data, while releasing used buffers and closing the channel afterwards.
public Mono<Void> store(Flux<DataBuffer> bufferFlux) {
var channelMono = Mono.defer(() -> {
try {
log.info("opening file {}", filePath);
return Mono.just(AsynchronousFileChannel
.open(filePath, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE));
} catch (IOException ex) {
log.error("error opening file", ex);
return Mono.error(ex);
}
});
// calculate identifier
// store buffers to AsynchronousFileChannel
return DataBufferUtils
.write(bufferFlux, fileChannel)
.doOnNext(DataBufferUtils.releaseConsumer())
.doFinally(f -> {
try {
fileChannel.close();
} catch (IOException ioException) {
log.error("error closing file channel", ioException);
}
})
.then();
}
The problem is, that I just started with reactive programming and have no clue how I could bring these two building blocks together, so that
the data is written to the channel
all buffers are gracefully released
the channel is closed after writing the data
the whole operation just signals complete or error (I guess this is what Mono<Void> is used for)
Can anyone help me choose the right operators or point me to a conceptual problem (perhaps there is a good reason why I cannot find a suitable operator)? :)
Thank you!

How can I use Kotlin to handle asynchronous speech recognition?

The Code A is from the artical https://cloud.google.com/speech-to-text/docs/async-recognize
It write with Java, I don't think the following code is a good code, it make the app interrupt.
while (!response.isDone()) {
System.out.println("Waiting for response...");
Thread.sleep(10000);
}
...
I'm a beginner of Kotlin. How can I use Kotlin to write the better code? maybe using coroutines ?
Code A
public static void asyncRecognizeGcs(String gcsUri) throws Exception {
// Instantiates a client with GOOGLE_APPLICATION_CREDENTIALS
try (SpeechClient speech = SpeechClient.create()) {
// Configure remote file request for FLAC
RecognitionConfig config =
RecognitionConfig.newBuilder()
.setEncoding(AudioEncoding.FLAC)
.setLanguageCode("en-US")
.setSampleRateHertz(16000)
.build();
RecognitionAudio audio = RecognitionAudio.newBuilder().setUri(gcsUri).build();
// Use non-blocking call for getting file transcription
OperationFuture<LongRunningRecognizeResponse, LongRunningRecognizeMetadata> response =
speech.longRunningRecognizeAsync(config, audio);
while (!response.isDone()) {
System.out.println("Waiting for response...");
Thread.sleep(10000);
}
List<SpeechRecognitionResult> results = response.get().getResultsList();
for (SpeechRecognitionResult result : results) {
// There can be several alternative transcripts for a given chunk of speech. Just use the
// first (most likely) one here.
SpeechRecognitionAlternative alternative = result.getAlternativesList().get(0);
System.out.printf("Transcription: %s\n", alternative.getTranscript());
}
}
}
You will have to provide some context to understand what you are trying to achieve, but it looks like coroutine is not really necessary here, as longRunningRecognizeAsync is already non-blocking and returns OperationFuture response object. You just need to decide what to do with that response, or just store Future and check it later. There is nothing implicitly wrong with while (!response.isDone()) {}, that's how Java Futures are supposed to work. Also check OperationFuture, if its normal Java Future, it should implement get() method, that will let you wait for result if necessary, without having to do explicit Thread.sleep().

WCF Streaming - who closes the file?

According to Microsoft's samples, here's how one would go about streaming a file throuhg WCF:
// Service class which implements the service contract
public class StreamingService : IStreamingSample
{
public System.IO.Stream GetStream(string data)
{
//this file path assumes the image is in
// the Service folder and the service is executing
// in service/bin
string filePath = Path.Combine(
System.Environment.CurrentDirectory,
".\\image.jpg");
//open the file, this could throw an exception
//(e.g. if the file is not found)
//having includeExceptionDetailInFaults="True" in config
// would cause this exception to be returned to the client
try
{
FileStream imageFile = File.OpenRead(filePath);
return imageFile;
}
catch (IOException ex)
{
Console.WriteLine(
String.Format("An exception was thrown while trying to open file {0}", filePath));
Console.WriteLine("Exception is: ");
Console.WriteLine(ex.ToString());
throw ex;
}
}
...
Now, how do I know who's responsible for releasing the FileStream when the transfer is done?
EDIT: If the code is put inside a "using" block the stream gets shut down before the client receives anything.
The service should clean up and not the client. WCF's default for OperationBehaviorAttribute.AutoDisposeParameters seems to be true, therefore it should do the disposing for you. Although there doesn't seem to be a fixed answer on this.
You could try using the OperationContext.OperationCompleted Event:
OperationContext clientContext = OperationContext.Current;
clientContext.OperationCompleted += new EventHandler(delegate(object sender, EventArgs args)
{
if (fileStream != null)
fileStream.Dispose();
});
Put that before your return.
Check this blog
Short answer: the calling code, via a using block.
Long answer: sample code should never be held up as an exemplar of good practice, it's only there to illustrate one very specific concept. Real code would never have a try block like that, it adds no value to the code. Errors should be logged at the topmost level, not down in the depths. Bearing that in mind, the sample becomes a single expression, File.OpenRead(filePath), that would be simply plugged into the using block that requires it.
UPDATE (after seeing more code):
Just return the stream from the function, WCF will decide when to dispose it.
The stream needs to be closed by party who is responsible to read it. For example, if service returns the stream to client, it's client application responsibility close the stream as Service doesn't know or have control when client finishes reading stream. Also, WCF will not take care of closing the stream again because of the fact that it doesn't know when receiving party has finished reading. :)
HTH,
Amit Bhatia

confuse about EndUploadFromStream

I am confusing with a problem about upload blobs asynchronously, hopes find answer here.
Please take a look at my code snippet first,
public void UploadMultipleBlobs(List<string> filelocations, string containerName, AsyncCallback callback = null, string path = null)
{
try
{
Parallel.ForEach(filelocations, fileLocation =>
{
//File to Stream
MemoryStream str = new MemoryStream();
byte[] file = File.ReadAllBytes(fileLocation);
str.Write(file, 0, file.Length);
str.Seek(0, SeekOrigin.Begin);
//Operations
if (callback == null)
callback = new AsyncCallback(OnUploadCompleted);
BlobRequestOptions blobRequestOptions = new BlobRequestOptions();
blobRequestOptions.Timeout = new TimeSpan(1, 0, 0);
blobRequestOptions.RetryPolicy = retry;
CloudBlob currentBlob = container.GetBlobReference(blobName);
var result = currentBlob.BeginUploadFromStream(str, blobRequestOptions, callback, new Object[] { currentBlob, str });
currentBlob.EndUploadFromStream(result);
});
}
catch
{
throw;
}
}
private void OnUploadCompleted(IAsyncResult result)
{
try
{
// Get array passed to callback
Object[] states = (Object[])result.AsyncState;
var blob = (CloudBlob)states[0];
var stream = (MemoryStream)states[1];
// End the operation
//blob.EndUploadFromStream(result);
// Close the stream
stream.Close();
}
catch
{
throw;
}
}
I need to upload mutil files to Azure blob, number of files may be 10-50,000, each file is about 10KB-50KB. The code snippet works fine for me currently. However, if I call EndUploadFromStream in callback, it always throw an exception when uploading over 2,000 files. I mean if I remove EndUploadFromStream in upload method and call EndUploadFromStream in callback(OnUploadCompleted method), the exception happens. The exception message as below:
Unable to read data from the transport connection: The connection was closed., StackTrace: at Microsoft.WindowsAzure.StorageClient.Tasks.Task`1.get_Result()
at Microsoft.WindowsAzure.StorageClient.CloudBlob.EndUploadFromStream(IAsyncResult asyncResult)
I don't know why it happens...hopes got answer from you guys.
Thanks.
The Begin/End code looks OK. I notice that you're not doing anything to wait for the asynchronous operations to complete, so the problem may be related to that. eg, if you're running this from a console application then the application may exit before all the uploads have completed and then give you those errors. That would not be a problem if the EndUploadFromStream() call is inside the Parallel.ForEach() because it will cause the Parallel.ForEach() call to block until all the uploads have completed.
So try adding code to wait for all the uploads to complete and see if that fixes it. A simple way would be a counter that is initialized to the total number of uploads, and then decremented (using Interlocked.Decremement() for thread safety) inside of the callback. Another options would be use Task.FromAsync() to get an array of Task objects, then using Task.WaitAll() to wait for them to complete.
As an aside, using both Parallel.ForEach() and Begin/End methods at the same time is usually redundant - Begin/End is asynchronous already so there's usually no point using multiple threads to invoke it. Since you have such a big list of items it might make some difference in this case, but probably not much. You're probably better off using a simple foreach loop instead of Parallel.ForEach() unless you've actually measured a significant difference.

Eclipse plugin: about the function --println(String) in org.eclipse.ui.console.MessageConsoleStream

Recently I am doing a eclipse plugin project with eclipse_RCP. But I encountered some issues with eclipse UI when I wanted to print a large number of messages in the console of plugin.
The messages are from a complex process which could be considered as a factory producing messages all the time and never stop (until the client stop the process of course).
When I printed the message before (the message is short), I just needed to call the function -org.eclipse.ui.console.MessageConsoleStream.println().
BUT this time ,when I tried like before at first , the runtime-EclipseApplication (launch the debug mode) stopped responding and then tell me out of memory.
It seems like that the eclipse will read all the messages in the memory and THEN print them to the console one time .So when the number of message is large ,it will out of memory.
My issue is what can I do if I want to print the message line by line in the console ?
My description may be not accurate. Below is the java code:
public void print(Process p) {
BufferedReader in = new BufferedReader(
new InputStreamReader(p.getInputStream()),1024);
String line = "";
try {
while ((line = in.readLine()) != null) {
//it is correct when print in the main console
System.out.println(line);
//when print in plugin console .it is out of memory
//this is the function
//org.eclipse.ui.console.MessageConsoleStream.println()
println(line);
}
in.close();
this.flush();
this.close();
p.destroy();
}
catch (IOException e) {
e.printStackTrace();
}
}
Then I try to write to a file at first and let the MessageConsoleStream read from the file every 1000 messages,but it looks like the same .
public void print(Process p) {
BufferedReader in = new BufferedReader(
new InputStreamReader(p.getInputStream()),1024);
String line = "";
char []tem = new char[1024];
int i = 0 ;
try {
File temp = File.createTempFile("temp", ".tep",new File("E:/"));
FileWriter out = new FileWriter(temp);
MessageConsoleStream mcs = null;
while((line = in.readLine())!=null){
if(i<=1000){
System.out.println(line);
out.write(line+"\n", 0, line.length()+1);
i++;
}
else{
i=0 ;
out.flush();
out.close();
FileReader fr=new FileReader(temp);
mcs = CConsole.getMessageStream("consoleName", "file name");
while( fr.read(tem, 0, 1024)!=-1){
mcs.print(String.valueOf(tem));
}
mcs.flush();
mcs.close();
fr.close();
out = new FileWriter(temp,false);
}
}
if(i!= 0){
mcs = CConsole.getMessageStream("consoleName", "file name");
out.flush();
out.close();
FileReader fr=new FileReader(temp);
while( fr.read(tem, 0, 1024)!=-1){
mcs.print(String.valueOf(tem));
}
mcs.flush();
mcs.close();
}
in.close();
p.destroy();
}
catch (IOException e) {
e.printStackTrace();
}
}
All the ways above will make the eclipse out of memory when the number of messages more than 600,000 (then I stop the process ,otherwise it will out of memory).
It looks like the ecplipse wants to print all of them one time but not line by line.So it reads and reads again until out of memory.
BTW,I find a note in the org.eclipse.ui.console.MessageConsoleMessage.java——
Clients should avoid writing large amounts of output to this stream
in the UI thread. The console needs to process the output in the UI
thread and if the client hogs the UI thread writing output to the
console, the console will not be able to process the output.
That is not the real reason ,isn't it ?
I also notice that both the cdt and jdt are ok when printing a large number of message .How did they do ?
THANKS!
You have to use the flush() method every so often to write the MessageConsoleStream out to the console.
The flush() method is part of the IOConsoleOutputStream class, in the org.eclipse.ui.console package. The flush() method is not well documented, so I can see how you might have missed it.