Spring Integration - how to delegate error handling to separate thread - error-handling

My Spring Integration Flow looks like the below,
Read file from directory
Add a header called errorChannel:'exceptionChannel' to the message
Perform business logic
Print output to a 'success directory'
If there is an error on step 3, the exception is sent to the exceptionChannel,and is written to a 'failed directory'.
I want the error flow to be delegated to a separate thread.
What I have:
If there are 5 files and the third file has an error,
-> 2 files are written to the success directory
-> 1 file is written to the failed directory.
-> The flow stops after the error file.
What I need:
If there are 5 files and the third file has an error,
-> First 2 files must be written to success directory
-> 3rd file must be written to failed directory
-> Last 2 files must be written to success directory
Code for Success flow:
#Bean(name="exceptionChannel")
MessageChannel exceptionChannel() {
return MessageChannels.direct()
.get();
}
#Bean
public IntegrationFlow migrateInputToOutput() {
return IntegrationFlows.from(Files.inboundAdapter(new File(INPUT_DIR))),
.enrichHeaders(h -> h.header("errorChannel", "exceptionChannel", true))
.handle(someBusinessLogic) // ANY EXCEPTION HERE IS SENT TO exceptionChannel
.handle(Files.outboundAdapter(new File(OUTPUT_SUCCESS_DIR))
.get();
}
Code for handling error:
#Bean
public IntegrationFlow handleErrorInMigration() {
return IntegrationFlows.from("exceptionChannel"),
.handle(errorLogicToPrintException)
.handle(Files.outboundAdapter(new File(OUTPUT_ERROR_DIR))
.get();
}

The Files.inboundAdapter() is pollable source, so there is somewhere something like Poller configured in your code. That one has a errorChannel() option. It is really would be better to do that way. If your poller is global one and you don't want to modify it, it is better to have a poller configured exactly for this endpoint:
IntegrationFlows.from(Files.inboundAdapter(new File(INPUT_DIR)),
e -> e.poller(p -> p.fixedDelay().errorChannel(exceptionChannel())))
This way you won't effect all other polling endpoints.
To move your error handling to different thread, you need to consider to make that exceptionChannel as an ExecutorChannel:
#Bean(name="exceptionChannel")
MessageChannel exceptionChannel() {
return MessageChannels.executor(myExecutor)
.get();
}

Related

Ktor server Progress on incoming files

I have a ktor server running on android and I would like to display a progress bar for files being uploaded to the server via POST.
In my each part handling code I have something like this:
call.receiveMultipart().forEachPart { part ->
when (part) {
is PartData.FileItem -> {
val input = part.provider()
// Read from input by chunks
}
else -> {}
}
}
but I the when part is called after the file is uploaded to the server.
I had a look at (custom) Plugins and Logging to see if I can get any progress update on incoming bytes but no luck.

Need Elastic APM support for Ktor backend server

Trying to monitor performance of our Ktor backend application and are able to attach Elastic APM agent to it. Server is visible at Kibana dashboard as a service. But it's not creating transactions automatically for each incoming request. When we manually start a transaction and end it in a specific route, then only it is recording performance for that request. Is there another way to solve this situation?
Tried following approach
Intercepted each request in setup phase and started a transaction, but could not end the transaction facing issue while intercepting the same call at the end.
For each request in controller/route defined below piece of code and it is working.
get("/api/path") {
val transaction: Transaction = ElasticApm.startTransaction()
try {
transaction.setName("MyTransaction#getApi")
transaction.setType(Transaction.TYPE_REQUEST)
// do business logic and response
} catch (e: java.lang.Exception) {
transaction.captureException(e)
throw e
} finally {
transaction.end()
}
}
Adding below line for better search result for other developers.
How to add interceptor on starting and ending on each request in ktor. Example of ApplicationCallPipeline.Monitoring and proceed()
You can use the proceed method that executes the rest of a pipeline to catch any occurred exceptions and finish a transaction:
intercept(ApplicationCallPipeline.Monitoring) {
val transaction: Transaction = ElasticApm.startTransaction()
try {
transaction.setName("MyTransaction#getApi")
transaction.setType(Transaction.TYPE_REQUEST)
proceed() // This will call the rest of a pipeline
} catch (e: Exception) {
transaction.captureException(e)
throw e
} finally {
transaction.end()
}
}
Also, you can use attributes to store a transaction for a call duration (begins when the request has started and ends when the response has been sent).

Camel aws-s3 not working

I am trying to create a camel route to transfer a file from an FTP server to an AWS S3 storage.
I have written the following route
private static class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() throws Exception
{
from("sftp://<<ftp_server_name>>&noop=true&include=<<file_name>>...")
.process(new Processor(){
#Override
public void process(Exchange ex)
{
System.out.println("Hello");
}
})
.to("aws-s3://my-dev-bucket ?
accessKey=ABC***********&secretKey=12abc********+**********");
}
The issue is, this gives me the following exception:
Exception in thread "main" org.apache.camel.FailedToCreateRouteException: Failed to create route route1 at: >>> To[aws-s3://my-dev-bucket?accessKey=ABC*******************&secretKey=123abc******************** <<< in route: Route(route1)[[From[sftp://<<ftp-server>>... because of Failed to resolve endpoint: aws-s3://my-dev-bucket?accessKey=ABC***************&secretKey=123abc************** due to: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I then tried to do this the other way. i.e.writing a method like this:
public void boot() throws Exception {
// create a Main instance
main = new Main();
// enable hangup support so you can press ctrl + c to terminate the JVM
main.enableHangupSupport();
// bind MyBean into the registery
main.bind("foo", new MyBean());
// add routes
AWSCredentials awsCredentials = new BasicAWSCredentials("ABC*****************", "123abc*************************");
AmazonS3 client = new AmazonS3Client(awsCredentials);
//main.bind("client", client);
main.addRouteBuilder(new MyRouteBuilder());
main.run();
}
and invoking using the bound variable #client. This approach does not give any exceptions, but the file transfer does not work.
To make sure that there's nothing wrong with my approach, I tried aws-sqs instead of aws-s3 and that works fine (file succesfully transfers to the SQS queue)
Any idea why this is happening? Is there some basic issue with "aws-s3" connector for camel?
Have you tried of using RAW() function to wrap as like RAW(secretkey or accesskey).
It will help you to pass your keys as it is without encoding.
Any plus signs in you secret key need to be url encoded as %2B, in your case **********+*********** becomes **********%2B***********
When you configure Camel endpoints using URIs then the parameter values gets url encoded by default.
This can be a problem when you want to configure passwords as is.
To do that you can tell Camel to use the raw value, by enclosing the value with RAW(value). See more details at How do I configure endpoints which has an example also.
See Camel Documentation
Your url should looks like:
aws-s3:bucketName?accessKey=RAW(XXXX)&secretKey=RAW(XXXX)

Moving files on moveFailed isn't work

I'm new in working with Apache Camel. Could you help me with moving files? I have such route:
from("file:data?noop=true?move={{package.success}}&moveFailed={{package.failed}}")
.split(ExpressionBuilder.beanExpression(new InvoiceIteratorFactory(), "createIterator"))
.streaming()
.process(new ValidatorProcessor())
.choice()
.when(new Predicate() {
#Override
public boolean matches(Exchange exchange) {
..;
}
})
.to("jpa://...?consumer.transacted=true")
.otherwise()
.aggregate(header(PropertyNameConstants.AGGREGATOR_HEADER), new ErrorsAggregationStrategy())
.completionPredicate(new Predicate() {
#Override
public boolean matches(Exchange exchange) {
...;
}
})
.to("smtps://smtp.gmail.com?username={{remote.e-mail}}&password={{remote.password}}");
So, files with errors should be moved to directory "failed" and files without errors to directory "success". I try to generate exception after aggregating required messages (while parsing file with errors), so that to move file to directory "failed", but all files moved to directory "success", even there was an exception.
If I throw exception before aggregator, file moved to "failed" directory but last "to" (sending mail) isn't work.
If you have a copy of Camel in Action, then I suggest to read chapter 8 about the aggregator EIP to understand how it works, and the fact its a stateful EIP, so there is a handoff of the Exchange, so the consumer completes. And the aggregated exchange that comes out of the aggregator is executed independently from the original consumed exchange.
Also you may want to look at the composed message processor eip, and use the spliiter only version
http://camel.apache.org/composed-message-processor.html

Apache Http Client Put Request Error

I'm trying to upload a file using the Apache Http Client's PUT method. The code is as below;
def putFile(resource: String, file: File): (Int, String) = {
val httpClient = new DefaultHttpClient(connManager)
httpClient.getCredentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(un, pw))
val url = address + "/" + resource
val put = new HttpPut(url)
put.setEntity(new FileEntity(file, "application/xml"))
executeHttp(httpClient, put) match {
case Success(answer) => (answer.getStatusLine.getStatusCode, "Successfully uploaded file")
case Failure(e) => {
e.printStackTrace()
(-1, e.getMessage)
}
}
}
When I tried running the method, I get to see the following error:
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:101)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:633)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:454)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
I do not know what has gone wrong? I'm able to do GET requests, but PUT seems not to work! Any clues as to where I should look for?
Look on the server. If GET Works, but PUT does not, then you have to figure out the receiving end.
Also, you may want to write a simple HTML File that has a form with PUT Method in it to rule out your Java Part.
As a sidenode: Its technically possible that something in between stops the request from going through or the response reaching you. Best setup a dummy HTTP Server to do the testing against.
Maybe its also a timeout issue, so the server takes to long to process your PUT.
The connection you are trying to use is a stale connection and therefore the request is failing.
But why are you only seeing an error for the PUT request and you are not seeing it for the GET request?
If you check the DefaultHttpRequestRetryHandler class you will see that by default HttpClient attempts to automatically recover from I/O exceptions. The default auto-recovery mechanism is limited to just a few exceptions that are known to be safe.
HttpClient will make no attempt to recover from any logical or HTTP protocol errors (those derived from HttpException class).
HttpClient will automatically retry those methods that are assumed to be idempotent. Your GET request, but not your PUT request!!
HttpClient will automatically retry those methods that fail with a transport exception while the HTTP request is still being transmitted to the target server (i.e. the request has not been fully transmitted to the server).
This is why you don't notice any error with your GET request, because the retry mechanism handles it.
You should define a CustomHttpRequestRetryHandler extending the DefaultHttpRequestRetryHandler. Something like this:
public class CustomHttpRequestRetryHandler extends DefaultHttpRequestRetryHandler {
#Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
if(exception instanceof NoHttpResponseException) {
return true;
}
return super.retryRequest(exception, executionCount, context);
}
}
Then just assign your CustomHttpRequestRetryHandler
final HttpClientBuilder httpClientBuilder = HttpClients.custom();
httpClientBuilder.setRetryHandler(new CustomHttpRequestRetryHandler());
And that's it, now your PUT request is handled by your new RetryHandler (like the GET was by the default one)