Moving files on moveFailed isn't work - file-io

I'm new in working with Apache Camel. Could you help me with moving files? I have such route:
from("file:data?noop=true?move={{package.success}}&moveFailed={{package.failed}}")
.split(ExpressionBuilder.beanExpression(new InvoiceIteratorFactory(), "createIterator"))
.streaming()
.process(new ValidatorProcessor())
.choice()
.when(new Predicate() {
#Override
public boolean matches(Exchange exchange) {
..;
}
})
.to("jpa://...?consumer.transacted=true")
.otherwise()
.aggregate(header(PropertyNameConstants.AGGREGATOR_HEADER), new ErrorsAggregationStrategy())
.completionPredicate(new Predicate() {
#Override
public boolean matches(Exchange exchange) {
...;
}
})
.to("smtps://smtp.gmail.com?username={{remote.e-mail}}&password={{remote.password}}");
So, files with errors should be moved to directory "failed" and files without errors to directory "success". I try to generate exception after aggregating required messages (while parsing file with errors), so that to move file to directory "failed", but all files moved to directory "success", even there was an exception.
If I throw exception before aggregator, file moved to "failed" directory but last "to" (sending mail) isn't work.

If you have a copy of Camel in Action, then I suggest to read chapter 8 about the aggregator EIP to understand how it works, and the fact its a stateful EIP, so there is a handoff of the Exchange, so the consumer completes. And the aggregated exchange that comes out of the aggregator is executed independently from the original consumed exchange.
Also you may want to look at the composed message processor eip, and use the spliiter only version
http://camel.apache.org/composed-message-processor.html

Related

Spring Integration - how to delegate error handling to separate thread

My Spring Integration Flow looks like the below,
Read file from directory
Add a header called errorChannel:'exceptionChannel' to the message
Perform business logic
Print output to a 'success directory'
If there is an error on step 3, the exception is sent to the exceptionChannel,and is written to a 'failed directory'.
I want the error flow to be delegated to a separate thread.
What I have:
If there are 5 files and the third file has an error,
-> 2 files are written to the success directory
-> 1 file is written to the failed directory.
-> The flow stops after the error file.
What I need:
If there are 5 files and the third file has an error,
-> First 2 files must be written to success directory
-> 3rd file must be written to failed directory
-> Last 2 files must be written to success directory
Code for Success flow:
#Bean(name="exceptionChannel")
MessageChannel exceptionChannel() {
return MessageChannels.direct()
.get();
}
#Bean
public IntegrationFlow migrateInputToOutput() {
return IntegrationFlows.from(Files.inboundAdapter(new File(INPUT_DIR))),
.enrichHeaders(h -> h.header("errorChannel", "exceptionChannel", true))
.handle(someBusinessLogic) // ANY EXCEPTION HERE IS SENT TO exceptionChannel
.handle(Files.outboundAdapter(new File(OUTPUT_SUCCESS_DIR))
.get();
}
Code for handling error:
#Bean
public IntegrationFlow handleErrorInMigration() {
return IntegrationFlows.from("exceptionChannel"),
.handle(errorLogicToPrintException)
.handle(Files.outboundAdapter(new File(OUTPUT_ERROR_DIR))
.get();
}
The Files.inboundAdapter() is pollable source, so there is somewhere something like Poller configured in your code. That one has a errorChannel() option. It is really would be better to do that way. If your poller is global one and you don't want to modify it, it is better to have a poller configured exactly for this endpoint:
IntegrationFlows.from(Files.inboundAdapter(new File(INPUT_DIR)),
e -> e.poller(p -> p.fixedDelay().errorChannel(exceptionChannel())))
This way you won't effect all other polling endpoints.
To move your error handling to different thread, you need to consider to make that exceptionChannel as an ExecutorChannel:
#Bean(name="exceptionChannel")
MessageChannel exceptionChannel() {
return MessageChannels.executor(myExecutor)
.get();
}

How to set custom headers on RabbitMQ message using Apache Camel?

I'm trying to add custom headers on my message, so whenever an exception occurs and it ends up in the dead-letter-queue, I can see what the exception was. However all my attempts at this have failed.
using .setHeader()
setting header on the outMessage
setting property of the exchange
Setting the exception as a property in the payload is not allowed.
#Component
public class ProcessRoute extends RouteBuilder {
...
#Override
public void configure() throws Exception {
onException(Exception.class)
.log("Error for ${body}! Requeue")
.redeliveryDelay(2000)
.maximumRedeliveries(3)
.handled(true)
.setHeader("TEST", constant("TEST"))
.process(e -> {
e.getOut().setHeader("TEST", "TEST");
e.setProperty("TEST","TEST");
});
from(SOME_ROUTE)
.doSomeStuff()
.to(RABBITMQ);
}
...
}
RABBITMQ-string:
rabbitmq://foo
?exchangeType=topic
&addresses=localhost:1234
&routingKey=#
&autoDelete=false
&queue=bar
&autoAck=false
&deadLetterExchange=DLX
&deadLetterQueue=bar.dlq
&deadLetterExchangeType=direct
&deadLetterRoutingKey=#
&username=foo
&password=bar
Resulting message on the dead-letter-queue:
If you use a header key following the pattern that the Camel RabbitMQ component has established, then your custom header will get picked up when the message is published to RabbitMQ.
Taking from your code above, instead of:
.setHeader("TEST", constant("TEST"))
Do this:
.setHeader("rabbitmq.TEST", constant("TEST"))
The Camel RabbitMQ component seems to ignore all the other non- "rabbitmq.*" headers that might be on the Camel exchange, and probably for good reason. There could be quite a few and most of them wouldn't make sense in the context of a message published to RabbitMQ.

Camel aws-s3 not working

I am trying to create a camel route to transfer a file from an FTP server to an AWS S3 storage.
I have written the following route
private static class MyRouteBuilder extends RouteBuilder {
#Override
public void configure() throws Exception
{
from("sftp://<<ftp_server_name>>&noop=true&include=<<file_name>>...")
.process(new Processor(){
#Override
public void process(Exchange ex)
{
System.out.println("Hello");
}
})
.to("aws-s3://my-dev-bucket ?
accessKey=ABC***********&secretKey=12abc********+**********");
}
The issue is, this gives me the following exception:
Exception in thread "main" org.apache.camel.FailedToCreateRouteException: Failed to create route route1 at: >>> To[aws-s3://my-dev-bucket?accessKey=ABC*******************&secretKey=123abc******************** <<< in route: Route(route1)[[From[sftp://<<ftp-server>>... because of Failed to resolve endpoint: aws-s3://my-dev-bucket?accessKey=ABC***************&secretKey=123abc************** due to: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I then tried to do this the other way. i.e.writing a method like this:
public void boot() throws Exception {
// create a Main instance
main = new Main();
// enable hangup support so you can press ctrl + c to terminate the JVM
main.enableHangupSupport();
// bind MyBean into the registery
main.bind("foo", new MyBean());
// add routes
AWSCredentials awsCredentials = new BasicAWSCredentials("ABC*****************", "123abc*************************");
AmazonS3 client = new AmazonS3Client(awsCredentials);
//main.bind("client", client);
main.addRouteBuilder(new MyRouteBuilder());
main.run();
}
and invoking using the bound variable #client. This approach does not give any exceptions, but the file transfer does not work.
To make sure that there's nothing wrong with my approach, I tried aws-sqs instead of aws-s3 and that works fine (file succesfully transfers to the SQS queue)
Any idea why this is happening? Is there some basic issue with "aws-s3" connector for camel?
Have you tried of using RAW() function to wrap as like RAW(secretkey or accesskey).
It will help you to pass your keys as it is without encoding.
Any plus signs in you secret key need to be url encoded as %2B, in your case **********+*********** becomes **********%2B***********
When you configure Camel endpoints using URIs then the parameter values gets url encoded by default.
This can be a problem when you want to configure passwords as is.
To do that you can tell Camel to use the raw value, by enclosing the value with RAW(value). See more details at How do I configure endpoints which has an example also.
See Camel Documentation
Your url should looks like:
aws-s3:bucketName?accessKey=RAW(XXXX)&secretKey=RAW(XXXX)

Apache Http Client Put Request Error

I'm trying to upload a file using the Apache Http Client's PUT method. The code is as below;
def putFile(resource: String, file: File): (Int, String) = {
val httpClient = new DefaultHttpClient(connManager)
httpClient.getCredentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(un, pw))
val url = address + "/" + resource
val put = new HttpPut(url)
put.setEntity(new FileEntity(file, "application/xml"))
executeHttp(httpClient, put) match {
case Success(answer) => (answer.getStatusLine.getStatusCode, "Successfully uploaded file")
case Failure(e) => {
e.printStackTrace()
(-1, e.getMessage)
}
}
}
When I tried running the method, I get to see the following error:
org.apache.http.NoHttpResponseException: The target server failed to respond
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:101)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:252)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:247)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:219)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:298)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:633)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:454)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
I do not know what has gone wrong? I'm able to do GET requests, but PUT seems not to work! Any clues as to where I should look for?
Look on the server. If GET Works, but PUT does not, then you have to figure out the receiving end.
Also, you may want to write a simple HTML File that has a form with PUT Method in it to rule out your Java Part.
As a sidenode: Its technically possible that something in between stops the request from going through or the response reaching you. Best setup a dummy HTTP Server to do the testing against.
Maybe its also a timeout issue, so the server takes to long to process your PUT.
The connection you are trying to use is a stale connection and therefore the request is failing.
But why are you only seeing an error for the PUT request and you are not seeing it for the GET request?
If you check the DefaultHttpRequestRetryHandler class you will see that by default HttpClient attempts to automatically recover from I/O exceptions. The default auto-recovery mechanism is limited to just a few exceptions that are known to be safe.
HttpClient will make no attempt to recover from any logical or HTTP protocol errors (those derived from HttpException class).
HttpClient will automatically retry those methods that are assumed to be idempotent. Your GET request, but not your PUT request!!
HttpClient will automatically retry those methods that fail with a transport exception while the HTTP request is still being transmitted to the target server (i.e. the request has not been fully transmitted to the server).
This is why you don't notice any error with your GET request, because the retry mechanism handles it.
You should define a CustomHttpRequestRetryHandler extending the DefaultHttpRequestRetryHandler. Something like this:
public class CustomHttpRequestRetryHandler extends DefaultHttpRequestRetryHandler {
#Override
public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
if(exception instanceof NoHttpResponseException) {
return true;
}
return super.retryRequest(exception, executionCount, context);
}
}
Then just assign your CustomHttpRequestRetryHandler
final HttpClientBuilder httpClientBuilder = HttpClients.custom();
httpClientBuilder.setRetryHandler(new CustomHttpRequestRetryHandler());
And that's it, now your PUT request is handled by your new RetryHandler (like the GET was by the default one)

Multiple subscribers in NServiceBus

I'm getting started with NServiceBus and have a question about the Pubsub sample.
My intention was to have multiple instances of Publisher1 running and receiving the message sent from the publisher. I also hacked the Publisher to only send messages of the eventMessage type.
But if I start the publisher and three instances of Subscriber1, only one of them gets the message at a time.
why is that? Is it a config setting or something else?
This is what I've tried which returns an exception "Exception when starting endpoint, error has been logged. Reason: Cannot configure property before the component has been configured. Please call 'Configure' first.":
using NServiceBus;
namespace Subscriber1
{
public class EndpointConfig : IConfigureThisEndpoint, AsA_Server
{
}
public class OverrideInputQueue : IWantCustomInitialization
{
public void Init()
{
Configure
.Instance
.Configurer
.ConfigureProperty<NServiceBus.Config.MsmqTransportConfig>(t => t.InputQueue, "testQueue");
}
}
}
/J
NServiceBus assumes that you have one input queue per process. Make sure that each of your subscribers are configured with a unique input queue. If not all three will be polling the same queue producing the behavior you're describing.
To do this you would probably have to copy paste sub1 to 3 different folders, modfying the app.config and start them up.
Hope this helps!
You should use this-
Configure.Instance.Configurer.ConfigureProperty<NServiceBus.Unicast.Transport.Msmq.MsmqTransport>(msmq => msmq.InputQueue, "SomeQueueHere");
Make sure you use MsmqTransport and not MsmqTransportConfig as you mentioned.