As I have looked , camel sql-component is not supporting cron expressions but fixed delays etc. I have checked source code of the component but I could not find an easy way to customize it. Is there any other way to make it or should I extend all component, endpoint , consumer and producer in order to make it?
Thanks
See the documentation about polling consumer: http://camel.apache.org/polling-consumer.html at the section further below for scheduled poll consumers.
You can configure to use a different scheduler such as spring/quartz2 that has cron capabilities.
I blogged about how to do this: http://www.davsclaus.com/2013/08/apache-camel-212-even-easier-cron.html but it should work with the sql component also.
I second #Neron's comment above. I believe this is a bug in the camel-sql compnent. I am currently using version 2.16.2, but I don't see any changes in a higher version that would have resolved this.
For those interested, you can work around this by creating a subclass of the SQLComponent like this.
public class SQLComponentPatched extends SqlComponent {
#Override
protected Endpoint createEndpoint(String uri, String remaining, Map<String, Object> parameters) throws Exception {
Endpoint endpoint = super.createEndpoint(uri, remaining, parameters);
setProperties(endpoint, parameters);
return endpoint;
}
}
Related
If I have a simple IntegrationFlow like this:
#Bean
public IntegrationFlow downloadFlow() {
return IntegrationFlows.from("rabbitQueue1")
.handle(longRunningMessageHandler)
.channel("rabbitQueue2")
.get();
}
... and if the rabbitQueue1 is filled with messages,
what should I do to handle multiple messages at the same time? Is that possible?
It seems that by default, the handler executes one message at a time.
Yes, that's true, by default endpoints are wired with DirectChannel. That's like performing a plain Java instructions one by one. So, to do some parallel job in Java you need an Executor to shift a call to the separate thread.
The same is possible with Spring Integration via an ExecutiorChannel. You can make that rabbitQueue1 as an ExecutorChannel bean or use this instead of that plain name:
IntegrationFlows.from(MessageChannels.executor("rabbitQueue1", someExecturorBean)
and all the messages arriving to this channel are going to be paralleled in the threads provided by an executor. That longRunningMessageHandler are going to process your messages in parallel.
See more info in the Reference Manual: https://docs.spring.io/spring-integration/docs/current/reference/html/#channel-implementations
I am using Apache Flinke 1.7.2 RabbitMQ connector: https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/rabbitmq.html
I want to access the message_id in the amqp message properties sent along with the body of the amqp message. i want to be able to group by that message ID. The problem is that i only get the body of the message out from the source after i build it.
Is there an easy way that doesn't require me to rewrite the source class from scratch ?
I guess this is not possible. Looking at the source-code of the connector you can see that they extract only the body of the RMQ message:
#Override
public void run(SourceContext<OUT> ctx) throws Exception {
while (running) {
QueueingConsumer.Delivery delivery = consumer.nextDelivery();
synchronized (ctx.getCheckpointLock()) {
OUT result = schema.deserialize(delivery.getBody());
// ....
ctx.collect(result);
}
}
}
I guess you have to find another connector (3rd party) or implement this by your own. Sorry for the bad news!
May be this issue is already reported and resolved .I didn't find the solution and any open issues which talk about this, so creating new one.
I am trying to handle error while publishing data to kafka topic.
With kafka spring steam we are pushing to kafka by using this
if (processor.output().send(messsage , kafkaTimeoutInMS) && acknowledgment != null)
{
LOGGER.debug("Acknowledgment provided");
LOGGER.info("Sending to Kafka successful");
acknowledgment.acknowledge();
}
else
{
LOGGER.error("Sending to Kafka failed", message);
}
Send() method always returns true, I tried stopping kafka manual while running in debug mode, but still it returns true. I have that read it is asynchronous.
I Tried setting
bindings: output: producer: sync: true
This didnt help.
But I see some error which I cant use in my logic to decide whether there is failure or success.
We are manually acknowledging hence we are only supposed to acknowledge when its sent to topic successfully and we need to log all failed messages.
Any suggestions?
I believe you've misinterpreted on how spring-cloud-stream works.
As a framework there is certain contract between the user and the framework and when it comes to messaging the acks, retries, DLQ and many more aspects are handled automatically to ensure the user doesn't have to be exposed to this manually (as you are trying to do).
Consider spending a little time and going through the user guide - https://docs.spring.io/spring-cloud-stream/docs/Fishtown.M3/reference/htmlsingle/
Also, here is the very basic example that will demonstrates a typical interaction of user(developer) with the framework. As you can see, all you're doing is implementing a simple handler which receives and returns a piece of data. The rest (the actual receive from Kafka and send to Kafka or any other messaging system) is handled by the framework provided binders.
#SpringBootApplication
#EnableBinding(Processor.class)
public class ProcessorApplication {
public static void main(String[] args) {
SpringApplication.run(ProcessorApplication.class);
}
#StreamListener(Processor.INPUT)
#SendTo(Processor.OUTPUT)
public String echo(String message) {
return message;
}
}
I have read multiple times the documentation on the website. I am reading again and again the same articles and I cannot understand what they are trying to achieve with sagas. Besides, there are almost no resources in internet related to this subject.
But I am completely stuck trying to understand the purpose and benefit of defining so called sagas. I understand handlers (IHandleMessages) - these are interceptors. But I can't understand what Saga is for. The language in the documentation assumes that I am supposed to know something special to grasp that idea, but I dont.
Can someone explain to me in simple words, hopefully with real-life example a situation where I must or should define Saga, and what is the benefit of doing so? I have created an app with multiple endpoints and Saga definition as shown in samples, it works (I guess) but I don't understand what these sagas were defined for... In many samples they use RequestTimeout() method in Saga class. Why, why would anyone want to cause a timeout intentionally? I dont want to put any code fragments here, because its unrelated, I need to understand why I would want to use "Sagas" whatever that means?
Thank you.
NServiceBus Saga is a variant of a Process Manager described in the Enterprise Integration Patterns book.
To understand when to use Saga, one has to need it. Let's assume you're using regular message handlers only to implement new user registration process. At some point in time, you discover that only 40% of the brand-new registrants confirm their email address and becoming active user accounts. There are two things you'd like to address.
Remind new registrants to confirm their email after 24 hours after registration by sending a reminder.
Remove registrant info (email for example) from the data store to be compliant with GDPR within 48 hours.
Now how do you do that with a regular message handler? A handler would receive the initial request (first message, m1) to kick off registration by generating an email with a confirmation link and that's it. Once the handler is done, it's done for good. But your process is not finished. It's a long-running logical process that has to span 48 hours before completed. It's no longer just a single message processing, but a workflow at this point. A workflow with multiple checkpoints. Similar to a state machine. To move from one state to another, a certain condition has to be fulfilled. In case of NServiceBus, those would be messages. A message to send a reminder after 24 hours (let's call it m2) is not going to be triggered by any user action. It's a "system" message. A timed message that should be kicked off automatically. So is with the message to instruct the system to remove registrant information if validation link was not activated. The theme can be observed: need to schedule messages in the future to re-hydrate the workflow and continue from the state it was left last time.
That's what timeouts are. Those are requests to re-hydrate/continue saga/workflow from the point it was left last time at a certain point in time - minutes, hours, days, months, years.
This is what this kind of workflow would look like as a saga (oversimplified and doesn't take into consideration all the edge cases).
class RegistrationWorkflow :
Saga<WorkflowState>,
IAmStartedByMessages<RegisterUser>,
IHandleMessages<ActivationReceived>,
IHandleTimeouts<NoResponseFor24Hours>,
IHandleTimeouts<NoResponseFor48Hours>
{
protected override void ConfigureHowToFindSaga(SagaPropertyMapper<WorkflowState> mapper)
{
// omitted for simplicity, see message correlation
// https://docs.particular.net/nservicebus/sagas/message-correlation
}
public async Task Handle(RegisterUser message, IMessageHandlerContext context)
{
Data.RegistrationId = message.RegistrationEmail;
await RequestTimeout<NoResponseFor24Hours>(context, TimeSpan.FromHours(24));
}
public async Task Handle(ActivationReceived message, IMessageHandlerContext context)
{
Data.ConfirmationReceived = true;
// email was confirmed and account was activated
await context.Send(new PromoteCandidateToUser
{
CandidateEmail = Data.RegistrationEmail
});
MarkAsComplete()
}
public async Task Timeout(NoResponseFor24Hours timeout, IMessageHandlerContext context)
{
if (Data.ConfirmationReceived)
{
return;
}
await context.Send(new SendReminderEmailToActivateAccount { Email = Data.RegistrationEmail });
await RequestTimeout(context, TimeSpan.FromHours(24), new NoResponseFor48Hours());
}
public async Task Timeout(NoResponseFor48Hours timeout, IMessageHandlerContext context)
{
if (Data.ConfirmationReceived)
{
return;
}
context.Send(new CleanupRegistrationInformationForGDPRCompliancy
{
RegistrationEmail = Data.RegistrationEmail
});
MarkAsComplete();
}
}
Since this is a state machine, the state is persisted between Saga invocations. Invocation would be caused either by a message a saga can handle (RegisterUser and ActivationReceived) or by timeouts that are due (NoResponseFor24Hours and NoResponseFor48Hours). For this specific saga, the state is defined by the following POCO:
class WorkflowState : ContainSagaData
{
public string RegistrationEmail { get; set; }
public bool ConfirmationReceived { get; set; }
}
Timeouts are nothing but plain IMessages that get deferred. The timeouts used in this samples would be
class NoResponseFor24Hours : IMessage {}
class NoResponseFor48Hours : IMessage {}
Hope this clarifies the idea of Sagas in general, what Timeouts are and how they are used. I did not go into Message Correlation, Saga Concurrency, and some other details as those can be found at the documentation site you've referred to. Which bring us to the next point.
I have read multiple times the documentation on their website. It is absolutely terrible. I am reading again and again the same articles and I cannot comprehend what they are trying to achieve.
The site has a feedback mechanism you should absolutely provide.
Besides there almost no resources in internet related to this subject.
Hope to see you posting a blog (or a series of posts) on this topic. By doing so you'll have a positive contribution.
Full disclaimer: I work on NServiceBus
Is there a elegant way of scheduling tasks using NServiceBus. There is one way I found while searching the net. Does NServiceBus give internal APIs for scheduling.
NServiceBus now has this built in
From here http://docs.particular.net/nservicebus/scheduling-with-nservicebus
public class ScheduleStartUpTasks : IWantToRunWhenTheBusStarts
{
public void Run()
{
Schedule.Every(TimeSpan.FromMinutes(5)).Action(() =>
Console.WriteLine("Another 5 minutes have elapsed."));
Schedule.Every(TimeSpan.FromMinutes(3)).Action(
"MyTaskName",() =>
{
Console.WriteLine("This will be logged under MyTaskName’.");
});
}
}
Note the caveat
When not to use it. You can look at a scheduled task as a simple never
ending saga. But as soon as your task starts to get some logic
(if-/switch-statements) you should consider moving to a saga.
Note: This answer was valid for NServiceBus Version 2.0, but is correct no longer. Version 3 has this functionality. Go read Simon's answer, it is valid for Version 3!
NServiceBus does not have a built-in scheduling system. It is (at a very simple level) a message processor.
You can create a class that implements IWantToRunAtStartup (Run and Stop methods) and from there, create a timer or do whatever logic you need to do to drop messages onto the bus at certain times.
Other people have used Quartz.NET together with NServiceBus to get more granular scheduling functionality.