I have rabbitmq consumer with reactive spring cloud stream
#Bean
public <T> Consumer<Flux<Message<Subscription>>> exportConsumer() {
return new ExportSubscriptionConsumer<>();
}
and below are the configurations
spring:
cloud:
function:
definition: exportConsumer
stream:
rabbit:
bindings:
exportConsumer-in-0:
consumer:
bindQueue: true
queueNameGroupOnly: true
declareExchange: false
bindingRoutingKey: smfexportconsumer
function:
bindings:
exportConsumer-in-0:
destination: jms.durable.queues
bindings:
exportConsumer-in-0:
group: smfexportconsumer
Issue i have is it, after application consuming the few messages suddenly it throws me a error.
Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'exportConsumer-in-0' in vhost '9509bf37-8afc-486e-ac98-61034066671d', class-id=50, method-id=20)
If i restart the application it works for a while but again same issue!
Do i have option to configure retry here, couldn't found any such configurations here
or
Do i have option to configure my custom retry logic.
Kindly help.
[EDIT]
messageFlux
.doOnNext(s -> logger.info("Got message: {}", s))
.map(Message::getPayload)
.flatMap(this::converToCharacters)
.map(User::getId)
.flatMap(this::getUser)
.onErrorContinue((throwable, o) -> logger.error(throwable.getMessage()))
Inside the convert method, I have an API call.
return webClient.get()
.uri(uriBuilderFactory.builder().path("/authorize").build())
.retrieve().bodyToMono(String.class)
.publishOn(Schedulers.immediate())
.timeout(Duration.ofSeconds(10))
.map(s1 -> new User(s, s1));
Now the question I have is if I get a time out with the above endpoint, how do I handle it? is it using
onErrorContinue
or do I have the option of global error handling?
[EDIT-2]
Sample code here
Related
I have read the spring cloud stream binder reference document which mentioned DLQ processing using #RabbitListener. https://docs.spring.io/spring-cloud-stream-binder-rabbit/docs/3.0.10.RELEASE/reference/html/spring-cloud-stream-binder-rabbit.html#rabbit-dlq-processing
Can we achieve the same via Spring cloud function like we can do the same for consumers?
Like
#Bean
public Consumer<Message> dlqprocess(DLQProcess dlqprocess) {
return t -> dlqprocess.do(t);
}
I am not sure whether we can do this or not. If this allows what are the other configuration we have to do?
If you aim is to requeue failed messages, the function can just throw exceptions as described in docs.
Furthermore, if you need more fine-grained control about send and requeued messages you can use StreamBrdidge. Here you need to explicitly define DLQ binding in the configuration file:
spring.cloud.stream.bindings.myDlq-out-0.destination=DLX
spring.cloud.stream.rabbit.bindings.myDlq-out-0.producer.exchangeType=direct
spring.cloud.stream.rabbit.bindings.myDlq-out-0.producer.routingKeyExpression='myDestination.myGroup'
spring.cloud.stream.source=myDlq
Finally, the function controls whether to send and requeue the message:
#Bean
public Consumer<Message> process(StreamBridge streamBridge) {
return t -> {
// ....
if(republish) streamBridge.send("myDlq-out-0", t);
if(sendToDestination) streamBridge.send("myDestination-out-0", t);
// ....
};
}
I'm migrating to the new spring cloud stream version.
Spring Cloud Stream 3.1.0
And I have the following consumer configuration:
someChannel-in-0:
destination: Response1
group: response-channel
consumer:
concurrency: 4
someChannel-out-0:
destination: response2
I have connected this channel to the new function binders
// just sample dummy code
#Bean
public Function<Flux<String>, Flux<String>> aggregate() {
return inbound -> inbound.
.map(w -> w.toLowerCase());
}
And when I'm starting the app I'm getting the following error:
Concurrency > 1 is not supported by reactive consumer, given that project reactor maintains its own concurrency mechanism.
My question is what is the equivalent of concurrency: 4 in project reactor and how do I implement this ?
Basically, Spring cloud stream manage consumers through MessageListenerContainer and it provides a hook which allow users create a bean and inject some advanced configurations. And so here comes to the solution if you are using RabbitMQ as messaging middleware.
#Bean
public ListenerContainerCustomizer<AbstractMessageListenerContainer> listenerContainerCustomizer() {
return (container, destinationName, group) -> {
if (container instanceof SimpleMessageListenerContainer) {
((SimpleMessageListenerContainer) container).setConcurrency("3");
}
};
}
I have a consumer application with spring cloud stream. This application is consuming messages from a queue (Kafka), and per each message the app makes 4 different HTTP calls to 4 different servers, one of then very slow (10 seconds to answer). When the queue is full of messages, like 6000, the app crashes because several reasons (1 - netty runs out of direct memory, 2 - we are using reactor and the thread pool gets empty).
Is there some way to limit the consumption velocity on consumer side either through spring-cloud-stream or kafka? Something like maximum messages per seconds would be nice.
Here you can see the configuration for kafka (application.yml)
spring:
kafka:
bootstrap-servers: my-cloud-kafka-instance
admin:
ssl:
protocol: SSL
properties:
security.protocol: SSL
cloud:
stream:
bindings:
input:
group: my-group
destination: my-destination
content-type: application/json
And here is my consumer (in kotlin):
#Controller
#EnableBinding(Processor::class)
class MyConsumer(
myDependendies
) {
#StreamListener(Processor.INPUT)
fun myMethod(
#Headers headers: Map<String, String>,
#Payload myMessage: Message
) {
myBussinessLogic
}
}
Pollable Message Source allows consumers to control consumption rates. For example, to illustrate briefly, we first define an interface:
public interface PolledProcessor {
#Input
PollableMessageSource destIn();
#Output
MessageChannel destOut();
}
Use examples:
#Autowired
private PolledProcessor polledProcessor;
#Scheduled(fixedDelay = 5_000)
public void poll() {
polledProcessor.destIn().poll(message -> {
byte[] bytes = (byte[]) message.getPayload();
String payload = new String(bytes);
logger.info("Received: " + payload);
polledProcessor.destOut().send(MessageBuilder.withPayload(payload.toUpperCase())
.copyHeaders(message.getHeaders())
.build());
});
}
Reference resources:
https://spring.io/blog/2018/02/27/spring-cloud-stream-2-0-polled-consumers
I'd like to know what is the canonical way to handle errors in the following situation (code is a minimal working example):
Messages are sent through a messaging gateway which defines its defaultRequestChannel and a #Gateway method:
#MessagingGateway(name = MY_GATEWAY, defaultRequestChannel = INPUT_CHANNEL)
public interface MyGateway
{
#Gateway
public void sendMessage(String message);
Messages are read from the channel and sent through an AMQP outbound adapter:
#Bean
public IntegrationFlow apiMutuaInputFlow()
{
return IntegrationFlows
.from(INPUT_CHANNEL)
.handle(Amqp.outboundAdapter(rabbitConfig.myTemplate()))
.get();
}
The RabbitMQ configuration is skeletal:
#Configuration
public class RabbitMqConfiguration
{
#Autowired
private ConnectionFactory rabbitConnectionFactory;
#Bean
public RabbitTemplate myTemplate()
{
RabbitTemplate r = new RabbitTemplate(rabbitConnectionFactory);
r.setExchange(INPUT_QUEUE_NAME);
r.setConnectionFactory(rabbitConnectionFactory);
return r;
}
}
I generally include a bean to define the RabbitMQ configuration I'm relying upon (exchange, queues and bindings), and it actually works fine. But while testing for failure scenarios, I found a situation I don't know how to properly handle using Spring Integration. The steps are:
Remove the beans that configure RabbitMQ
Run the flow against an unconfigured, vanilla RabbitMQ instance.
What I would expect is:
The message cannot be delivered because the exchange cannot be found.
Either I find some way to get an exception from the messaging gateway on the caller thread.
Either I find some way to otherwise intercept this error.
What I find:
The message cannot be delivered because the exchange cannot be found, and indeed this error message is logged every time the #Gateway method is called.
2020-02-11 08:18:40.746 ERROR 42778 --- [ 127.0.0.1:5672] o.s.a.r.c.CachingConnectionFactory : Channel shutdown: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no exchange 'my.exchange' in vhost '/', class-id=60, method-id=40)
The gateway is not failing, nor have I find a way to configure it to do so (e.g.: adding throws clauses to the interface methods, configuring a transactional channel, setting wait-for-confirm and a confirm-timeout).
I haven't found a way to otherwise catch that CachingConectionFactory error (e.g.: configuring a transactional channel).
I haven't found a way to catch an error message on another channel (specified on the gateway's errorChannel), or in Spring Integration's default errorChannel.
I understand such a failure may not be propagated upstream by the messaging gateway, whose job is isolating callers from the messaging API, but I definitely expect such an error to be interceptable.
Could you point me in the right direction?
Thank you.
RabbitMQ is inherently async, which is one reason that it performs so well.
You can, however, block the caller by enabling confirms and returns and setting this option:
/**
* Set to true if you want to block the calling thread until a publisher confirm has
* been received. Requires a template configured for returns. If a confirm is not
* received within the confirm timeout or a negative acknowledgment or returned
* message is received, an exception will be thrown. Does not apply to the gateway
* since it blocks awaiting the reply.
* #param waitForConfirm true to block until the confirmation or timeout is received.
* #since 5.2
* #see #setConfirmTimeout(long)
* #see #setMultiSend(boolean)
*/
public void setWaitForConfirm(boolean waitForConfirm) {
this.waitForConfirm = waitForConfirm;
}
(With the DSL .waitForConfirm(true)).
This also requires a confirm correlation expression. Here's an example from one of the test cases
#Bean
public IntegrationFlow flow(RabbitTemplate template) {
return f -> f.handle(Amqp.outboundAdapter(template)
.exchangeName("")
.routingKeyFunction(msg -> msg.getHeaders().get("rk", String.class))
.confirmCorrelationFunction(msg -> msg)
.waitForConfirm(true));
}
#Bean
public CachingConnectionFactory cf() {
CachingConnectionFactory ccf = new CachingConnectionFactory(
RabbitAvailableCondition.getBrokerRunning().getConnectionFactory());
ccf.setPublisherConfirmType(CachingConnectionFactory.ConfirmType.CORRELATED);
ccf.setPublisherReturns(true);
return ccf;
}
#Bean
public RabbitTemplate template(ConnectionFactory cf) {
RabbitTemplate rabbitTemplate = new RabbitTemplate(cf);
rabbitTemplate.setMandatory(true); // for returns
rabbitTemplate.setReceiveTimeout(10_000);
return rabbitTemplate;
}
Bear in mind this will slow down things considerably (similar to using transactions) so you may want to reconsider whether you want to do this on every send (unless performance is not an issue).
I have an issue in getting the message to spring-cloud-stream spring-boot app.
I am using rabbitMq as message engine.
Message producer is a non spring-boot app, which sends a message using Spring RestTemplate.
Queue Name: "audit.logging.rest"
The consumer application is setup to listen that queue. This app is spring-boot app(spring-cloud-stream).
Below is the consumer code
application.yml
cloud:
stream:
bindings:
restChannel:
binder: rabbit
destination: audit.logging
group: rest
AuditServiceApplication.java
#SpringBootApplication
public class AuditServiceApplication {
#Bean
public ByteArrayMessageConverter byteArrayMessageConverter() {
return new ByteArrayMessageConverter();
}
#Input
#StreamListener(AuditChannelProperties.REST_CHANNEL)
public void receive(AuditTestLogger logger) {
...
}
AuditTestLogger.java
public class AuditTestLogger {
private String applicationName;
public String getApplicationName() {
return applicationName;
}
public void setApplicationName(String applicationName) {
this.applicationName = applicationName;
}
}
Below is the request being sent from the producer App in JSON format.
{"applicationName" : "AppOne" }
Found couple of issues:
Issue1:
What I noticed is the below method is getting triggered only when the method Parameter is mentioned as Object, as spring-cloud-stream is not able to parse the message into Java POJO object.
#Input
#StreamListener(AuditChannelProperties.REST_CHANNEL)
public void receive(AuditTestLogger logger) {
Issue2:
When I changed the method to receive object. I see the object is of type RMQTextMessage which cannot be parsed. However I see actual posted message within it against text property.
I had written a ByteArrayMessageConverter which even didn't help.
Is there any way to tell spring cloud stream to extract the message from RMQTextMessage using MessageConverter and get the actual message out of it.
Thanks in Advance..
RMQTextMessage? Looks like it is a part of rabbitmq-jms-client.
In case of RabbitMQ Binder you should rely only on the Spring AMQP.
Now let's figure out what your producer application is doing.
Since you get RMQTextMessage as value for the #StreamListener method that says me that the sender really uses rabbitmq-jms-client for producing, and therefore the real AMQP message in queue has that RMQTextMessage as a wrapper for real payload.
Why don't use Spring AMQP there as well?
It's a late reply but I have the exact problem and solved it by sending and receiving the messages in application/json format. use this in the spring cloud stream config.
content-type: application/json