I am with a Spring Boot project with WebFlux + Spring Reactor, and it calls other services with ReactiveFeignClient.
How can I integrate Prometheus so that I could monitor response time (with a #Timer) and the percentage of each status code returned by feign call? Like 200, 400, 404... I have only found ways to:
monitor endpoint response time/status code(http_server_requests_seconds)
monitor RestTemplate (as explained here: https://docs.spring.io/spring-metrics/docs/current/public/prometheus, but I use feign)
After all the work I have done, I have seen no reactive feign clients meters in Prometheus output, even though I defined the bean of logger like:
#Bean
public MetricsWebClientFilterFunction metricsWebClientFilterFunction(PrometheusMeterRegistry meterRegistry,
WebClientExchangeTagsProvider provider) {
return new MetricsWebClientFilterFunction(
meterRegistry,
provider,
APP_NAME + "reactive-client-request",
AutoTimer.ENABLED
);
}
#Bean
public MicrometerReactiveLogger feignReactiveLogger(Clock clock,
PrometheusMeterRegistry meterRegistry) {
return new MicrometerReactiveLogger(
clock,
meterRegistry,
APP_NAME + ".feign.client_metrics",
MetricsTag.getMandatory()
);
}
Also, I found it impossible to enable /actuator/prometheus, but only /_system/check/prometheus. I did enable and expose the endpoints of metrics and prometheus.
management:
health:
diskspace:
enabled: false
endpoint:
metrics.enabled: true
prometheus.enabled: true
endpoints:
web:
base-path: /_system/check
exposure:
include: info,health,loggers,metrics,prometheus
path-mapping:
health: /simple
I found the problem: the metrics related with reactive feign clients will only show after you actually do feign calls. Before that they are hidden.
Reactive Feign Client uses WebClient underneath and will be measured automatically with presence of Spring Actuator.
Related
My application set up is mentioned as part of issue# Correct way of using spring webclient in spring amqp
where I am trying to use Spring webclient to make API calls in Spring AMQP rabbit MQ consumer threads.
Issue seems to be that parallel flux blocking call just stalls or takes a very long time after first few requests are fired.
To simulate this, I did below minimalistic set up -
Dependencies used
Spring boot 2.2.6.RELEASE
spring-boot-starter-web
spring-boot-starter-webflux
reactor-netty 0.9.14.RELEASE
As mentioned in the other linked issue, below is configuration for webclient -
#Bean
public WebClient webClient() {
ConnectionProvider connectionProvider = ConnectionProvider
.builder("fixed")
.lifo()
.pendingAcquireTimeout(Duration.ofMillis(200000))
.maxConnections(100)
.pendingAcquireMaxCount(3000)
.maxIdleTime(Duration.ofMillis(290000))
.build();
HttpClient client = HttpClient.create(connectionProvider);
client.tcpConfiguration(<<connection timeout, read timeout, write timeout is set here....>>);
Webclient.Builder builder =
Webclient.builder().baseUrl(<<base URL>>).clientConnector(new ReactorClientHttpConnector(client));
return builder.build();
}
Below is #Service class with parallel flux webclient calls -
#Service
public class FluxtestService {
public Flux<Response> getFlux(List<Request> reqList) {
return Flux
.fromIterable(reqList)
.parallel()
.runOn(Schedulers.elastic())
.flatMap(s -> {
return webClient
.method(POST)
.uri(<<downstream url>>)
.body(BodyInserters.fromValue(s))
.exchange()
.flatMap(response -> {
if(response.statusCode().isError()){
return Mono.just(new Response());
}
return response.bodyToMono(Response.class);
})
}).sequential();
}
}
}
To simulate Spring AMQP rabbit mq consumer/listener, I created below #RestController -
#RestController
public class FluxTestController
#Autowired
private FluxtestService service;
#PostMapping("/fluxtest")
public List<Response> getFlux (List<Request> reqlist) {
return service.getFlux(reqlist).collectList().block();
}
I tried firing requests from jmeter with around 15 threads. First few set of requests are processed very quickly. While requests are being served, I can see below set of logs in log file -
Channel cleaned, now 32 active connections and 68 inactive connections
Once I submit more set of requests, the active connections keeps increasing till it reaches max configured 100. I don't see it decreasing at all. Till this point, response time is ok.
But any subsequent requests start taking very long time. Also I don't see the active connections reducing much at all even though there are no requests being fired.
Also after some time, I see below exceptions -
reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 200000 ms
This probably shows that the downstream connection is not being released. Please help advise on this issue and possible fixes.
Seems issue was because the underlying connection was not being properly released in case webclient downstream call responded with error status. While using "exchange" with "webclient", it seems we need to ensure that the response is properly released; else it can lead to connections leak. Below are the changes that seemed to fix this issue -
Replace
.flatMap(response -> {
if(response.statusCode().isError()) {
return Mono.just(new Response());
}
return response.bodyToMono(Response.class);
})
with
.flatMap(response -> {
if(response.statusCode().isError()) {
response.releaseBody().thenReturn(Mono.just(new Response()));
}
return response.bodyToMono(Response.class);
})
I am using Spring reactor and WebClient for load balance purposes. I want to use the Request-based Sticky Session for LoadBalancer as follows:
Have service A that can invoke REST end points on service B or service C. Both B and C can have multiple instances of them running at a given time.
I want to use the standard round robin balancer for service B.
I want to use the request-based sticky session balancer for service C.
This is my code in the service A (that can invokes both service B and service C) through the WebClient. But it always ends up with round robin algorithm for service C. Can someone please
help me to figure out what I am doing wrong or missed to do? I referred to the below link
https://docs.spring.io/spring-cloud-commons/docs/current/reference/html/#request-based-sticky-session-for-loadbalancer
and also looked into the below SO
Request-based Sticky Session configuration with Spring Cloud LoadBalancer
#Configuration
public class WebClientConfiguration {
#Bean
#LoadBalanced
#Qualifier("loadBalancedWebClientBuilder")
WebClient.Builder builder() {
return WebClient.builder();
}
#Bean
#Qualifier("webClientBuilder")
WebClient webClient(WebClient.Builder builder) {
return builder.build();
}
}
application.yml (I am not very sure where the sticky-session definition should be added; i.e. service A or service C and hence repeated the below in both the services)
spring:
application:
name: service-A
cloud:
loadbalancer:
ribbon:
enabled: 'false'
sticky-session:
instanceIdCookieName: sc-lb-instance-id
addServiceInstanceCookie: true
Code in the service A to invoke service C with request based sticky session. I have a different file that does the same for service B (without the sticky session but uses the same WebClient Builder).
I know the service instance id that I want the WebClient to use to route the call.
#Component
public class ServiceCClient {
#Autowired
#Qualifier("loadBalancedWebClientBuilder")
private WebClient.Builder webClientBuilder;
public Mono<Void> triggerServiceCEndPoint(String paramA, String svcInstanceId) {
return webClientBuilder.baseUrl("http://service-C").build()
.post().uri(uriBuilder -> uriBuilder.path("/v1/example/end/point")
.queryParam("param-a",paramA).build())
.headers(httpHeaders -> {
httpHeaders.set("sc-lb-instance-id",svcInstanceId);
})
.cookie("sc-lb-instance-id", svcInstanceId)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(Void.class);
}
}
I'm migrating to the new spring cloud stream version.
Spring Cloud Stream 3.1.0
And I have the following consumer configuration:
someChannel-in-0:
destination: Response1
group: response-channel
consumer:
concurrency: 4
someChannel-out-0:
destination: response2
I have connected this channel to the new function binders
// just sample dummy code
#Bean
public Function<Flux<String>, Flux<String>> aggregate() {
return inbound -> inbound.
.map(w -> w.toLowerCase());
}
And when I'm starting the app I'm getting the following error:
Concurrency > 1 is not supported by reactive consumer, given that project reactor maintains its own concurrency mechanism.
My question is what is the equivalent of concurrency: 4 in project reactor and how do I implement this ?
Basically, Spring cloud stream manage consumers through MessageListenerContainer and it provides a hook which allow users create a bean and inject some advanced configurations. And so here comes to the solution if you are using RabbitMQ as messaging middleware.
#Bean
public ListenerContainerCustomizer<AbstractMessageListenerContainer> listenerContainerCustomizer() {
return (container, destinationName, group) -> {
if (container instanceof SimpleMessageListenerContainer) {
((SimpleMessageListenerContainer) container).setConcurrency("3");
}
};
}
So I've created a custom filter that, when accessed, will create a webflux client and post to a predetermined url. This seems to work fine when running, but when testing this code the test is hanging (until I cancel the test). So I feel there is a possible memory leak on top of not being able to complete the test to make sure this route is working properly. If I switch the WebClient method to get() then a resulting test of the filter works fine. Something with a post() I am not sure what is missing.
#Component
class ProxyGatewayFilterFactory: AbstractGatewayFilterFactory<ProxyGatewayFilterFactory.Params>(Params::class.java) {
override fun apply(params: Params): GatewayFilter {
return OrderedGatewayFilter(
GatewayFilter { exchange, chain ->
exchange.request.mutate().header("test","test1").build()
WebClient.create().post()
.uri(params.proxyBasePath)
.body(BodyInserters.fromDataBuffers(exchange.request.body))
.headers { it.addAll(exchange.request.headers) }
.exchange()
.flatMap {
println("the POST statusCode is "+it.statusCode())
Mono.just(it.statusCode().is2xxSuccessful)
}
.map {
exchange.request.mutate().header("test", "test2").build()
println("exchange request uri is " + exchange.request.uri)
println("exchange response statusCode is "+ exchange.response.statusCode)
exchange
}
.flatMap(chain::filter)
}, params.order)
}
Taken from the documentation, if using exchange you have an obligation to consume the body.
Unlike retrieve(), when using exchange(), it is the responsibility of the application to consume any response content regardless of the scenario (success, error, unexpected data, etc). Not doing so can cause a memory leak. The Javadoc for ClientResponse lists all the available options for consuming the body. Generally prefer using retrieve() unless you have a good reason for using exchange() which does allow to check the response status and headers before deciding how to or if to consume the response.
Spring framework 5.2.9 Webclient
This api has been changed in the latest version of the spring framework 5.3.0 now spring will force you to consume the body, because developers didn't actually read the docs.
I have an issue in getting the message to spring-cloud-stream spring-boot app.
I am using rabbitMq as message engine.
Message producer is a non spring-boot app, which sends a message using Spring RestTemplate.
Queue Name: "audit.logging.rest"
The consumer application is setup to listen that queue. This app is spring-boot app(spring-cloud-stream).
Below is the consumer code
application.yml
cloud:
stream:
bindings:
restChannel:
binder: rabbit
destination: audit.logging
group: rest
AuditServiceApplication.java
#SpringBootApplication
public class AuditServiceApplication {
#Bean
public ByteArrayMessageConverter byteArrayMessageConverter() {
return new ByteArrayMessageConverter();
}
#Input
#StreamListener(AuditChannelProperties.REST_CHANNEL)
public void receive(AuditTestLogger logger) {
...
}
AuditTestLogger.java
public class AuditTestLogger {
private String applicationName;
public String getApplicationName() {
return applicationName;
}
public void setApplicationName(String applicationName) {
this.applicationName = applicationName;
}
}
Below is the request being sent from the producer App in JSON format.
{"applicationName" : "AppOne" }
Found couple of issues:
Issue1:
What I noticed is the below method is getting triggered only when the method Parameter is mentioned as Object, as spring-cloud-stream is not able to parse the message into Java POJO object.
#Input
#StreamListener(AuditChannelProperties.REST_CHANNEL)
public void receive(AuditTestLogger logger) {
Issue2:
When I changed the method to receive object. I see the object is of type RMQTextMessage which cannot be parsed. However I see actual posted message within it against text property.
I had written a ByteArrayMessageConverter which even didn't help.
Is there any way to tell spring cloud stream to extract the message from RMQTextMessage using MessageConverter and get the actual message out of it.
Thanks in Advance..
RMQTextMessage? Looks like it is a part of rabbitmq-jms-client.
In case of RabbitMQ Binder you should rely only on the Spring AMQP.
Now let's figure out what your producer application is doing.
Since you get RMQTextMessage as value for the #StreamListener method that says me that the sender really uses rabbitmq-jms-client for producing, and therefore the real AMQP message in queue has that RMQTextMessage as a wrapper for real payload.
Why don't use Spring AMQP there as well?
It's a late reply but I have the exact problem and solved it by sending and receiving the messages in application/json format. use this in the spring cloud stream config.
content-type: application/json